# reality-offscreen-renderer

> Use this skill when the user asks to implement off-screen rendering in RealityKit, create custom rendering pipelines, generate shadow maps, or implement stereoscopic 3D rendering with MTLTexture output on visionOS.

- Author: xander
- Repository: XanderXu/RealitySkills
- Version: 20260123110406
- Stars: 4
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/XanderXu/RealitySkills
- Web: https://mule.run/skillshub/@@XanderXu/RealitySkills~reality-offscreen-renderer:20260123110406

---

---
name: reality-offscreen-renderer
description: Use this skill when the user asks to implement off-screen rendering in RealityKit, create custom rendering pipelines, generate shadow maps, or implement stereoscopic 3D rendering with MTLTexture output on visionOS.
---

# RealityKit Offscreen Renderer

## Overview

Implement off-screen rendering in RealityKit using `RealityRenderer` to render 3D content to Metal textures. Supports single and stereo camera configurations for effects like shadows, projections, and stereoscopic 3D.

## Quick Start

**Basic offscreen renderer:**

```swift
let renderer = try RealityRenderer()
let camera = Entity()
camera.components.set(OrthographicCameraComponent())
renderer.activeCamera = camera
renderer.entities.append(camera)

let texture = device.makeTexture(descriptor: descriptor)!
let output = try RealityRenderer.CameraOutput(.singleProjection(colorTexture: texture))

try await renderer.updateAndRender(deltaTime: 0.016, cameraOutput: output) { _ in }
```

See `references/CODE_EXAMPLES.md` for stereo renderer and camera controllers.

## Architecture Principles

1. **Async Rendering**: Thread-safe rendering with async/await patterns
2. **Rendering Guards**: Prevent concurrent renders with `isRendering` flag
3. **Resource Reuse**: Cache texture descriptors and entities
4. **Camera Control**: Auto-framing and dynamic camera adjustments
5. **Performance Optimization**: Batch updates and limit per-frame calculations

## Workflow Decision Tree

### 1) Single camera offscreen rendering
Create `OffscreenRenderer` with orthographic or perspective camera → Add entities to scene → Render to MTLTexture for materials/effects

### 2) Stereo 3D rendering
Create `StereoOffscreenRenderer` with dual cameras → Configure IPD (Interpupillary Distance) → Render separate textures for each eye

### 3) Shadow/projection effects
Use orthographic camera for parallel projection → Position camera above scene → Use rendered texture as shadow map

## Core Pipeline

```
RealityRenderer → Camera Entity → Entities (3D Content)
                                            ↓
                                    CameraOutput (MTLTexture)
                                            ↓
                                    Rendered Texture
```

## Core Guidelines

### Requirements
- **visionOS**: 1.0+ for RealityRenderer, 2.0+ for advanced features
- **Xcode**: 15.0+ for visionOS development
- **Device**: Apple Vision Pro (required)

### Best Practices
- **Rendering guard**: Always use `isRendering` flag to prevent concurrent renders
- **Camera setup**: Use orthographic for shadows, perspective for 3D scenes
- **Texture format**: `.rgba8Unorm` or `.bgra8Unorm` for most use cases
- **Auto-framing**: Use `cameraLookAtBoundingBoxCenter()` for automatic camera positioning
- **Async pattern**: Use `withCheckedThrowingContinuation` for async rendering

### Performance Optimizations
- **Texture reuse**: Create descriptor once, reuse for multiple textures
- **Batch updates**: Update all entities before single render call
- **Limited auto-look**: Restrict auto-framing to first 5 frames
- **Concurrent guard**: Check `isRendering` before starting render
- **Resource cleanup**: Remove entities and set references to nil when done

## Camera Types

| Camera | Use Case | Key Parameters |
|--------|----------|----------------|
| **Orthographic** | Shadows, 2D views | `scale` (zoom level) |
| **Perspective** | 3D scenes, depth | `fieldOfView` (degrees) |

## Common Issues

### Black Output Texture
- Camera not pointing at entities → Use `cameraLookAtBoundingBoxCenter()`
- Entities outside frustum → Adjust camera scale or position
- Wrong scale → Verify orthographic camera scale

### Rendering Freezes
- `isRendering` not reset → Use `defer` to guarantee cleanup
- Concurrent renders → Always check `isRendering` flag
- Error in continuation → Handle errors in both success/failure paths

### Poor Performance
- Too many entities → Reduce entity count or use LOD
- No render guard → Add `isRendering` check
- Frequent auto-look → Limit to first 5 frames

## Reference Material

- `references/TECHNICAL_GUIDE.md` - Architecture, camera config, performance optimization, troubleshooting
- `references/CODE_EXAMPLES.md` - Complete single/stereo renderer implementations, camera controllers, integration examples