# reality-video-mps-cifilter

> Use this skill when the user asks to implement real-time video filters, custom video compositors, CVPixelBuffer to Metal texture conversion, or compare MPS vs CIFilter performance for video streams on visionOS.

- Author: xander
- Repository: XanderXu/RealitySkills
- Version: 20260123110406
- Stars: 4
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/XanderXu/RealitySkills
- Web: https://mule.run/skillshub/@@XanderXu/RealitySkills~reality-video-mps-cifilter:20260123110406

---

---
name: reality-video-mps-cifilter
description: Use this skill when the user asks to implement real-time video filters, custom video compositors, CVPixelBuffer to Metal texture conversion, or compare MPS vs CIFilter performance for video streams on visionOS.
---

# Reality Video MPS & CIFilter

## Overview

Apply GPU-accelerated video processing on visionOS using Metal Performance Shaders (MPS) for real-time effects or Core Image Filter (CIFilter) for artistic filters. Integrate processed frames into RealityKit through custom compositors and VideoMaterial.

## Quick Start

**Simple CIFilter video composition:**

```swift
let composition = try await AVMutableVideoComposition.videoComposition(with: asset) { request in
    let filter = CIFilter(name: "CIGaussianBlur")
    filter?.setValue(request.sourceImage, forKey: kCIInputImageKey)
    filter?.setValue(10.0, forKey: kCIInputRadiusKey)

    if let output = filter?.outputImage {
        request.finish(with: output, context: CIContext(options: [.cacheIntermediates: false]))
    }
}
```

**High-performance MPS (40-80% faster):** See `references/CODE_EXAMPLES.md` for implementation.

## Architecture Principles

1. **Separation of Concerns**: Compositor extracts frames, processor applies effects
2. **Testability**: Independent testing of frame extraction and processing logic
3. **Resource Reuse**: Cache Metal resources (texture cache, command queues)
4. **Async Communication**: AsyncStream for efficient updates
5. **Performance Monitoring**: Built-in GPU timing and tracking

## Workflow Decision Tree

### 1) Real-time video filter (Recommended for MPS)
- Load video using AVURLAsset, create `TextureExtractorCompositor` AVVideoCompositing class
- Emit CVPixelBuffer via AsyncStream to external `VideoProcessor`
- Convert: CVPixelBuffer → MTLTexture → MPS → LowLevelTexture
- **Optimization**: Reuse CVMetalTextureCache and command queues

### 2) Simple CIFilter composition
- Create AVMutableVideoComposition with async filtering callback
- Configure CIFilter with video frames and parameters
- **Best for**: Rapid prototyping and simple filter chains

### 3) Optimize existing performance
- Profile with Metal System Trace, switch CIFilter to MPS if frame drops
- Reuse CVMetalTextureCache, use completion handlers instead of blocking

## Core Pipeline

```
AVURLAsset → AVAssetTrack → CVPixelBuffer
                                    ↓
                    TextureExtractorCompositor
                                    ↓
                      AsyncStream<CVPixelBuffer>
                                    ↓
                          VideoProcessor
                                    ↓
                            MTLTexture → MPS
                                    ↓
                            LowLevelTexture
                                    ↓
                            TextureResource → Material
                                    ↓
                              ModelEntity
```

## Core Guidelines

### Requirements
- **visionOS**: 2.0+ for LowLevelTexture with video processing
- **Xcode**: 15.0+ for visionOS development
- **Frameworks**: AVFoundation, Metal, MetalPerformanceShaders
- **Device**: Apple Vision Pro (required for video capture)

### Best Practices
- **Approach**: Custom compositor for MPS, AVMutableVideoComposition for CIFilter
- **Pixel format**: `kCVPixelFormatType_32BGRA` with `kCVPixelBufferMetalCompatibilityKey: true`
- **Resource reuse**: Create CVMetalTextureCache and command queues once per session
- **Async processing**: Use completion handlers instead of `waitUntilCompleted()`

### Performance Optimizations
- **Texture cache reuse**: Create once, reuse for all frames (12-15% gain)
- **Async completion**: `commandBuffer.addCompletedHandler` instead of blocking
- **Actor-based state**: Swift actors for thread-safe processing
- **Resource pooling**: Reuse intermediate textures for multi-pass effects

## Common Video Filters

See [shared/MPS_FILTERS_REFERENCE.md](../shared/MPS_FILTERS_REFERENCE.md) for comprehensive MPS filters documentation.

- **MPS (High Performance)**: `MPSImageGaussianBlur`, `MPSImageSobel`, `MPSImageHistogram`, `MPSImageThresholdBinary`
- **CIFilter (Rich Effects)**: `CIGaussianBlur`, `CIColorControls`, `CIVignette`, `CIPhotoEffectMono`

## Reference Material

- `references/TECHNICAL_GUIDE.md` - Architecture overview, CVPixelBuffer conversion, performance benchmarks, troubleshooting
- `references/CODE_EXAMPLES.md` - Complete implementations: CIFilter composition, MPS custom compositor with separated concerns, RealityView integration