# reality-hand-tracking

> Use this skill when the user asks to track hand joints, recognize hand gestures, drive hand models with skeleton data, or implement hand-object interaction on visionOS. Supports both simple AnchorEntity (no permissions) and full ARKit HandAnchor (27 joints, requires Full Immersive).

- Author: xander
- Repository: XanderXu/RealitySkills
- Version: 20260123110406
- Stars: 4
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/XanderXu/RealitySkills
- Web: https://mule.run/skillshub/@@XanderXu/RealitySkills~reality-hand-tracking:20260123110406

---

---
name: reality-hand-tracking
description: Use this skill when the user asks to track hand joints, recognize hand gestures, drive hand models with skeleton data, or implement hand-object interaction on visionOS. Supports both simple AnchorEntity (no permissions) and full ARKit HandAnchor (27 joints, requires Full Immersive).
---

# Reality Hand Tracking

## Overview

Track hand joints in 3D space on visionOS. Use **RealityKit AnchorEntity** for simple single-joint tracking (no permissions), or **ARKit HandAnchor** for full 27-joint skeletal data (requires Full Immersive authorization).

## Quick Start

### AnchorEntity in RealityKit

**Track thumb tip with AnchorEntity (no permissions):**

```swift
RealityView { content in
    let thumbTip = AnchorEntity(.hand(.left, .thumbTip))
    content.add(thumbTip)
    thumbTip.addChild(ModelEntity(mesh: .generateSphere(radius: 0.01)))
}
```
### HandAnchor in ARKit

See `references/CODE_EXAMPLES.md` for full HandAnchor (27 joints, requires Full Immersive).


**Privacy Permissions:**

Add to your `Info.plist` when using HandAnchor:

```xml
<key>NSHandsTrackingUsageDescription</key>
<string>This app needs hand tracking to [explain why - e.g., "recognize gestures for interactive controls"]</string>
```

### Requirements
- **visionOS**: 1.0+ for AnchorEntity, 2.0+ for HandAnchor features
- **Xcode**: 15.0+ for visionOS development
- **Authorization**: `NSHandsTrackingUsageDescription` for HandAnchor (not required for AnchorEntity)

## Core Guidelines

### Quick Decision

| Need | Solution |
|------|----------|
| Follow finger with UI element | AnchorEntity |
| Access all joints / gestures | HandAnchor |
| Animate hand model | HandAnchor + jointTransforms |
| Detect pinch/fist/victory | HandAnchor + gesture logic |

### Core Pipeline

```
Request Authorization → Run ARKitSession → Subscribe Updates → Process HandAnchor → Update Entities
```

### Joint Hierarchy (27 joints)

```
Wrist (root)
├── Thumb: Knuckle → IntermediateBase → IntermediateTip → Tip
├── Index: Metacarpal → Knuckle → IntermediateBase → IntermediateTip → Tip
├── Middle: Metacarpal → Knuckle → IntermediateBase → IntermediateTip → Tip
├── Ring: Metacarpal → Knuckle → IntermediateBase → IntermediateTip → Tip
└── Little: Metacarpal → Knuckle → IntermediateBase → IntermediateTip → Tip

Plus: ForearmWrist, ForearmArm
```

### Best Practices

**Coordinate Systems:**
- Use `anchorFromJointTransform` for positioning
- Use `parentFromJointTransform` for animation
- Left hand axes differ from RealityKit - test on device!

**Performance:**
- Check `isTracked` before using joint data (occlusion happens)
- Throttle gesture recognition to ~10Hz (90Hz tracking is overkill)
- Use rotation-only updates to prevent model deformation
- Reuse entities instead of creating/destroying every frame

**Common Patterns:**
- Pull updates with `SceneEvents.Update` for simple logic
- Push updates with `anchorUpdates` for complex logic
- Skip occluded joints (`!isTracked`)

### Common Issues

**Left hand coordinate system differs** → Test on device, axes are flipped vs RealityKit

**Model deformation** → Use rotation-only updates, ignore scale in transforms

**Occlusion tracking errors** → Always check `isTracked` before using joint data

**Performance degradation** → Throttle gesture recognition, reuse entities

## Reference Material

- `references/TECHNICAL_GUIDE.md` - Architecture, coordinate systems, and optimization
- `references/CODE_EXAMPLES.md` - Complete HandAnchor implementation, gesture detection, skeletal animation
- [Apple Hand Tracking Guide](https://developer.apple.com/documentation/visionos/tracking-and-visualizing-hand-movement)
- [HandVector Library](https://github.com/XanderXu/HandVector) for gesture matching