--- name: pi-subagent-orchestration-git-only description: Orchestrate subagents in pi with git-based logging and Mission Control. Use when spawning multiple agents that need audit trails, rollback, branching, and real-time monitoring. Covers per-agent git repos, turn-level commits, workspace setup, and the shadow-git extension. --- # Pi Subagent Orchestration **YOU ARE AN ORCHESTRATING AGENT.** Your job is to decompose complex tasks, spawn subagents, coordinate their work, and synthesize results. You never execute leaf-level work yourself when subagents are more appropriate. --- ## ⚠️ THE RESOURCE COMMANDMENTS ⚠️ **SUBAGENTS CONSUME REAL RESOURCES. EVERY AGENT YOU SPAWN:** - Uses CPU and memory on the user's machine - Costs money (API calls to LLM providers) - Runs until killed or max-turns reached - Can accumulate silently if forgotten ### The 7 Commandments 1. **THOU SHALT KILL WHAT THOU SPAWNS** - Every spawn must have a corresponding cleanup plan - Never assume agents will terminate themselves 2. **THOU SHALT TRACK ALL PROCESSES** - Record PIDs or tmux session names - Maintain a list of what you spawned 3. **THOU SHALT SET MAX-TURNS** - Always use `--max-turns` to prevent runaway agents - 20-30 turns for research, 50 max for complex tasks 4. **THOU SHALT CLEAN UP ON EXIT** - Before ending your session, kill all spawned agents - `tmux kill-server` or `kill $(cat agents\/*\/pid)` 5. **THOU SHALT CHECK BEFORE SPAWNING** - Run `tmux ls` to see existing sessions - Run `ps aux | grep pi` to find orphaned agents - Kill stale processes before starting new work 6. **THOU SHALT NOT SPAWN OPUS LIGHTLY** - `claude-opus-4-5` costs 15x more than haiku - Use haiku for research, sonnet for synthesis - Opus only for genuinely hard problems 7. **THOU SHALT MONITOR RESOURCE USAGE** - Check `top` or Activity Monitor periodically - Watch for runaway CPU\/memory consumption - Kill stuck agents immediately ### Cleanup Commands (Memorize These) ```bash # See all tmux sessions tmux ls # Kill ALL tmux sessions (nuclear option) tmux kill-server # Kill specific session tmux kill-session -t agent-name # Find orphaned pi processes ps aux | grep -E "[p]i --model" # Kill by PID kill # Check system load uptime ``` ### Before You Start ANY Orchestration ```bash # 1. Check for existing agents tmux ls 2>\/dev\/null || echo "No tmux sessions" ps aux | grep -E "[p]i " | grep -v grep # 2. Kill any stale agents from previous work tmux kill-server 2>\/dev\/null # 3. Check system resources uptime # Load should be < number of CPU cores # 4. ⚠️ CRITICAL: Force correct model in settings (CLI flags are IGNORED!) cat > ~\/.pi\/agent\/settings.json << 'EOF' { "defaultProvider": "anthropic", "defaultModel": "claude-haiku-4-5", "defaultThinkingLevel": "none" } EOF echo "Model set to: $(jq -r .defaultModel ~\/.pi\/agent\/settings.json)" ``` **⚠️ WARNING: `--model` CLI flag is IGNORED! Pi uses `~\/.pi\/agent\/settings.json` instead.** **If you skip step 4, you WILL spawn Opus agents at 15x the cost of Haiku.** **IF YOU SPAWN IT, YOU OWN IT. CLEAN UP AFTER YOURSELF.** --- ## Quick Start: Spawn a Single Agent If you just need to spawn ONE agent quickly, here's the minimum viable setup: ```bash # 1. Create workspace WORKSPACE="$HOME\/workspaces\/$(date +%Y%m%d)-task" mkdir -p "$WORKSPACE\/agents\/scout1"\/{workspace,output} cd "$WORKSPACE" # 2. Write the plan cat > agents\/scout1\/plan.md << 'EOF' # Plan: Research Task ## Objective Research X and produce findings. ## Steps 1. Read the codebase\/docs 2. Document findings in output\/findings.md ## Output - output\/findings.md EOF # 3. Spawn (using --print for non-interactive mode) (cd $WORKSPACE\/agents\/scout1 && \ pi \ --model claude-haiku-4-5 \ --max-turns 20 \ --print \ 'Read plan.md and execute it.' \ 2>&1 | tee output\/run.log) & echo $! > agents\/scout1\/pid # 4. Check status ps -p $(cat agents\/scout1\/pid) >\/dev\/null 2>&1 && echo "Running" || echo "Done" # 5. View output when done cat agents\/scout1\/output\/findings.md ``` **Alternative: Use tmux for observability** (can attach to see progress): ```bash tmux new-session -d -s scout1 \ "cd $WORKSPACE\/agents\/scout1 && \ pi --model claude-haiku-4-5 --max-turns 20 \ 'Read plan.md and execute it.' \ 2>&1 | tee output\/run.log" ``` **That's it.** For more complex orchestration (multiple agents, audit trails, coordination), read on. --- ## Table of Contents 0. [THE RESOURCE COMMANDMENTS](#️-the-resource-commandments-️) ← **READ THIS FIRST** 1. [Critical Concepts](#critical-concepts) 2. [Pre-Flight Decisions](#pre-flight-decisions) 3. [Workspace Structure](#workspace-structure) 4. [Spawning Methods](#spawning-methods) 5. [Shadow Git Extension (Audit Trail)](#shadow-git-extension-audit-trail) 6. [Mission Control Dashboard](#mission-control-dashboard) 7. [Output Aggregation Patterns](#output-aggregation-patterns) 8. [Git-Based Coordination](#git-based-coordination) 9. [Logging Protocol](#logging-protocol) 10. [Error Handling and Recovery](#error-handling-and-recovery) 11. [Quick Reference](#quick-reference) 12. [Gotchas and Hard-Won Lessons](#gotchas-and-hard-won-lessons) ← **LEARN FROM THESE** --- ## Critical Concepts ### Why Subagents? | Problem | Solution | |---------|----------| | Task too large for one context window | Decompose into focused subtasks | | Need parallelism | Spawn multiple agents simultaneously | | Need isolation | Each agent has clean context | | Need audit trail | Shadow git tracks everything | | Task may exceed max turns | Handoff to continuation agent | ### Pi Sessions vs Shadow Git **Pi has built-in session management** — it stores conversation history in `~\/.pi\/agent\/sessions\/`. This is great for single-agent work but has limitations for orchestration: | Feature | Pi Sessions | Shadow Git | |---------|-------------|------------| | Scope | Per-agent, per-cwd | Per-agent repos, unified workspace | | Branching | Tree structure, UI-driven | Git branches, CLI-driven | | Multi-agent | Separate session files | Per-agent repos, shared manifest | | Audit queries | Limited | `git log`, `jq` on JSONL | | Rollback | Limited | `\/shadow-git rollback ` | | Target repo changes | Not tracked | Captured as patches | **Use both**: Pi sessions for individual agent state, shadow git for orchestration-level audit trail. ### The TTY Problem (CRITICAL) **Pi has a Terminal User Interface (TUI)** that shows progress, token counts, and costs. This TUI needs a real terminal (TTY) to render. **The problem:** When you run `pi` in the background with `&` or `nohup`, there's no terminal. The TUI tries to render, fails, and the process crashes or hangs. **Simple decision tree:** ``` Do you need to OBSERVE the agent while it runs? │ ├── YES → Use tmux (provides virtual terminal) │ tmux new-session -d -s name "pi ..." │ └── NO → Use --print flag (disables TUI, non-interactive mode) (pi --print ...) & ``` **NEVER do this:** ```bash pi ... & # WRONG: No TTY, no --print → crash nohup pi ... & # WRONG: nohup doesn't provide TTY ``` --- ## Pre-Flight Decisions **Before spawning ANY subagent, decide and document:** ### 1. Execution Mode | Mode | When | How | |------|------|-----| | **Blocking** | Need result before continuing | `pi --print ...` (no `&`) | | **Non-blocking** | Fire and forget, check later | tmux or `(pi --print ...) &` | | **Parallel + Join** | Multiple independent tasks, wait for all | Spawn all, then `wait` | ### 2. Environment Strategy | Strategy | When | Setup | |----------|------|-------| | **Shared workspace** | Read-only operations | All agents in same dir | | **Separate directories** | Writes, no git | `agents\/{name}\/workspace\/` | | **Git worktrees** | Writes, need isolation + merge | `git worktree add ...` | | **Git branches** | Sequential handoffs | Branch per stage | ### 3. Output Aggregation | Strategy | When | |----------|------| | **Orchestrator reads all** | Few agents (3-5), small outputs | | **Aggregator subagent** | Many agents, need synthesis | | **Programmatic** | JSON\/CSV, deterministic combine | | **User explores** | Uncertain value, user guides | **Record these decisions** in `orchestrator\/decisions.md` before spawning. --- ## Workspace Structure ### Per-Agent Git Repos (v2.0+) Each agent has its **own isolated git repository**, eliminating lock conflicts: ``` {workspace_root}\/ ├── manifest.json # Agent registry (auto-created by extension) ├── orchestrator\/ │ ├── decisions.md # Pre-flight decisions │ ├── log.md # Orchestrator's execution log │ └── synthesis\/ # Final outputs │ └── agents\/ ├── {agent_name}\/ │ ├── .git\/ # Agent's OWN repo (auto-created) │ ├── .gitignore # Excludes audit.jsonl │ ├── audit.jsonl # Real-time event log (NOT in git) │ ├── state.json # Checkpoint state (IN git) │ ├── plan.md # REQUIRED: What to do │ ├── log.md # Agent's execution log │ └── output\/ # Agent's deliverables (IN git) │ └── {agent_name_2}\/ ├── .git\/ # Completely isolated from other agents └── ... ``` **Key Benefits:** - **Zero lock conflicts** — Parallel agents never compete for `.git\/index.lock` - **Turn-level commits** — ~10x fewer commits (per turn, not per tool) - **Clean separation** — `audit.jsonl` for real-time observability, git for checkpoints - **Rollback\/branch** — Per-agent history with `\/shadow-git rollback` and `\/shadow-git branch` ### Creating a Workspace ```bash # Create workspace (no git init needed at root - agents create their own) WORKSPACE="$HOME\/workspaces\/$(date +%Y%m%d)-$TASK_NAME" mkdir -p "$WORKSPACE"\/{orchestrator,agents} cd "$WORKSPACE" # Create orchestrator files cat > orchestrator\/decisions.md << 'EOF' # Pre-Flight Decisions ## Task {description} ## Execution Mode {blocking|non-blocking|parallel-join} ## Environment {shared|directories|worktrees|branches} ## Output Aggregation {orchestrator|aggregator|programmatic|user-explores} ## Agents | Name | Role | Model | Tools | |------|------|-------|-------| | scout1 | research | claude-haiku-4-5 | read,bash,browser | | worker1 | implement | claude-sonnet-4-5 | read,write,edit,bash | EOF echo "# Orchestrator Log" > orchestrator\/log.md ``` ### Creating an Agent Directory ```bash AGENT="scout1" mkdir -p "agents\/$AGENT"\/{workspace,output} # Write plan.md (REQUIRED) cat > "agents\/$AGENT\/plan.md" << 'EOF' # Plan: {Task Name} ## Objective {One sentence: what this agent must accomplish} ## Scope - IN: {what IS in scope} - OUT: {what is NOT in scope} ## Steps ### STEP-01: {Title} - Do: {action} - Output: {file or result} ### STEP-02: {Title} ... ### FINAL: Compile Output - Write findings to output\/ - Ensure log.md is complete ## Boundaries - Do NOT exceed scope - Log ambiguity, choose simplest path - All deliverables → output\/ EOF ``` --- ## Spawning Methods ### Method 1: tmux (RECOMMENDED for observability) **Why tmux?** It provides a virtual terminal, so pi's TUI works. You can attach to observe progress, costs, and debug in real-time. ```bash WORKSPACE="$(pwd)" AGENT="scout1" EXT="$HOME\/.pi\/agent\/extensions\/shadow-git.ts" # Single agent with shadow-git logging tmux new-session -d -s "$AGENT" \ "cd $WORKSPACE\/agents\/$AGENT && \ PI_WORKSPACE_ROOT='$WORKSPACE' \ PI_AGENT_NAME='$AGENT' \ pi \ --model claude-haiku-4-5 \ --tools read,bash,browser \ --max-turns 30 \ -e '$EXT' \ 'Read plan.md and execute it.' \ 2>&1 | tee output\/run.log" # Check if running tmux has-session -t "$AGENT" 2>\/dev\/null && echo "Running" || echo "Done" # Attach to observe (Ctrl+B then D to detach) tmux attach -t "$AGENT" # Kill if stuck tmux kill-session -t "$AGENT" ``` **Multiple agents in parallel:** ```bash WORKSPACE="$(pwd)" EXT="$HOME\/.pi\/agent\/extensions\/shadow-git.ts" AGENTS="scout1 scout2 scout3" for agent in $AGENTS; do tmux new-session -d -s "$agent" \ "cd $WORKSPACE\/agents\/$agent && \ PI_WORKSPACE_ROOT='$WORKSPACE' \ PI_AGENT_NAME='$agent' \ pi --model claude-haiku-4-5 --max-turns 30 \ -e '$EXT' 'Read plan.md and execute.' 2>&1 | tee output\/run.log" echo "Spawned: $agent" done # List all sessions tmux ls # Wait for all to complete (poll) while tmux ls 2>\/dev\/null | grep -qE "scout[123]"; do echo "Waiting... $(date)" sleep 30 done echo "All done" ``` ### Method 2: Bash Background with --print (RECOMMENDED for headless) **When to use:** Simple tasks where you don't need to observe progress. Fire-and-forget style. **CRITICAL:** The `--print` flag (or `-p`) is MANDATORY. It disables the TUI and runs non-interactively. ```bash WORKSPACE="$(pwd)" AGENT="scout1" EXT="$HOME\/.pi\/agent\/extensions\/shadow-git.ts" # Single agent (cd $WORKSPACE\/agents\/$AGENT && \ PI_WORKSPACE_ROOT="$WORKSPACE" \ PI_AGENT_NAME="$AGENT" \ pi \ --model claude-haiku-4-5 \ --max-turns 20 \ --print \ -e "$EXT" \ 'Read plan.md and execute.' \ 2>&1 | tee output\/run.log) & echo $! > agents\/$AGENT\/pid # Check if running ps -p $(cat agents\/$AGENT\/pid) >\/dev\/null 2>&1 && echo "Running" || echo "Done" # Wait for completion wait $(cat agents\/$AGENT\/pid) echo "Exit code: $?" ``` ### Method 3: Blocking (Sequential) **When to use:** Agent B needs Agent A's output. ```bash WORKSPACE="$(pwd)" EXT="$HOME\/.pi\/agent\/extensions\/shadow-git.ts" # Agent 1 (blocking - wait for result) cd $WORKSPACE\/agents\/scout PI_WORKSPACE_ROOT="$WORKSPACE" PI_AGENT_NAME="scout" \ pi --model claude-haiku-4-5 --max-turns 20 --print \ -e "$EXT" 'Read plan.md and execute.' > output\/result.txt 2>&1 # Agent 2 (uses Agent 1's output) FINDINGS=$(cat $WORKSPACE\/agents\/scout\/output\/result.txt) cd $WORKSPACE\/agents\/planner PI_WORKSPACE_ROOT="$WORKSPACE" PI_AGENT_NAME="planner" \ pi --model claude-sonnet-4-5 --max-turns 20 --print \ -e "$EXT" "Based on these findings: $FINDINGS - create implementation plan." \ > output\/result.txt 2>&1 ``` --- ## Shadow Git Extension (Audit Trail) The shadow-git extension creates a **per-agent git-based audit trail**. Every turn completion is committed to the agent's isolated git repo. ### Why Use It? | Benefit | How | |---------|-----| | **Audit trail** | `git log` shows all agent turns | | **Rollback** | `\/shadow-git rollback ` to go back | | **Branching** | `\/shadow-git branch ` to fork execution | | **Zero lock conflicts** | Each agent has own `.git` | | **Structured queries** | `jq` on audit.jsonl for analytics | | **Killswitch** | `\/shadow-git disable` or env var to stop | | **Fail-open** | Git errors logged but don't block agent | ### Installation **Option A: Pi Package (Recommended)** ```bash # Install globally - includes extension AND this skill pi install git:github.com\/EmZod\/pi-subagent-with-logging # Or install for project only pi install -l git:github.com\/EmZod\/pi-subagent-with-logging # Or try without installing pi -e git:github.com\/EmZod\/pi-subagent-with-logging ``` Once installed, the extension is automatically loaded. No `-e` flag needed. **Option B: Manual Installation** ```bash # Clone the repo git clone https:\/\/github.com\/EmZod\/pi-subagent-with-logging.git # Copy to global extensions mkdir -p ~\/.pi\/agent\/extensions cp pi-subagent-with-logging\/extensions\/shadow-git.ts ~\/.pi\/agent\/extensions\/ cp pi-subagent-with-logging\/extensions\/lib\/mission-control.ts ~\/.pi\/agent\/extensions\/ ``` > **Note on `-e` flag:** Examples below show `-e "$EXT"` for manual installation. > If you installed via `pi install`, the extension auto-loads and you can omit the `-e` flag entirely. ### Usage **Environment variables:** | Variable | Required | Description | |----------|----------|-------------| | `PI_WORKSPACE_ROOT` | Yes | Absolute path to workspace | | `PI_AGENT_NAME` | Yes | Agent name (used in commits and paths) | | `PI_TARGET_REPOS` | No | Comma-separated paths to track external repos | | `PI_TARGET_BRANCH` | No | Branch name to include in commits | | `PI_SHADOW_GIT_DISABLED` | No | Set to `1` to disable (killswitch) | **Spawning with shadow-git:** ```bash WORKSPACE="$(pwd)" AGENT="scout1" EXT="$HOME\/.pi\/agent\/extensions\/shadow-git.ts" # Option A: tmux (for observability) tmux new-session -d -s "$AGENT" \ "cd $WORKSPACE\/agents\/$AGENT && \ PI_WORKSPACE_ROOT='$WORKSPACE' \ PI_AGENT_NAME='$AGENT' \ pi \ --model claude-haiku-4-5 \ --max-turns 30 \ -e '$EXT' \ 'Read plan.md and execute.' \ 2>&1 | tee output\/run.log" # Option B: Background with --print (headless) (cd $WORKSPACE\/agents\/$AGENT && \ PI_WORKSPACE_ROOT="$WORKSPACE" \ PI_AGENT_NAME="$AGENT" \ pi \ --model claude-haiku-4-5 \ --max-turns 30 \ --print \ -e "$EXT" \ 'Read plan.md and execute.' \ 2>&1 | tee output\/run.log) & ``` ### Commands | Command | Description | |---------|-------------| | `\/shadow-git` | Show current status | | `\/shadow-git enable` | Enable logging | | `\/shadow-git disable` | Disable logging (killswitch) | | `\/shadow-git history` | Show last 20 commits | | `\/shadow-git stats` | Show commit\/error counts | | `\/shadow-git rollback ` | Reset to previous turn checkpoint | | `\/shadow-git branch [turn]` | Create branch, optionally from turn | | `\/shadow-git branches` | List all branches | ### Killswitch (Emergency Disable) If logging is causing problems during an incident: **Runtime (no restart):** ``` \/shadow-git disable ``` **Environment (for new agents):** ```bash PI_SHADOW_GIT_DISABLED=1 pi -e shadow-git.ts ... ``` The extension **fails open**: git errors are logged but don't block the agent. ### What Gets Logged **Git commits (turn-level):** ``` 9187798 [scout1:turn-3] no tools 8e0685e [scout1:turn-2] 1 tools b705c6a [scout1:turn-1] 2 tools 95b964d [scout1:start] session began 6780e53 agent initialized ``` **Audit JSONL** (`agents\/scout1\/audit.jsonl`): ```json {"ts":1704567890123,"event":"tool_call","agent":"scout1","turn":3,"tool":"write","input":{...}} {"ts":1704567890456,"event":"tool_result","agent":"scout1","turn":3,"tool":"write","error":false} {"ts":1704567890789,"event":"turn_end","agent":"scout1","turn":3,"toolResultCount":1} ``` ### Querying the Audit Trail ```bash # View agent history (in agent directory) cd agents\/scout1 && git log --oneline # Query audit events jq 'select(.event == "tool_call")' agents\/scout1\/audit.jsonl # Find errors jq 'select(.error == true)' agents\/scout1\/audit.jsonl # Event timeline jq -c '{ts: .ts, event: .event, tool: .tool}' agents\/scout1\/audit.jsonl ``` ### Rollback and Branching ```bash # During execution, use commands: \/shadow-git rollback 2 # Go back to turn 2 \/shadow-git branch fix # Create branch from current state \/shadow-git branch alt 1 # Create branch from turn 1 \/shadow-git branches # List all branches # Or via git directly (in agent directory): cd agents\/scout1 git log --oneline git checkout -b alt-approach abc1234 ``` --- ## Mission Control Dashboard A real-time TUI dashboard for monitoring multiple agents. ### Quick Start ```bash # Set workspace root and start pi with the extension PI_WORKSPACE_ROOT="\/path\/to\/workspace" pi -e ~\/.pi\/agent\/extensions\/shadow-git.ts # Open dashboard \/mc # or \/mission-control ``` ### Features - Real-time status for all agents (running, done, error, pending) - Turn count, tool calls, error count per agent - Auto-refresh every 2 seconds - Scrollable list for 100s of agents - Sort by status, activity, or name - Detail panel for selected agent ### Keyboard Controls | Key | Action | |-----|--------| | `↑\/↓` or `j\/k` | Navigate agents | | `Enter` | Toggle detail panel | | `s` | Cycle sort mode | | `r` | Manual refresh | | `q` or `Esc` | Close dashboard | ### Persistent Widget Shows compact status above the editor while you work: ``` 🚀 Mission Control: ● 2 running │ ○ 1 pending │ ✓ 1 done ``` Toggle with `\/mc-widget` or `Ctrl+Shift+M`. ### Data Sources Mission Control reads: 1. `manifest.json` — Agent registry (status, spawn time, PID) 2. `agents\/*\/audit.jsonl` — Event logs for turn\/tool counts 3. `agents\/*\/state.json` — Checkpoint state --- ## Output Aggregation Patterns ### Pattern A: Orchestrator Reads All (Simple) **Use when:** 3-5 agents, small outputs. ```bash # After all agents complete, orchestrator reads outputs for agent in scout1 scout2 scout3; do echo "=== $agent ===" cat agents\/$agent\/output\/findings.md echo "" done ``` ### Pattern B: Aggregator Subagent **Use when:** Many agents, complex synthesis needed. ```bash # Create aggregator after scouts complete mkdir -p agents\/aggregator\/{workspace,output} cat > agents\/aggregator\/plan.md << 'EOF' # Plan: Synthesize Findings ## Objective Read all scout outputs, produce unified synthesis. ## Input - ..\/scout1\/output\/findings.md - ..\/scout2\/output\/findings.md - ..\/scout3\/output\/findings.md ## Steps 1. Read all findings 2. Identify themes, conflicts, gaps 3. Write output\/synthesis.md ## Output - output\/synthesis.md (executive summary + details) EOF # Spawn aggregator (blocking - we need the result) cd agents\/aggregator PI_WORKSPACE_ROOT="$(pwd)\/..\/.." PI_AGENT_NAME="aggregator" \ pi --model claude-sonnet-4-5 --max-turns 15 --print \ -e ~\/.pi\/agent\/extensions\/shadow-git.ts \ 'Read plan.md and execute.' 2>&1 | tee output\/run.log ``` ### Pattern C: Programmatic Combination **Use when:** Structured outputs (JSON, CSV), deterministic merge. ```bash # Combine JSON jq -s 'add' agents\/*\/output\/data.json > results\/combined.json # Combine markdown for agent in scout1 scout2 scout3; do echo "# $agent" cat agents\/$agent\/output\/findings.md echo "---" done > results\/all-findings.md ``` --- ## Git-Based Coordination ### Git Worktrees (Parallel Isolation) **Use when:** Multiple agents editing same repo, need isolation + merge. ```bash # Setup git worktree add agents\/worker1\/workspace feature\/worker1 git worktree add agents\/worker2\/workspace feature\/worker2 # Spawn agents (they work in isolated copies) for worker in worker1 worker2; do tmux new-session -d -s $worker \ "cd agents\/$worker\/workspace && pi ... 'Make changes per plan.md'" done # After completion, merge git checkout main git merge feature\/worker1 --no-ff -m "Worker 1 changes" git merge feature\/worker2 --no-ff -m "Worker 2 changes" # Cleanup git worktree remove agents\/worker1\/workspace git worktree remove agents\/worker2\/workspace ``` ### Git Branches (Sequential Handoffs) **Use when:** Review gates, audit trail needed. ```bash git checkout -b task\/implement-feature # Stage 1: Research cd agents\/scout && pi ... 'Research and commit findings' git add -A && git commit -m "Scout: research complete" # Stage 2: Plan (builds on scout's commit) cd agents\/planner && pi ... 'Review commits, create plan' git add -A && git commit -m "Planner: implementation plan" # Stage 3: Implement cd agents\/worker && pi ... 'Execute plan' git add -A && git commit -m "Worker: implementation complete" ``` --- ## Logging Protocol **Every subagent MUST maintain an append-only log.** ### Why? 1. **Resumability** — If agent dies, new agent reads log, continues 2. **Debugging** — See exactly what happened 3. **Audit** — Record of all decisions ### Log Format ```markdown # Agent Log: {agent_name} ## Current step_id: STEP-01 status: IN_PROGRESS objective: {what we're doing} --- ## STEP-01: {Title} ### Pre-Execution **Objective**: {what to accomplish} **Target**: {files\/resources} **Assumptions**: {what we believe to be true} ### Execution - Did X - Found Y - Wrote Z ### Post-Execution **Outcome**: PASS | PARTIAL | FAIL **Notes**: {anything important} **STEP-01 COMPLETE** --- ## STEP-02: ... ``` ### Rules 1. **APPEND-ONLY** — Never edit previous entries 2. **FORWARD-ONLY FIXES** — Bug in STEP-03? Create STEP-04 to fix. Never go back. 3. **ATOMIC STEPS** — Pre → Execution → Post, then sealed 4. **LOG UNCERTAINTY** — Don't hide ambiguity, document it --- ## Error Handling and Recovery ### Agent Crashes ```bash # Check if agent is running tmux has-session -t scout1 2>\/dev\/null || echo "Crashed or completed" # Check exit in log tail -20 agents\/scout1\/output\/run.log # Check last step in log.md grep -A5 "STEP-" agents\/scout1\/log.md | tail -20 ``` ### Resuming Failed Agent ```bash # Create continuation agent cat > agents\/scout1-resume\/plan.md << 'EOF' # Plan: Continue scout1 Work ## Context Previous agent crashed. Read ..\/scout1\/log.md to find last completed step. ## Steps 1. Read ..\/scout1\/log.md — find last COMPLETE step 2. Continue from next step 3. Complete remaining work 4. Output to ..\/scout1\/output\/ (same location) EOF tmux new-session -d -s scout1-resume \ "cd agents\/scout1-resume && pi ... 'Read plan.md and continue work.'" ``` ### Using Rollback for Recovery ```bash # If agent made bad changes, rollback to earlier turn # First, attach to agent or use commands file: \/shadow-git rollback 5 # Go back to turn 5 # Or via git: cd agents\/scout1 git log --oneline git reset --hard abc1234 # Reset to good commit ``` --- ## Quick Reference ### Spawning Commands | Task | Command | |------|---------| | Spawn with tmux | `tmux new-session -d -s NAME "cd DIR && PI_WORKSPACE_ROOT=... PI_AGENT_NAME=... pi --model MODEL -e shadow-git.ts ..."` | | Spawn headless | `(cd DIR && PI_WORKSPACE_ROOT=... PI_AGENT_NAME=... pi --model MODEL --print -e shadow-git.ts ...) &` | | Spawn blocking | `cd DIR && PI_WORKSPACE_ROOT=... PI_AGENT_NAME=... pi --model MODEL --print -e shadow-git.ts ...` | ### Process Management | Task | tmux | bash background | |------|------|-----------------| | Check running | `tmux has-session -t NAME` | `ps -p $(cat pid)` | | Wait | Poll with `tmux ls` | `wait $(cat pid)` | | Kill | `tmux kill-session -t NAME` | `kill $(cat pid)` | | Observe | `tmux attach -t NAME` | `tail -f output\/run.log` | | List all | `tmux ls` | `ps aux \| grep pi` | ### Shadow Git Commands | Command | Description | |---------|-------------| | `\/shadow-git` | Show status | | `\/shadow-git enable\/disable` | Toggle logging | | `\/shadow-git history` | Show commits | | `\/shadow-git stats` | Show counts | | `\/shadow-git rollback ` | Reset to turn | | `\/shadow-git branch [turn]` | Create branch | | `\/shadow-git branches` | List branches | | `\/mc` | Open Mission Control | | `\/mc-widget` | Toggle status widget | ### Models | Model | Use For | Cost | Relative | |-------|---------|------|----------| | `claude-haiku-4-5` | Fast research, simple tasks | ~$0.001\/turn | 1x | | `claude-sonnet-4-5` | Complex reasoning, implementation | ~$0.01\/turn | 10x | | `claude-opus-4-5` | Hardest problems, synthesis | ~$0.015\/turn | 15x | **⚠️ WARNING: CLI `--model` flag is IGNORED!** Pi uses `~\/.pi\/agent\/settings.json`. --- ## Gotchas and Hard-Won Lessons ### 1. Use `--print` Not `--print-last` **Wrong:** `pi --print-last ...` (flag doesn't exist!) **Right:** `pi --print ...` or `pi -p ...` The `--print` flag runs pi in non-interactive mode: it processes the prompt and exits without the TUI. ### 2. Use `-e` Not `--hook` **Wrong:** `pi --hook shadow-git.ts ...` (deprecated) **Right:** `pi -e shadow-git.ts ...` ### 3. Set PI_WORKSPACE_ROOT for Mission Control ```bash # Mission Control needs this to find agents PI_WORKSPACE_ROOT=\/path\/to\/workspace pi -e shadow-git.ts # Then: \/mc ``` ### 4. Extension Path Must Be Absolute in tmux ```bash # WRONG: Relative path may fail -e .\/shadow-git.ts # RIGHT: Absolute path -e \/full\/path\/to\/shadow-git.ts # or -e ~\/.pi\/agent\/extensions\/shadow-git.ts ``` ### 5. Env Vars Must Be Set Inside tmux Command ```bash # WRONG: Vars not passed to tmux subshell PI_WORKSPACE_ROOT="$PWD" tmux new-session -d -s agent "pi ..." # RIGHT: Set vars inside the command string tmux new-session -d -s agent "PI_WORKSPACE_ROOT='$PWD' PI_AGENT_NAME='agent' pi ..." ``` ### 6. Check manifest.json for Agent Status ```bash # Quick status check cat workspace\/manifest.json | jq '.agents | to_entries[] | {name: .key, status: .value.status}' ``` --- ## Checklist Before Spawning ``` [ ] Pre-flight decisions documented [ ] Execution mode (blocking\/non-blocking\/parallel) [ ] Environment (shared\/directories\/worktrees) [ ] Aggregation strategy [ ] Workspace setup [ ] agents\/{name}\/ directories created [ ] orchestrator\/decisions.md written [ ] Per-agent setup [ ] plan.md with clear steps [ ] output\/ directory exists [ ] Spawn command ready [ ] TTY decision: tmux OR --print [ ] settings.json configured with correct model [ ] PI_WORKSPACE_ROOT and PI_AGENT_NAME set (for shadow-git) [ ] -e with absolute path to shadow-git.ts [ ] --max-turns set [ ] Monitoring plan [ ] Mission Control ready (\/mc) if using shadow-git [ ] Know how to check status [ ] Know how to kill if stuck ``` --- ## Final Reminder: CLEANUP **Before you finish ANY orchestration session:** ```bash # 1. List what's running tmux ls ps aux | grep -E "[p]i --model" | grep -v grep # 2. Kill everything you spawned tmux kill-server # or kill specific sessions # 3. Verify cleanup tmux ls # Should show "no server running" ``` **You spawned it. You kill it. No exceptions.**