--- name: agent-eval-harness description: CLI tool for capturing agent trajectories. Execute prompts against headless CLI agents via schema-driven adapters, capture full trajectories (tools, thoughts, plans), and output structured JSONL for downstream scoring. compatibility: Bun >= 1.2.9 --- # Agent Eval Harness ## Purpose CLI tool for capturing trajectories from headless CLI agents, optimized for TypeScript\/JavaScript projects using Bun. **The harness captures. You score.** | Harness Provides | You Provide | |------------------|-------------| | Prompt execution via headless adapters | Scoring logic (Braintrust, custom scripts) | | Full trajectory capture (thoughts, tools, plans) | Pass\/fail determination via graders | | Structured JSONL output | LLM-as-judge prompts | | Reproducible execution environment | CI integration, golden file comparison | **Use this when:** - Capturing trajectories for downstream evaluation - Generating training data (SFT\/DPO) with full context - Building regression test fixtures for agent behavior - Comparing agent responses across configurations ## Installation ```bash # Run without installing (recommended) bunx @plaited\/agent-eval-harness capture prompts.jsonl --schema .\/claude.json -o results.jsonl # Or install as project dependency bun add @plaited\/agent-eval-harness ``` ## Core Principle: Capture Once, Derive Many Views ```mermaid flowchart LR Prompts["prompts.jsonl"] --> Capture["capture\/trials"] Schema["headless schema"] --> Capture Capture --> Results["results.jsonl (full trajectory)"] Results --> Summarize["summarize"] Results --> Calibrate["calibrate"] Results --> Custom["(your tools)"] Summarize --> Views["summary.jsonl \/ .md"] Calibrate --> Report["calibration.md"] Custom --> Pipeline["any scoring platform"] ``` **Single output format:** Full trajectory JSONL (always) **No `--format` flag:** Derive views with separate commands **Schema exports:** Zod schemas + JSON Schema for any tooling ## Commands ### Core Commands | Command | Input | Output | Purpose | |---------|-------|--------|---------| | `capture` | prompts.jsonl + schema | results.jsonl | Trajectory capture (full) | | `trials` | prompts.jsonl + schema | trials.jsonl | Multi-run + optional metrics | | `summarize` | results.jsonl | summary.jsonl or .md | Derive compact views | | `calibrate` | results.jsonl | calibration.md | Sample failures for review | | `validate-refs` | prompts.jsonl | validation.jsonl | Check reference solutions | | `balance` | prompts.jsonl | balance.json | Analyze test set coverage | | `schemas` | (none) | JSON Schema | Export schemas for non-TS users | ### Pipeline Commands (Unix-style) | Command | Input | Output | Purpose | |---------|-------|--------|---------| | `run` | prompts.jsonl + schema | raw.jsonl | Execute prompts, raw output | | `extract` | raw.jsonl + schema | extracted.jsonl | Parse trajectories | | `grade` | extracted.jsonl + grader | graded.jsonl | Apply grader scoring | | `format` | results.jsonl | jsonl\/markdown\/csv | Convert output format | | `compare` | multiple results.jsonl | comparison.json | Compare runs (aggregate report) | All commands support optional `--grader .\/grader.ts` for scoring. ## Capture Command ### Basic Usage ```bash bunx @plaited\/agent-eval-harness capture --schema [options] ``` ### Arguments | Argument\/Flag | Description | Default | |------|-------------|---------| | `prompts.jsonl` | Input file with prompts to execute | Required | | `-s, --schema` | Path to headless adapter schema | Required | | `-o, --output` | Output file\/path | stdout | | `-c, --cwd` | Working directory for agent | current | | `-t, --timeout` | Request timeout in ms | `60000` | | `--progress` | Show progress to stderr | false | | `--append` | Append to output file | false | | `-g, --grader` | Path to grader module | none | | `--debug` | Show detailed CLI output for debugging | false | ### Examples ```bash # Basic capture bunx @plaited\/agent-eval-harness capture prompts.jsonl --schema .\/claude.json -o results.jsonl # Using a local adapter script bunx @plaited\/agent-eval-harness capture prompts.jsonl bun .\/my-adapter.ts -o results.jsonl # With grader (adds score to each result) bunx @plaited\/agent-eval-harness capture prompts.jsonl --schema .\/claude.json --grader .\/grader.ts -o results.jsonl ``` ## Trials Command Run each prompt multiple times for pass@k\/pass^k analysis. ```bash # Capture only (no grader) bunx @plaited\/agent-eval-harness trials prompts.jsonl --schema .\/claude.json -k 5 -o trials.jsonl # With grader (computes pass@k, pass^k) bunx @plaited\/agent-eval-harness trials prompts.jsonl --schema .\/claude.json -k 5 --grader .\/grader.ts -o trials.jsonl ``` ### Output Without grader: ```jsonl {"id":"search-001","input":"Find the CEO","k":5,"trials":[{"trialNum":1,"output":"...","trajectory":[...],"duration":1234},...]} ``` With grader: ```jsonl {"id":"search-001","input":"Find the CEO","k":5,"passRate":0.8,"passAtK":0.99,"passExpK":0.33,"trials":[{"trialNum":1,"output":"...","pass":true,"score":1.0},...]} ``` ## Summarize Command Derive compact views from full trajectory results. ```bash # Summary JSONL (for jq analysis) bunx @plaited\/agent-eval-harness summarize results.jsonl -o summary.jsonl # Markdown (for LLM-as-judge) bunx @plaited\/agent-eval-harness summarize results.jsonl --markdown -o results.md ``` ## Calibrate Command Sample failures for grader review. Calibration helps you distinguish between **agent failures** (agent did wrong thing) and **grader bugs** (agent was correct, grader too strict). ```bash # Sample failures for human review bunx @plaited\/agent-eval-harness calibrate results.jsonl --sample 10 -o calibration.md # Re-score with different grader to compare bunx @plaited\/agent-eval-harness calibrate results.jsonl --grader .\/loose-grader.ts --sample 10 -o comparison.md ``` See [eval-concepts.md](references\/eval-concepts.md#grader-calibration) for why calibration matters. ## Validate-Refs Command Check that reference solutions pass your grader before evaluating agents. ```bash # Validate reference solutions bunx @plaited\/agent-eval-harness validate-refs prompts.jsonl --grader .\/grader.ts -o validation.jsonl # Check for failures cat validation.jsonl | jq 'select(.pass == false)' ``` ### Why Use This? If your reference solution fails your own grader: - The task definition is ambiguous - The grader is too strict - The hint is wrong **Fix the eval before evaluating the agent.** ### Input Format Prompts must include a `reference` field: ```jsonl {"id":"test-001","input":"Create a button component","hint":"