# streaming-output

> Stream long-form content to markdown files with resume capability. Writes content incrementally with section markers, enabling recovery if context limits are hit. Use when generating long documents (over 1000 lines), B-SPEC or specification writing, multi-section reports, any task where context compaction may occur mid-generation, or when user explicitly requests streaming output. Commands: init, write, status, resume, finalize, repair.

- Author: Claude Code
- Repository: ddunnock/claude-plugins
- Version: 20260131182317
- Stars: 1
- Forks: 0
- Last Updated: 2026-02-08
- Source: https://github.com/ddunnock/claude-plugins
- Web: https://mule.run/skillshub/@@ddunnock/claude-plugins~streaming-output:20260131182317

---

---
name: streaming-output
description: >
  Stream long-form content to markdown files with resume capability. Writes content incrementally
  with section markers, enabling recovery if context limits are hit. Use when generating long
  documents (over 1000 lines), B-SPEC or specification writing, multi-section reports, any task where
  context compaction may occur mid-generation, or when user explicitly requests streaming output.
  Commands: init, write, status, resume, finalize, repair.
---

# Streaming Output

Write long-form content incrementally to markdown files with automatic resume capability. If context limits are hit mid-generation, work persists and can be continued.

## ⚠️ MANDATORY USE CASES

**ALWAYS use this skill when:**
- Generating B-SPEC documents (typically 1,500-4,000 lines)
- Writing any document expected to exceed 1,000 lines
- Creating multi-section specifications or reports
- Context compaction has already occurred in the conversation
- The continuation prompt mentions streaming-output

**DO NOT use manual heredoc appends (`cat >> file << 'EOF'`) for long documents.** This pattern fails silently when context compaction occurs mid-write, causing content corruption that is difficult to detect and repair.

---

## Commands

| Command | Purpose |
|---------|---------|
| `/stream.init` | Initialize output file with section plan |
| `/stream.write` | Write next section to file |
| `/stream.status` | Show progress, detect resume point, check integrity |
| `/stream.resume` | Continue from last completed section |
| `/stream.repair` | Fix corrupted/partial sections |
| `/stream.finalize` | Strip markers, validate completeness |

---

## Quick Start

```bash
# 1. Initialize with a plan
/stream.init report.md --sections "intro,methodology,findings,conclusion"

# 2. Write sections (repeat for each)
/stream.write intro
/stream.write methodology
# ... if interrupted, resume later with:
/stream.resume

# 3. Finalize when complete
/stream.finalize
```

---

## /stream.init

Initialize an output file with a section plan.

**Usage**: `/stream.init <filepath> --sections "<comma-separated-list>" [--template <template-name>]`

**Workflow**:
1. Create output file (or detect existing)
2. Write header with section plan as YAML frontmatter
3. Generate content hash placeholder for integrity checking
4. Present checklist for tracking

**Templates** (optional):
- `bspec` - 15-section B-SPEC structure
- `report` - Standard report structure
- `spec` - Generic specification structure

**Example**:
```bash
# Standard initialization
python scripts/stream_write.py init report.md \
  --sections "introduction,background,analysis,recommendations,conclusion"

# B-SPEC template (pre-defined 15 sections)
python scripts/stream_write.py init b-spec-010.md --template bspec
```

**Output file structure**:
```markdown
---
stream_plan:
  version: "2.0"
  sections:
    - id: introduction
      status: pending
      hash: null
    - id: background
      status: pending
      hash: null
    - id: analysis
      status: pending
      hash: null
  created: 2024-01-15T10:30:00
  last_modified: null
  integrity_check: true
---

# Report

<!-- Content will be streamed below -->
```

---

## /stream.write

Write a single section to the file with markers and integrity verification.

**Usage**: `/stream.write <section-id>`

**Workflow**:
1. Check section exists in plan and is pending
2. Generate content for section
3. Compute content hash
4. Write to temporary location first
5. Validate write completed (check for SECTION_END marker)
6. Append to main file with `SECTION_START` and `SECTION_END` markers
7. Update section status to `completed` with hash

**Script**:
```bash
python scripts/stream_write.py write report.md introduction "Your content here..."
```

**Markers in file**:
```markdown
<!-- SECTION_START: introduction | hash:a1b2c3d4 -->
## Introduction

Your introduction content here...

<!-- SECTION_END: introduction | hash:a1b2c3d4 -->
```

**Important**: 
- Write ONE section at a time
- Verify success before proceeding
- Hash in START and END markers must match (integrity check)

### Write Verification

After each write, the skill automatically verifies:
1. `SECTION_START` marker exists
2. `SECTION_END` marker exists
3. Hashes in both markers match
4. Content between markers is non-empty

If verification fails, the write is flagged and `/stream.repair` is recommended.

---

## /stream.status

Show current progress, identify resume point, and check integrity.

**Usage**: `/stream.status <filepath> [--verify]`

**Script**:
```bash
python scripts/stream_status.py report.md
python scripts/stream_status.py report.md --verify  # Full integrity check
```

**Standard Output**:
```
Stream Status: report.md

Sections:
  [x] introduction (completed) ✓
  [x] background (completed) ✓
  [ ] analysis (pending) <- RESUME HERE
  [ ] recommendations (pending)
  [ ] conclusion (pending)

Progress: 2/5 sections (40%)
Next section: analysis
```

**With --verify flag (integrity check)**:
```
Stream Status: report.md

Integrity Check:
  [x] introduction - hash:a1b2c3d4 ✓ valid
  [x] background - hash:e5f6g7h8 ✓ valid
  [!] analysis - CORRUPTED (START without END)
  [ ] recommendations - pending
  [ ] conclusion - pending

⚠️  CORRUPTION DETECTED in section: analysis
    Run `/stream.repair analysis` to fix

Progress: 2/5 sections (40%)
Next section: analysis (requires repair)
```

### Corruption Detection

The status command detects:
- **Orphaned START**: `SECTION_START` exists without matching `SECTION_END`
- **Hash mismatch**: START and END marker hashes don't match
- **Empty section**: Markers exist but no content between them
- **Duplicate sections**: Same section ID appears multiple times

---

## /stream.resume

Continue writing from the last incomplete section.

**Usage**: `/stream.resume <filepath>`

**Workflow**:
1. Run status with verification to find resume point
2. Check for corrupted sections (repair if needed)
3. Read existing content for context
4. Continue with `/stream.write` for next pending section

**Script**:
```bash
python scripts/stream_status.py report.md --resume
```

**Output**:
```
Resume Point: report.md

Last completed: background
Next pending: analysis

Context from previous sections loaded (2,450 tokens)
Ready to write: analysis

Command: /stream.write analysis
```

If corruption is detected:
```
Resume Point: report.md

⚠️  CORRUPTION DETECTED
Section 'analysis' has incomplete markers.

Recommended action:
1. Run `/stream.repair analysis` to remove partial content
2. Then run `/stream.write analysis` to regenerate

Command: /stream.repair analysis
```

---

## /stream.repair

Fix corrupted or partial sections.

**Usage**: `/stream.repair <filepath> <section-id> [--strategy <strategy>]`

**Strategies**:
- `remove` (default): Remove partial content, reset section to pending
- `complete`: Attempt to add missing END marker (use with caution)
- `backup`: Create backup before repair

**Workflow**:
1. Create backup of current file (if --strategy backup)
2. Locate corrupted section
3. Remove content from `SECTION_START` to end of partial content
4. Update section status to `pending`
5. Report repair results

**Script**:
```bash
python scripts/stream_repair.py report.md analysis --strategy remove
```

**Output**:
```
Repair Report: report.md

Section: analysis
Issue: Orphaned SECTION_START (no SECTION_END found)
Strategy: remove
Action: Removed 847 characters of partial content

Before: 
  <!-- SECTION_START: analysis | hash:null -->
  ## Analysis
  
  Partial content here...
  [truncated]

After:
  Section 'analysis' reset to pending status

Backup created: report.md.backup.20240115-103045

Ready to regenerate: /stream.write analysis
```

---

## /stream.finalize

Strip markers and validate completeness.

**Usage**: `/stream.finalize <filepath> [--output <output-filepath>]`

**Workflow**:
1. Run full integrity check
2. Verify all sections completed
3. Verify all hashes valid
4. Remove `SECTION_START` and `SECTION_END` markers
5. Remove YAML frontmatter stream metadata
6. Validate no incomplete markers remain
7. Write to output file (or overwrite in place)

**Script**:
```bash
python scripts/stream_cleanup.py report.md --output final_report.md
```

**Pre-finalize validation**:
```
Finalize Check: report.md

Sections:
  [x] introduction ✓
  [x] background ✓
  [x] analysis ✓
  [x] recommendations ✓
  [x] conclusion ✓

All sections complete: YES
All hashes valid: YES
Ready to finalize: YES

Finalizing...
Output written to: final_report.md
Markers removed: 10
Lines in final document: 1,247
```

**If validation fails**:
```
Finalize Check: report.md

⚠️  CANNOT FINALIZE - Issues detected:

  [ ] recommendations - PENDING (not written)
  [!] conclusion - CORRUPTED (hash mismatch)

Fix required before finalization:
1. /stream.write recommendations
2. /stream.repair conclusion
```

---

## Recovery Scenarios

### Scenario 1: Context limit hit mid-section

The section has `SECTION_START` but no `SECTION_END`:
```markdown
<!-- SECTION_START: analysis | hash:null -->
## Analysis

Partial content here...
[truncated - no SECTION_END]
```

**Recovery**:
1. `/stream.status --verify` detects incomplete section
2. `/stream.repair analysis` removes partial content **and preserves it as context**
3. `/stream.status` shows the preserved context file with preview
4. `/stream.write analysis` **continues** from where interrupted (don't restart!)

**Context Preservation**:
When repair runs, incomplete content is saved to a `.context` file:
```
report.analysis.context  # Contains the incomplete content
```

This allows Claude to:
- Review what was being written
- Continue coherently from where interrupted
- Not waste tokens regenerating already-written content

### Scenario 2: Session ended between sections

All written sections have both markers:
```markdown
<!-- SECTION_END: background | hash:e5f6g7h8 -->
```

**Recovery**:
1. `/stream.resume` finds next pending section
2. Continue with `/stream.write`

### Scenario 3: Hash mismatch detected

START and END hashes don't match (content corrupted during write):
```markdown
<!-- SECTION_START: analysis | hash:a1b2c3d4 -->
...
<!-- SECTION_END: analysis | hash:x9y8z7w6 -->
```

**Recovery**:
1. `/stream.status --verify` flags hash mismatch
2. `/stream.repair analysis --strategy remove`
3. `/stream.write analysis` regenerates

### Scenario 4: Context compaction during heredoc append (what to avoid)

**Problem**: Using `cat >> file << 'EOF'` without streaming-output markers:
```bash
# DON'T DO THIS for long documents
cat >> spec.md << 'EOF'
## Section 12
Content...
EOF
```

If context compaction occurs, the heredoc may be split, causing:
- Partial content written
- No markers to detect corruption
- No way to identify resume point

**Prevention**: Always use streaming-output for documents >1000 lines.

---

## B-SPEC Template

For B-SPEC documents, use the built-in template:

```bash
/stream.init b-spec-010-llm-integration.md --template bspec
```

This creates the standard 15-section structure:

```yaml
stream_plan:
  version: "2.0"
  template: bspec
  sections:
    - id: overview
      status: pending
      hash: null
    - id: architecture
      status: pending
      hash: null
    - id: section-03  # Domain-specific
      status: pending
      hash: null
    - id: section-04
      status: pending
      hash: null
    - id: section-05
      status: pending
      hash: null
    - id: section-06
      status: pending
      hash: null
    - id: section-07
      status: pending
      hash: null
    - id: section-08
      status: pending
      hash: null
    - id: section-09
      status: pending
      hash: null
    - id: section-10
      status: pending
      hash: null
    - id: section-11
      status: pending
      hash: null
    - id: database-schema
      status: pending
      hash: null
    - id: api-specification
      status: pending
      hash: null
    - id: implementation-guide
      status: pending
      hash: null
    - id: appendices
      status: pending
      hash: null
```

---

## Workflow Checklist

Copy and track progress:

```
Stream Progress: [document-name]
- [ ] Initialize file with section plan
- [ ] Write each section (one at a time):
  - [ ] Section 1: ___________
  - [ ] Section 2: ___________
  - [ ] Section 3: ___________
  - [ ] ...
- [ ] Run status --verify to check integrity
- [ ] Repair any corrupted sections
- [ ] Finalize to strip markers
```

---

## Context-Aware Resume

When context compaction interrupts a section write, the system preserves your work:

### Automatic Context Preservation

1. **During repair**: Incomplete content is saved to `<file>.<section>.context`
2. **On status check**: Context files are detected and previewed
3. **When resuming**: Review the context file to continue coherently

### Resume Workflow

```bash
# 1. Check status and see context
python scripts/stream_status.py report.md

# Output includes:
# 📝 Context files (from previous incomplete writes):
#    analysis: report.analysis.context (2847 bytes)
#    Preview (last 500 chars):
#    | The normalization formula uses...
#    | This ensures that all criteria...

# 2. Read the full context if needed
cat report.analysis.context

# 3. Continue writing (DON'T restart from scratch)
# Your content should pick up where the context file ended
```

### Important: Continue, Don't Restart

When a context file exists:
- **DO**: Read it, understand where you left off, continue from that point
- **DON'T**: Ignore it and regenerate the entire section from scratch

This saves significant tokens and maintains coherence.

---

## Best Practices

1. **Plan sections upfront**: Define all sections before starting
2. **One section at a time**: Write, verify, then proceed
3. **Check status often**: Especially after interruptions or compaction
4. **Keep sections reasonable**: 500-2000 words per section works well
5. **Save context**: Don't hold generated content in memory; write immediately
6. **Use --verify flag**: Run `/stream.status --verify` after resuming
7. **Never skip repair**: If corruption is detected, repair before continuing
8. **Use templates**: For known document types like B-SPECs, use built-in templates
9. **Review context files**: When resuming, always check for and use `.context` files

---

## Anti-Patterns to Avoid

| Anti-Pattern | Problem | Solution |
|--------------|---------|----------|
| Manual heredoc appends | No corruption detection | Use streaming-output |
| Skipping status checks | Miss corruption | Run status after each write |
| Ignoring hash mismatches | Propagate bad content | Repair immediately |
| Writing multiple sections at once | Can't identify failure point | One section at a time |
| Not using templates | Inconsistent structure | Use bspec template |
| Finalizing without verify | May include corrupted content | Always --verify first |

---

## Integration with Continuation Prompts

When creating continuation prompts for long document generation:

1. **Explicitly state**: "Use streaming-output skill for all document generation"
2. **Include status command**: "Run `/stream.status <file> --verify` to check progress"
3. **Document section plan**: List all sections and their status
4. **Specify resume point**: "Resume from section: X"

Example continuation prompt snippet:
```markdown
## Document Generation Status

Using streaming-output skill for B-SPEC-010.

Current status:
- Sections 1-5: Complete ✓
- Section 6: Pending (resume here)
- Sections 7-15: Pending

Resume command: `/stream.resume b-spec-010.md`
```

---

## Scripts Reference

| Script | Purpose | Usage |
|--------|---------|-------|
| `stream_write.py` | Init and write operations | `python scripts/stream_write.py <command> <args>` |
| `stream_status.py` | Progress, verification, and resume detection | `python scripts/stream_status.py <filepath> [--verify] [--resume]` |
| `stream_repair.py` | Fix corrupted sections | `python scripts/stream_repair.py <filepath> <section> [--strategy]` |
| `stream_cleanup.py` | Finalize and strip markers | `python scripts/stream_cleanup.py <filepath> [--output]` |

Run any script with `--help` for detailed options.

---

## Changelog

### v2.1 (2026-01-03)
- **Context preservation on repair**: Incomplete content now saved to `.context` files
- **Context-aware status**: Status command shows context files with previews
- **Resume guidance**: Clear instructions to continue from context, not restart
- Added "Context-Aware Resume" section with workflow guidance
- Updated recovery scenarios to emphasize context preservation

### v2.0 (2026-01-03)
- Added `/stream.repair` command for fixing corrupted sections
- Added hash-based integrity checking for section markers
- Added `--verify` flag to status command
- Added B-SPEC template (`--template bspec`)
- Added corruption detection (orphaned START, hash mismatch, empty sections)
- Added "Mandatory Use Cases" section
- Added "Anti-Patterns to Avoid" section
- Added "Integration with Continuation Prompts" guidance
- Improved recovery scenarios with specific commands
- Enhanced status output with integrity indicators

### v1.0 (Original)
- Initial streaming output implementation
- Basic section markers
- Status and finalize commands