# fetch-web-docs

> Fetch web documentation and convert to clean Markdown using readability + turndown.
Use when: need to download documentation from web URLs as markdown files.
Perfect for API docs, tutorials, technical articles.

- Author: Petro Torbynskyy
- Repository: ptorbynskyy/claude-code-utils
- Version: 20260126132734
- Stars: 1
- Forks: 0
- Last Updated: 2026-02-06
- Source: https://github.com/ptorbynskyy/claude-code-utils
- Web: https://mule.run/skillshub/@@ptorbynskyy/claude-code-utils~fetch-web-docs:20260126132734

---

---
name: fetch-web-docs
description: |
  Fetch web documentation and convert to clean Markdown using readability + turndown.
  Use when: need to download documentation from web URLs as markdown files.
  Perfect for API docs, tutorials, technical articles.
context: fork
---

# Fetch Web Documentation

Fetches web documentation and converts to clean, readable Markdown files using Mozilla's Readability algorithm combined with Turndown. Tested best-in-class: removes navigation/ads, preserves code blocks and structure.

## When to Use

✅ **USE** for:
- API documentation (Bun, Telegram, OpenAI, etc.)
- Technical articles and tutorials
- Framework guides and references
- Static documentation websites

❌ **DON'T USE** for:
- Dynamic content loaded by JavaScript
- Authentication-required pages
- Content behind paywalls
- Interactive demos or file downloads

**Alternatives:**
- Simple HTML: Use `WebFetch` tool
- Web search: Use `WebSearch` tool
- Raw HTML: Use `curl` via Bash

## How It Works

1. Fetches HTML from URL
2. Extracts main content using Readability algorithm (Firefox Reader Mode)
3. Converts to Markdown with Turndown
4. Saves file and returns JSON result

**Stack:** `@mozilla/readability` + `jsdom` + `turndown`

Tested against 5 converters - winner provides best balance of cleanliness, completeness, and 100% success rate.

## Usage

### From Agents/Commands

```markdown
Load /fetch-web-docs skill

Fetch https://bun.com/docs/api/http
and save to ai_docs/bun-http.md
```

### Direct CLI

```bash
bun run -i .claude/skills/fetch-web-docs/scripts/fetch-docs.ts https://example.com/docs ai_docs/example.md
```

Output:
```
📥 Fetching https://example.com/docs...
📄 Extracted: "Example Documentation"
✅ Saved (13.84 KB)
```

## Output Format

**Success (JSON to stdout):**
```json
{
  "success": true,
  "path": "ai_docs/bun-http.md",
  "size": 14156,
  "title": "HTTP Server - Bun",
  "url": "https://bun.com/docs/api/http"
}
```

**Failure:**
```json
{
  "success": false,
  "url": "https://example.com/docs",
  "error": "Failed to extract content from URL"
}
```

Human-readable logs output to stderr for visibility.

## Examples

### Single URL

```markdown
Use /fetch-web-docs to fetch https://mastra.ai/docs/workflows/overview
and save to ai_docs/mastra-workflows.md
```

**Result:** Clean markdown with code examples properly formatted, working links (absolute URLs), no navigation/footer.

### Batch Processing

For multiple URLs, use research-agent which calls this skill internally:

```markdown
@research-agent

Fetch these docs:
1. https://bun.com/docs/api/http
2. https://mastra.ai/docs/workflows/overview
3. https://core.telegram.org/bots/api

Save to ai_docs/research/
```

## Integration

Used by:
- `research-agent` - URL fetching with metadata
- `load_ai_docs` command - Batch fetching of managed URLs
- Any agent needing documentation downloads

**Script:** `.claude/skills/fetch-web-docs/scripts/fetch-docs.ts`

**Exit Codes:**
- `0` - Success
- `1` - Failure (fetch/parse/write error)