# baoyu-url-to-markdown

> Fetch any URL (including JS-heavy or login-gated pages) and convert to clean markdown. Uses Chrome CDP for high-fidelity rendering.

- Author: Shawn Shi
- Repository: shawnshi/shawnshi-skills
- Version: 20260210090013
- Stars: 0
- Forks: 0
- Last Updated: 2026-02-10
- Source: https://github.com/shawnshi/shawnshi-skills
- Web: https://mule.run/skillshub/@@shawnshi/shawnshi-skills~baoyu-url-to-markdown:20260210090013

---

---
name: baoyu-url-to-markdown
description: Fetch any URL (including JS-heavy or login-gated pages) and convert to clean markdown. Uses Chrome CDP for high-fidelity rendering.
---

# Web Content Miner (CDP Engine)

High-fidelity web scraper that converts HTML to Markdown. Supports JavaScript rendering, lazy loading, and login sessions.

## Core Capabilities
*   **CDP Rendering**: Uses Chrome DevTools Protocol to capture fully rendered DOM.
*   **Login Support**: `--wait` mode allows manual login before capture.
*   **Smart Parsing**: Auto-cleans navigation/ads, preserving main content.

## Workflow SOP

### 1. Standard Capture (Public Pages)
For news, blogs, or documentation:
```bash
npx -y bun C:\Users\shich\.gemini\skills\baoyu-url-to-markdown\scripts\main.ts "<URL>"
```

### 2. Login-Gated Capture (Private Pages)
For Twitter, Substack, or internal dashboards:
```bash
npx -y bun C:\Users\shich\.gemini\skills\baoyu-url-to-markdown\scripts\main.ts "<URL>" --wait
```
**Protocol**:
1.  Browser window will open.
2.  **Manually** log in or navigate to the target state.
3.  Return to terminal and press **Enter** to trigger capture.

### 3. Custom Output
Specify filename directly:
```bash
npx -y bun C:\Users\shich\.gemini\skills\baoyu-url-to-markdown\scripts\main.ts "<URL>" -o "docs/article.md"
```

## Troubleshooting
See `references/troubleshooting.md` for Chrome path configuration and timeout adjustments.