# Python

> Benefits of skill file approach: - Consistent, structured output format - Better awareness of what other steps do (prevents duplicate work) - Clear guidelines on what to prioritize - Improved source citation and confidence levels

- Author: Sean Borneman
- Repository: WillRoche2587/Tartanhacks-2026
- Version: 20260206223205
- Stars: 1
- Forks: 0
- Last Updated: 2026-02-07
- Source: https://github.com/WillRoche2587/Tartanhacks-2026
- Web: https://mule.run/skillshub/@@WillRoche2587/Tartanhacks-2026~Python:20260206223205

---

# OSINT Intelligence Gathering Skill

## Overview
Advanced Open Source Intelligence (OSINT) gathering tool that combines multiple AI models, web scraping, and social media discovery to build comprehensive profiles of individuals.

## Capabilities

### Primary Functions
1. **Google Search Integration** - Uses ScrapingDog API to bypass bot detection and gather search results
2. **Social Media Discovery** - Sherlock integration to find profiles across 300+ platforms
3. **Multi-Model AI Analysis** - Queries Claude Opus, GPT-4o, and Claude Sonnet for diverse perspectives
4. **Enhanced Web Research** - Dedalus MCP servers with Exa semantic search and Brave Search
5. **Content Scraping** - Extracts and saves text from discovered URLs
6. **Comprehensive Reporting** - Generates detailed reports with all findings

### Technical Stack
- **APIs**: Anthropic Claude, ScrapingDog, Dedalus
- **Tools**: Sherlock username search, BeautifulSoup web scraping
- **AI Models**:
  - Claude Sonnet 4.5 (with web search, extended thinking)
  - Claude Opus 4.5 (via Dedalus)
  - GPT-4o (via Dedalus)
- **MCP Servers**:
  - `tsion/exa` - Semantic search engine
  - `windsor/brave-search-mcp` - Privacy-focused web search

## Workflow

### 8-Step Process

**Step 1: Google Search (ScrapingDog)**
- Searches: `"{name}" {school} linkedin`, `"{name}" {school} github`, `"{name}" {school}`, `"{name}" {school} team`, `"{name}" {school} family`, social media sites
- Returns 50 results per query
- Categorizes: LinkedIn, GitHub, personal websites, social profiles, news articles
- Enhanced: Now includes queries for teammates and family mentions

**Step 2: Sherlock Username Search**
- Searches username variations across 300+ platforms
- Timeout: 5 seconds per platform
- Returns: Profile URLs for found accounts

**Step 3: Claude Direct Analysis**
- Model: Claude Sonnet 4.5
- Features: Extended thinking (20k tokens), web search enabled
- Max output: 32k tokens
- Conducts: Independent web research with sources
- Uses: `CLAUDE_OSINT_SKILL.md` context file for structured research
- Output format: Organized by Career, Academic, Skills, Achievements, Online Presence, Hobbies, Network
- Network section enhanced: Now specifically looks for friends, family, and personal connections

**Step 4: Multi-Model Analysis (Dedalus)**
- Models: Claude Opus 4.5, GPT-4o, Claude Sonnet 4.5
- Task: Analyze existing data (Sherlock + Google results)
- Output: Multiple AI perspectives on the target's digital footprint
- Note: No web search - analysis only

**Step 5: Enhanced Web Research (Dedalus MCP)**
- Model: GPT-4o with MCP servers
- Strategy: 10 specific searches combining `"{name}" AND "{school}"`
- Focus: News, academic papers, GitHub, blogs, awards, social media, teammates, family, friends
- Filtering: Prioritizes results mentioning both name and school together
- Enhanced: Includes queries for team members, collaborators, and family mentions

**Step 6: URL Content Scraping**
- Scrapes all discovered non-PDF URLs
- Rate limit: 1 request/second
- Saves: Individual text files per URL
- Creates: Index file for navigation

**Step 7: Social Connections Analysis**
- Model: GPT-4o via Dedalus
- Analyzes: Scraped content from Step 6 (first 10 files, 2000 chars each)
- Identifies: Friends, family members, teammates, colleagues, collaborators
- Output: Structured list with names, relationship types, context, and source evidence
- Conservative approach: Only reports clearly stated or strongly implied connections

**Step 8: Report Generation**
- Combines all data sources into comprehensive report
- Sections: Google results, Sherlock findings, Claude analysis, multi-model insights, MCP research, social connections
- Format: Timestamped text file with structured sections

## Configuration

### Required Files
- `Python/.env` - API keys (never commit to git)
- `Python/CLAUDE_OSINT_SKILL.md` - Context/instructions for Step 3 Claude research
- `Python/SKILL.md` - This documentation file
- `Python/OSINT.py` - Main script

### Required API Keys (in `.env`)
```
ANTHROPIC_API_KEY=your_key_here
SCRAPINGDOG_API_KEY=your_key_here
DEDALUS_API_KEY=your_key_here
```

### Installation
```bash
pip install -r requirements.txt
```

### Dependencies
- anthropic>=0.40.0
- requests>=2.31.0
- sherlock-project>=0.14.0
- beautifulsoup4>=4.12.0
- dedalus-labs>=0.1.0
- python-dotenv>=1.0.0

## Usage

### Basic Usage
```python
python OSINT.py
```

### Customization
Edit `main()` function to change target:
```python
name = "Target Name"
school = "Associated Institution"
```

## Output Structure

```
osint_results/
├── osint_report_[Name]_[Timestamp].txt  # Main report
└── scraped_pages/
    ├── INDEX.txt  # List of all scraped files
    ├── scraped_[url1].txt
    ├── scraped_[url2].txt
    └── ...
```

## Report Sections

### Section 1: Google Search Results
- LinkedIn profiles
- GitHub repositories
- Personal websites
- Social media profiles
- News articles

### Section 2: Sherlock Results
- Found social media accounts
- Profile URLs
- Platform names

### Section 3: Claude AI Analysis
- Web search findings with sources
- Thinking process included
- Comprehensive analysis

### Section 4: Multi-Model AI Analysis
- Claude Opus perspective
- GPT-4o perspective
- Claude Sonnet perspective
- Pattern analysis and insights

### Section 5: Enhanced Web Research
- MCP server findings (Exa + Brave)
- Semantic search results
- Filtered, relevant sources
- Verification of school affiliation

## Key Features

### Accuracy Improvements
- **Exact phrase matching** - Uses quotes in Google searches
- **School affiliation filtering** - Prioritizes results mentioning both name and school
- **Multi-source verification** - Cross-references findings across tools
- **PDF filtering** - Skips PDF files during content scraping

### Performance Optimizations
- **Parallel AI queries** - Step 4 runs 3 models concurrently
- **Async architecture** - Uses asyncio for non-blocking operations
- **Rate limiting** - Prevents API throttling (1 req/sec for scraping)
- **Result caching** - Deduplicates URLs automatically

### Privacy & Security
- **API keys in .env** - Never committed to version control
- **Public sources only** - Disclaimer in all reports
- **Rate limiting** - Respectful of server resources
- **User agent rotation** - Standard browser headers

## Advanced Prompting

### Step 3 Prompt Strategy (Claude with Skill Context)
Uses `CLAUDE_OSINT_SKILL.md` to provide:
- **System Context**: Understanding of 7-step pipeline and role within it
- **Research Categories**: Career, Academic, Skills, Achievements, Online Presence, Hobbies, Network
- **Search Guidelines**: Specific search query patterns and verification strategies
- **Output Format**: Structured template with confidence levels and source citations
- **Quality Standards**: Name disambiguation, fact vs. inference distinction, source quality assessment

Benefits of skill file approach:
- Consistent, structured output format
- Better awareness of what other steps do (prevents duplicate work)
- Clear guidelines on what to prioritize
- Improved source citation and confidence levels

### Step 4 Prompt Strategy
Analyzes existing data with focus on:
1. Profile summary
2. Online presence types
3. Pattern identification
4. Inferences about background
5. Recommendations for further research

### Step 5 Prompt Strategy
Conducts new research with:
1. Explicit filtering instructions (name AND school)
2. Seven specific search categories
3. Relevance verification requirements
4. Recency preferences (5 years + historical achievements)

## Error Handling

- Sherlock failures: Continues with empty results
- Google API errors: Reports error, continues workflow
- Dedalus errors: Individual model failures don't block others
- Scraping failures: Tracked and reported, doesn't halt process

## Limitations

- **Google Gemini**: Currently disabled due to API 500 errors
- **LinkedIn scraping**: Often blocked, relies on search results
- **Rate limits**: ScrapingDog and Dedalus have usage caps
- **Name disambiguation**: Common names may still have some noise despite filtering

## Best Practices

1. **Specific targets**: Works best with uncommon names or strong school affiliations
2. **Review results**: Always verify AI findings against primary sources
3. **Ethical use**: Only for legitimate research, verification, or security purposes
4. **Data privacy**: Handle generated reports with appropriate security measures

## Future Enhancements

- Add more MCP servers for specialized searches
- Implement result scoring/relevance ranking
- Add LinkedIn-specific authentication for better profile access
- Create interactive web UI for report browsing
- Add export formats (JSON, CSV, PDF)

## License & Disclaimer

This tool gathers only publicly available information from open sources. Users must comply with all applicable laws and regulations regarding privacy and data protection. Use responsibly and ethically.