web-scraper
11
总安装量
10
周安装量
#28153
全站排名
安装命令
npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper
Agent 安装分布
opencode
10
gemini-cli
10
codex
9
github-copilot
8
cursor
8
claude-code
7
Skill 文档
Web Scraper
Extract structured data from websites using BeautifulSoup and requests – turn any webpage into usable data.
When to Use This Skill
- Competitor research – Scrape pricing, features, positioning
- Lead generation – Extract contact info from directories
- Content audit – Pull headings, links, meta data
- Price monitoring – Track competitor pricing changes
- Data collection – Gather research data from multiple sources
What Claude Does vs What You Decide
| Claude Does | You Decide |
|---|---|
| Structures analysis frameworks | Strategic priorities |
| Synthesizes market data | Competitive positioning |
| Identifies opportunities | Resource allocation |
| Creates strategic options | Final strategy selection |
| Suggests implementation approaches | Execution decisions |
Dependencies
pip install beautifulsoup4 requests pandas click lxml
Commands
Scrape Elements
python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"
Extract Links
python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only
Extract Emails
python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2
Extract Structured Data
python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product
Examples
Example 1: Scrape Competitor Pricing
python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"
# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us
Example 2: Extract Article Content
python scripts/main.py structured https://blog.example.com/post --schema article
# Output: article_data.json
# {
# "title": "How to Scale Your Startup",
# "author": "Jane Doe",
# "date": "2024-01-15",
# "content": "...",
# "word_count": 1523
# }
CSS Selector Reference
| Selector | Description | Example |
|---|---|---|
tag |
Element type | h1, p, div |
.class |
Class name | .price, .title |
#id |
Element ID | #main-content |
tag.class |
Tag with class | div.product |
tag[attr] |
Has attribute | a[href] |
parent > child |
Direct child | ul > li |
tag1, tag2 |
Multiple | h1, h2, h3 |
Ethical Scraping Guidelines
- Check robots.txt – Respect site’s scraping policy
- Rate limit – Don’t overload servers (1-2 req/sec)
- Identify yourself – Use descriptive User-Agent
- Cache requests – Don’t re-scrape unchanged pages
- Terms of Service – Check if scraping is allowed
Skill Boundaries
What This Skill Does Well
- Structuring strategic analysis
- Identifying market opportunities
- Creating strategic frameworks
- Synthesizing competitive data
What This Skill Cannot Do
- Replace market research
- Guarantee strategic success
- Know proprietary competitor info
- Make executive decisions
Related Skills
- competitor-monitor – Monitor competitor changes
- pdf-extractor – Extract from PDFs
Skill Metadata
- Mode: centaur
category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week