SKILL.md
Preferred entry: call exports.py (don't hand-roll requests)
Ready-made helpers live in skills/web-crawler/exports.py. Prefer them over
writing your own proxied_get/proxied_post calls — they already inject the
proxy credentials, so there is no API key to find (don't read
$SCRAPECREATORS_API_KEY / $FIRECRAWL_API_KEY, don't check .env, don't ask
the user).
import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article") # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID") # ScrapeCreators
youtube_video("https://youtube.com/watch?v=ID") # metadata: title/description/uploader (+ transcript)
sc_get("/v1/tiktok/profile", handle="charlidamelio") # any SC endpoint
from exports import archive_fallback # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html") # archive snapshot