On the recordRead 12 Oct 202614 facts
crawler.sh
Plans and prices
Free
Free
Up to 50 pages per session · SEO analysis with CSV/TXT export · JSON and Sitemap XML export · No account required
Read 12 Oct 2026 · crawler.sh
CLI Pro
$99a year
Up to 10,000 pages per session · NDJSON, JSON, and Sitemap XML output · Markdown content extraction · 16-category SEO analysis with CSV/TXT export
Read 12 Oct 2026 · crawler.sh
Desktop Pro
$99a year
Visual dashboard with 8 interactive cards · Real-time crawl feed · SEO issues panel with per-URL details · Markdown content archive export
Read 12 Oct 2026 · crawler.sh
On the record
- AI content
- It extracts article content as clean Markdown with page metadata such as word count, author byline, language, and excerpt for RAG, fine-tuning, or agent context.crawler.sh · read 12 Oct 2026
- Cloud API
- Crawler Cloud is described as a hosted crawling API with scheduled crawls, webhooks, and a dashboard, marked as coming soon.crawler.sh · read 12 Oct 2026
- Crawl behavior
- The crawler respects robots.txt and adapts per-host pacing after 429 and 403 responses.crawler.sh · read 12 Oct 2026
- Data handling
- The maker says crawl data, results, and exported files stay on the user's computer and that it does not collect crawl targets, results, or extracted website content.crawler.sh · read 12 Oct 2026
- Integrations
- The documentation describes an MCP server for connecting crawler.sh output to tools and lists Claude Code, Claude Desktop, Cursor, OpenCode, and Zed as clients.crawler.sh · read 12 Oct 2026
- Interfaces
- The product includes a CLI for automation and scripting and a desktop app with a visual dashboard; both use the same crawling engine.crawler.sh · read 12 Oct 2026
- JavaScript rendering
- A built-in JavaScript engine renders React, Vue, Next, and similar single-page apps without headless Chrome.crawler.sh · read 12 Oct 2026
- Product
- crawler.sh is a local web crawler that extracts website content, analyzes SEO issues, and exports results in standard formats.crawler.sh · read 12 Oct 2026
- Security
- The privacy policy says device-to-API communication uses HTTPS/TLS and authentication tokens are generated with secure random values and expire after a set period.crawler.sh · read 12 Oct 2026
- SEO checks
- The SEO analysis runs 24 automated checks on successful HTML pages and can export findings as CSV or TXT.crawler.sh · read 12 Oct 2026
- Site audit
- Yescrawler.sh · read 21 Sep 2026
- Support
- Support is offered by email, with typical replies within one business day; the maker says it does not offer live chat, phone support, ticket SLAs, or dedicated account managers.crawler.sh · read 12 Oct 2026
- Windows support
- A native Windows desktop build is in progress, and the maker says its Linux CLI binary can run on Windows through WSL2.crawler.sh · read 12 Oct 2026
