Crawlr

Crawlr is a website crawler and technical SEO audit tool for finding technical issues, crawl inefficiencies and site architecture problems. It discovers URLs through internal links, sitemaps, canonicals and JavaScript, with controls for robots.txt compliance, rate limiting, include and exclude rules, and crawl depth. Its analysis covers response codes, indexability signals, metadata, headings, word counts, and internal and external links. Results can be explored through interactive graphs, directory trees, 3D site views and content-cluster visualizations. Crawls can run daily, weekly or on custom schedules, with email notifications when they finish. The API can start crawls, check their status, return structured JSON results, access historical data and send webhook events. The Free plan costs nothing and includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history. Starter costs $8 a month, billed yearly, and lists 5,000 tokens per month and five concurrent crawls. Unauthenticated sandbox crawls are limited to 25 pages.

Crawlr combines crawl controls, SEO analysis, visualizations and API access, with a free plan and paid tiers. Consider the sandbox page cap and annual billing on Starter when choosing an option.

Crawlr suits people auditing site structure and technical SEO, from single-domain users to teams and agencies described in its plan information. Its API may also suit users building SEO or AI pipelines.

Overview

Crawlr is a website crawler and technical SEO audit service for diagnosing crawl problems and understanding site structure. It suits SEO practitioners, agencies, and teams that need configurable scans alongside visual exploration or API workflows. Its free plan is a practical entry point, while token quotas and per-crawl page caps determine whether a paid tier can sustain larger workloads.

Key features

Crawlr discovers pages through internal links, sitemaps, canonical references, and JavaScript, with JavaScript rendering supported. Robots.txt compliance, rate limits, include and exclude rules, and crawl-depth settings let users target scans and control how the crawler moves through a site. That makes it useful for focused audits, though those controls do not increase the quota or page ceiling attached to a plan.

The audit checks response codes, indexability signals, metadata, headings, word counts, and internal and external links. Interactive graphs, directory trees, 3D site exploration, and content-cluster visualizations give users several ways to trace architecture and page groupings. This broader view is valuable when an audit needs to explain how a site is organized, not just surface individual issues.

Daily, weekly, or custom scheduled crawls can send email notifications on completion. The API can start crawls, check their status, return structured JSON, retrieve historical data, and emit webhook events. API-key authentication supports automated use; keys should stay out of client-side code, public repositories, and logs. Free provides CSV and JSON exports, while paid plans add LLM-ready Markdown and higher-tier bulk export capabilities.

The unauthenticated sandbox is rate limited and capped at 25 pages, so it is suited to a small initial check rather than a representative crawl of a large site. Google PageSpeed Insights is listed for Starter, and webhook notifications are listed for Growth as a beta integration.

Pricing

Free costs nothing, billed monthly, and includes 500 tokens per month, one concurrent crawl, full-site crawling, SEO intelligence and DIQ score, CSV and JSON exports, and 15-day crawl history. It is a low-commitment way to explore the workflow, but the small monthly token allowance and single crawl slot constrain repeated or overlapping work.

Starter costs $8 a month, billed yearly with two months free. It raises the allowance to 5,000 tokens per month, permits five concurrent crawls, caps each crawl at 5,000 pages, and keeps three months of history. Crawlr describes it as suited to single domains and portfolios; annual billing is the key trade-off for readers who want the lowest paid price.

Standard costs $73 a month, billed yearly with two months free. Its 35,000 monthly tokens, 25 concurrent crawls, 10,000-page crawl cap, and one-year history are a better fit for growing teams and agencies that need more simultaneous work and longer comparisons. Growth costs $291 a month, billed yearly with two months free, and provides 100,000 tokens per month, 100 concurrent crawls, a 25,000-page cap, and one-year history. Its higher throughput and webhook notifications suit high-volume AI and SEO pipelines; the annual commitment and price make it difficult to justify for occasional audits.

Enterprise has custom pricing and includes unlimited pages, custom crawl logic, dedicated resources, and a service-level agreement. Priority support and dedicated support are plan features in the pricing comparison. The stated crawl limit is 5,000 URLs per crawl, so readers should distinguish that general limit from the larger per-crawl page ceilings attached to higher plans when estimating a workload.

Platforms

Crawlr is available on the web and through an API. That combination serves both users who inspect audits directly and teams that want to feed crawl results into automated processes.

Who it's for

Crawlr is a strong fit for SEO specialists and agencies that need adjustable technical audits, visual ways to inspect architecture, and structured results for downstream workflows. Free works for learning the service or limited crawling; Starter is aimed at single domains and portfolios, Standard at growing agency or team workloads, and Growth at high-volume automation. Users who need extensive crawling without annual billing, or who want to work mainly from a local desktop crawler, may find another product a better fit.

Pros and cons

• Pros: URL discovery, JavaScript rendering, and detailed crawl controls support audits across complex sites.
• Pros: Graphs, directory trees, 3D exploration, and content-cluster views help make site structure easier to inspect.
• Pros: The API includes crawl management, structured results, history, and webhook events for automated workflows.
• Cons: Free is limited to 500 tokens per month and one concurrent crawl, which can restrict recurring use.
• Cons: Starter, Standard, and Growth are billed yearly, so the listed monthly rates require an annual commitment.
• Cons: The unauthenticated sandbox stops at 25 pages, limiting its usefulness for evaluating a full site.

Alternatives

Crawlix is worth considering for users who want a Linux or Windows crawler with a free tier capped at 100 pages per crawl, one project, and one device; its Pro plan is $15 a month billed monthly.

CrawlZilla offers API and web access, with a free plan that allows 10,000 URLs per session, 10 requests per second, three Lighthouse audits per crawl, and 20 monitored URLs. It may suit readers whose priority is those published session and monitoring allowances.

Netpeak Spider has a free edition for Windows and macOS with 500 URLs, five threads, and two devices per user, making it an option for desktop-based crawling.

SEO4Ajax combines API, extension, and web platforms; its free Developer Plan includes 1,000 pages, 2,500 snapshots per month, and community support.

Seodar provides API and web access, with a free plan for one site, 200 pages per crawl, 100 one-time credits, and one team member. That defined site and member allowance may suit a small single-site workflow.

A1 Website Analyzer is a Windows and macOS option with a free mode that continues after a 30-day evaluation period.

Crawlie is free and supports API, Linux, macOS, web, and Windows.

CrawlX has API and web access; its free plan includes 500 URLs per crawl, five crawls per month, one project, more than 150 SEO checks, and CSV, JSON, and PDF exports.

For broader comparisons, browse Technical SEO Software, SEO Crawler Software, and SEO tools.

Verdict

Choose Crawlr if you want technical SEO crawling, visual site-structure analysis, and API access in one service, with a free tier to start and paid capacity for larger pipelines. Its controls and visualizations make it particularly compelling for agencies and teams that need to investigate architecture as well as crawl findings. Look elsewhere if the annual billing on paid tiers or the plan-specific token and page ceilings do not fit the scale or cadence of your work.