- Home
- SEO Crawler Software
- Crowl
On the recordRead 9 Oct 202623 facts
Crowl
On the record
- Additional crawl features
- Release notes list exclusion patterns, proxies, rotating user-agents, HTTP authentication, request limits, duplicate-link removal and custom XPath extractors among Crowl features.crowl.tech · read 9 Oct 2026
- Authentication
- Crowl supports HTTP basic authentication settings for crawling environments that require a username and password.crowl.tech · read 9 Oct 2026
- Crawl control
- Crowl can stop and resume crawls.crowl.tech · read 9 Oct 2026
- Crawl controls
- Crowl supports stopping and resuming crawls and allows a custom user-agent.crowl.tech · read 9 Oct 2026
- Crawl data
- Crowl extracts links and anchor texts, and can extract page content.crowl.tech · read 9 Oct 2026
- Customization
- The maker describes Crowl as customizable and says crawl settings are configured in project-specific INI files.crowl.tech · read 9 Oct 2026
- Exports
- Crowl provides CSV export by default and supports storing crawl data in MySQL.crowl.tech · read 9 Oct 2026
- Exports and storage
- The maker lists CSV export and MySQL storage as built-in output options.crowl.tech · read 9 Oct 2026
- Integrations
- The documentation describes CSV files and MySQL as output options; it does not state integrations with third-party SEO services.crowl.tech · read 9 Oct 2026
- Intended users
- The maker describes Crowl as designed for SEO professionals and invites contributors to join its volunteer community.crowl.tech · read 9 Oct 2026
- Language detection
- Crowl can identify the language of crawled page content using a FastText model, which must be downloaded separately.crowl.tech · read 9 Oct 2026
- License
- Crowl is distributed under GNU GPL v3, which allows private or commercial use provided modified or redistributed code is shared under the same license.crowl.tech · read 9 Oct 2026
- Maintenance
- The release notes list version 0.3.1 dated July 27, 2021, as the latest release shown there.crowl.tech · read 9 Oct 2026
- Platform
- Crowl works with Windows, Mac OS, and Linux; its installation guide says it works best on UNIX-like systems, including Linux and macOS.crowl.tech · read 9 Oct 2026
- Product
- Crowl is an open-source SEO crawler made by SEOs for SEOs.crowl.tech · read 9 Oct 2026
- Request settings
- Configuration includes custom user agents, optional proxies, robots.txt compliance, request delays, and concurrent request settings.crowl.tech · read 9 Oct 2026
- Support
- The maker provides a contact form for questions and suggestions and links to Buy Me a Coffee for voluntary project support.crowl.tech · read 9 Oct 2026
- Support and community
- The project invites community contributions, provides a contact form, and links to optional supporter funding.crowl.tech · read 9 Oct 2026
