- Home
- SEO Log File Analysis Tools
- Log File Bot Traffic Cost Analyzer
Log File Bot Traffic Cost Analyzer
What it is
Log File Bot Traffic Cost Analyzer examines server logs to show AI bot traffic, estimated costs, crawl budget, and possible edge rules. It categorizes traffic using 127 bot signatures and 18 browser patterns, with bandwidth, IP, and TTFB details by category. Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior; heuristic findings are marked low confidence, and optional DoH reverse DNS is listed. Cost estimates use measured bytes and CDN pricing presets, with copyable rules for Cloudflare, Fastly, and AWS. Other modules address crawl traps, rendering gaps, bot-versus-human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies. Inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files. The project describes processing as client-side, with logs remaining on the user's machine; its only stated network request is for a demo file the user chooses to load. Browser analysis under 500 MB is sampled triage; CLI analysis from 500 MB to 50 GB is exact, with BigQuery for 50 GB and above.
For and against
For
- The project says logs stay on the user's machine.
- It supports multiple web, cloud, and compressed log formats.
- CSV and PDF downloads are listed.
- It generates robots.txt, llms.txt, and crawlers.json policies.
Against
- Browser analysis under 500 MB is sampled triage.
- CLI exact analysis is listed for 500 MB to 50 GB.
- BigQuery is listed for 50 GB and above.
- Heuristic bot findings are labeled low confidence.
The verdict
The analyzer combines bot categorization, traffic-cost estimates, and generated edge rules with additional log-analysis modules. Its processing paths differ by log size, and heuristic verification has an explicit low-confidence caveat.
Who it suits
This free, MIT-licensed project suits people doing private log triage, including cases where logs cannot leave the machine. Its documented browser, CLI, and BigQuery paths address different log sizes.
Plans and prices
Free
Browser: under 500 MB triage · CLI: 500 MB–50 GB exact · BigQuery: 50 GB+
Read 4 Oct 2026 · github.com
On the record
- Bot classification
- It lists 127 bot signatures and 18 browser patterns, with bandwidth, IPs, and TTFB by category.github.com · read 4 Oct 2026
- Bot verification
- Verification uses vendor IP data, TLS, cloud ASN heuristics, and behavior, with heuristics labeled low confidence; optional DoH reverse DNS is listed.github.com · read 4 Oct 2026
- Cost and edge rules
- It estimates costs from measured bytes and CDN pricing presets and offers copyable Cloudflare, Fastly, and AWS rules.github.com · read 4 Oct 2026
- Deployment
- The project identifies Cloudflare Pages as its primary deployment and GitHub Pages as a backup, and documents a Docker run command.github.com · read 4 Oct 2026
- Exports
- The README lists CSV and PDF downloads plus generated robots.txt, llms.txt, and crawlers.json policies.github.com · read 4 Oct 2026
- Intended users
- The README positions it for quick private log triage, including cases where DPDP/RBI logs cannot leave the machine.github.com · read 4 Oct 2026
- License and support
- The repository is MIT-licensed, welcomes pull requests, and labels some issues as good first issues.github.com · read 4 Oct 2026
- Log formats
- Supported inputs include JSON, JSONL/NDJSON, Apache Combined, Nginx, W3C Extended, AWS ALB, Cloudflare Logpush, and gzip files.github.com · read 4 Oct 2026
- Network behavior
- The README says it uses no XHR, WebSocket, or sendBeacon, and that the only network request is for a demo file the user clicks to load.github.com · read 4 Oct 2026
- Other features
- Listed modules cover crawl traps, render gaps, bot versus human performance, traffic bursts, 404 clusters, a GSC overlay, and security anomalies.github.com · read 4 Oct 2026
- Private processing
- The project describes itself as 100% client-side and says logs do not leave the user's machine.github.com · read 4 Oct 2026
- Purpose
- Analyzes server logs to break down AI bot traffic and estimated costs, quantify crawl budget, and generate edge rules.github.com · read 4 Oct 2026
- Scale limits
- The README describes browser analysis under 500 MB as sampled triage, CLI analysis from 500 MB to 50 GB as exact, and BigQuery for 50 GB and above.github.com · read 4 Oct 2026
