Crawler.sh is a web crawling tool designed to crawl entire websites rapidly while staying within the same domain. It offers configurable concurrency, depth limits, and polite delays between requests, making it suitable for handling thousands of pages efficiently.

The tool performs 16 automated SEO checks on every page, detecting issues such as missing titles, duplicate meta descriptions, noindex directives, thin content, and long URLs. It extracts the main article content from any page and converts it to clean Markdown, including word count, author byline, and excerpt for each page. Results can be exported in multiple formats including NDJSON, JSON arrays, W3C-compliant Sitemap XML, CSV, or human-readable TXT.

Crawler.sh operates as a local-first application, allowing users to crawl websites from their own machine using either a terminal interface or a native desktop app. It streams results as NDJSON during the crawl for real-time processing.

The benefits include SEO auditing to identify issues before they impact rankings, content archiving for backups or migrations, sitemap generation for accurate site mapping, and site monitoring to detect broken links or status code changes. Use cases involve running automated SEO checks, extracting readable content for feeding into other tools, generating updated sitemaps, and regular site health monitoring.

The tool is built for workflows requiring fast website crawling and analysis, targeting users involved in SEO auditing, content management, and site maintenance. It supports integration through export formats compatible with various data processing tools.