Crawler.sh

Key Features

• •The custom JavaScript render engine in Crawler.sh handles React, Vue, Next, and other single-page applications without the overhead of headless Chrome. It uses a Chrome 131 TLS fingerprint and shared cookie jar to correctly render session-walled pages. Users can auto-detect rendering per site or force it on or off, making dynamic content scraping seamless.
• •Polite crawling is built into Crawler.sh by default. It respects robots.txt directives including Disallow, Allow, and Crawl-delay. Additionally, it adapts per-host pacing when it receives 429 or 403 responses, using exponential backoff to slow down automatically on protected sites. This ensures ethical data collection and avoids bans.
• •Crawler.sh performs 24 automated SEO checks across every page it crawls. It detects missing titles, duplicate meta descriptions, noindex directives, thin content, broken links, long URLs, content freshness signals, and more. The identified issues can be exported as CSV or TXT, allowing users to fix problems before they impact rankings or dataset quality.
• •The Markdown extraction feature pulls the main article content from any page as clean Markdown, complete with word count, author byline, language, and excerpt. This makes it ideal for RAG pipelines, fine-tuning corpora, or agent context. Bulk export of the entire site as a Markdown archive is supported for offline use.
• •Crawler.sh supports multiple export formats including NDJSON, JSON, CSV, Sitemap XML, and Markdown archive. This flexibility allows users to integrate the output into various workflows such as AI training pipelines, content backups, sitemap generation, and SEO reporting. The default crawl output is in NDJSON format.
• •The local-first design of Crawler.sh means it runs entirely on your own machine without any cloud dependencies. There are no per-page fees, no API quotas, and no external service requirements. This ensures both privacy and cost-effectiveness, especially for large-scale crawling projects.

Use Cases

• •Content Archiving: Extract readable content from any website as clean Markdown. Perfect for backups, migrations, or feeding content into other tools. Outcome: A clean, structured archive of website content in Markdown format, easy to store or transfer.
• •SEO Auditing: Run 24 automated checks across every page to find missing titles, duplicate descriptions, thin content, content freshness signals, and more before they hurt rankings. Outcome: A comprehensive list of SEO issues per URL, exportable as CSV or TXT for fixing.
• •Sitemap Generation: Generate W3C-compliant Sitemap XML from a live crawl. Keep your sitemaps accurate and up to date without manual maintenance. Outcome: An automatically updated sitemap that helps search engines index your site correctly.
• •Site Monitoring: Crawl your site regularly to catch broken links, missing pages, and status code changes before your visitors do. Outcome: Early detection of issues, reducing the risk of a poor user experience and search engine penalties.
• •AI Dataset Preparation: Extract clean Markdown from targeted websites to build high-quality datasets for RAG pipelines, fine-tuning corpora, or agent context. Outcome: Clean text with metadata like word count, author byline, and language, ready for AI processing.

Who is this for?

Crawler.sh is designed for developers and engineering teams who need to scrape web content for AI training, fine-tuning, or agent context. SEO professionals will benefit from the 24 automated checks and issue export. Data scientists can use it to build clean Markdown datasets from multiple sources. Content teams can archive or migrate website content. It is also suitable for DevOps engineers who need automated site monitoring and sitemap generation. The tool is local-first, appealing to users who prioritize privacy and control over cloud dependencies.