Crawler.sh

Crawler.sh 介绍

Crawler.sh is a fast, local-first web crawler and SEO analysis tool that enables users to crawl entire websites quickly from the terminal or a desktop app. It performs automated SEO checks, extracts content as clean Markdown, and exports results in multiple formats like JSON, CSV, and Sitemap XML, making it ideal for technical SEO and content management workflows.

核心功能

Concurrent crawling with configurable depth, page limits, and delays

Automated SEO analysis with 16 checks for issues like missing titles and duplicate descriptions

Content extraction to clean Markdown, including word count and author byline

Multiple output formats: NDJSON, JSON, Sitemap XML, and SEO CSV/TXT

Real-time progress tracking with live streaming and dashboard visualization

典型使用场景

SEO professionals conducting technical audits and identifying on-page issues

Content marketers extracting articles for repurposing into blogs or AI training data

Developers automating web scraping tasks for monitoring or data collection

Startups analyzing competitor websites for SEO insights and content gaps

为什么适合初创团队

Startups need crawler.sh because it provides a cost-effective, local solution for conducting SEO audits and content extraction without high subscription fees. Its automation capabilities allow small teams to quickly analyze websites, identify technical issues, and gather content insights, which is crucial for improving online visibility and competing effectively in the digital space.

常见问题

Is crawler.sh free to use?

Yes, crawler.sh is a free tool available for download without requiring an account, offering both CLI and desktop app versions.

What platforms does it support?

It supports macOS (Universal binary for Apple Silicon and Intel), Linux (x86_64 and ARM64 via .deb package), and Windows (desktop app).

How many pages can it crawl by default?

By default, it crawls up to 100 pages, but this limit is configurable based on user settings.

What SEO checks does it perform?

It performs 16 automated SEO checks, including detection of missing titles, duplicate descriptions, noindex directives, thin content, and long URLs.