Crawling Basics & Settings

Starting Your First Crawl

In CheapCrawler, enter any root domain (e.g. https://example.com) and click the primary Start Crawl button. The local crawler streams discovered links directly into an embedded SQLite database.

Configuring Concurrency & Delays

Under Project Settings → Speed, adjust concurrent thread count:

• •Shared Web Hosting: Set concurrency to 2-4 threads with a 250ms request delay to avoid triggering 429 rate limits.
• •High-Performance Static / Jamstack: Set concurrency to 16-32 threads to crawl 200+ URLs per second.

User-Agents & Robots.txt

Choose between Googlebot Smartphone, Googlebot Desktop, or custom header strings. CheapCrawler automatically parses robots.txt, but you can toggle an override mode to audit pre-launch staging sites.