What is a crawl budget?

By the Variask team · Updated

Definition

Crawl budget is how many pages a crawler will fetch from your site in a given time. On large sites, wasting it on duplicate or low value pages can mean important pages are crawled less often by search and AI crawlers.

How it works

  • Crawlers limit how fast and how much they fetch from each site.
  • Fast, healthy servers get crawled more.
  • Duplicate, redirected and broken pages waste fetches.
  • Sitemaps and internal links guide crawlers to what matters.

What it means for brands

  • Most small sites do not need to worry about crawl budget.
  • Large sites should remove crawl waste and keep sitemaps clean.

Questions

Crawl budget:
common questions

Each crawler limits how much it fetches. The same practices that help Googlebot help AI crawlers.

Look at your server logs. The free AI bot log analyzer shows AI crawler visits by page.

Related terms

Keep exploring

Technical

AI crawlers

AI crawlers are bots run by AI companies that visit websites to collect content, either to train models or to fetch pages live when an assistant answers a question.

Read

Technical

Sitemap

A sitemap is an XML file that lists the pages on your site you want crawled, often with last modified dates. It helps search engines and other crawlers find your important pages.

Read

Technical

robots.txt

robots.txt is a text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not visit.

Read
Full glossary