Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. For most sites it does not matter: Google says that if new pages are usually crawled the day they are published, you do not need to think about it. It becomes a real concern for very large sites, or for sites whose content changes very often.
What decides a site's crawl budget?
Google describes crawl budget as the combination of two things: crawl capacity and crawl demand. Capacity is how much Googlebot can crawl without overloading your server. Demand is how much Google wants to crawl, based on how popular and how fresh your URLs are.
| Factor | What raises it | What lowers it |
|---|---|---|
| Crawl capacity | Fast, stable server responses | Slow responses, timeouts, 5xx errors |
| Crawl demand | Popular pages, frequently updated content | Duplicate, stale or low-value URLs |
Google also notes that crawling is not a ranking factor. More crawling does not mean higher rankings.
Which sites need to worry about crawl budget?
Google's guide to managing crawl budget is written for large sites. It gives rough thresholds rather than hard rules.
The sites it has in mind are:
- Sites with about a million or more unique pages whose content changes around once a week
- Sites with about 10,000 or more unique pages whose content changes daily
- Sites with a large share of URLs stuck as "Discovered - currently not indexed" in Search Console
A coaching institute with 300 course and blog pages is nowhere near these numbers. A large job portal, a marketplace listing thousands of tutors across Indian cities, or a news site may be.
What wastes crawl budget?
Crawl budget is mostly wasted on URLs that should never have existed or should not be crawled.
Common sources:
- Faceted filters that create endless combinations, such as
/courses?city=pune&mode=online&sort=fee - Session IDs and tracking parameters in internal links
- Duplicate pages without a clear canonical
- Long redirect chains, where each hop costs a fetch
- Soft 404 pages that return 200 instead of a real error
- Infinite spaces, such as calendars that link to every future month
- Hacked or spam pages added to the site
How do you see how Google crawls your site?
Open the Crawl Stats report in Google Search Console, under Settings. It shows total crawl requests, average response time and a breakdown by response code, file type and purpose. Look for rising response times, many 5xx errors, or a high share of requests going to redirects or parameter URLs.
Your server logs give the most complete picture. Filter for verified Googlebot requests and check which URLs it fetches most. If it spends much of its time on filters or old URLs, that is crawl budget you can recover.
How do you improve crawl efficiency?
Most fixes reduce the number of useless URLs Google can find and make the server respond faster.
In rough order of impact:
- Block crawling of URLs with no search value, such as filter combinations and internal search, in robots.txt.
- Fix redirect chains so links point straight to the final URL. See redirects and broken links.
- Return real 404 or 410 codes for removed pages.
- Keep your XML sitemap clean, listing only canonical, indexable URLs, with accurate
lastmoddates. - Improve server response time, especially for HTML pages.
- Link internally to important pages so Google sees them as worth crawling.
Does noindex save crawl budget?
No. Google still has to crawl a page to see its noindex tag, so noindex does not reduce crawling. Use robots.txt to stop crawling of URLs you never want fetched, and noindex for pages that must stay crawlable but out of search results.
Can you ask Google to crawl less or more?
You cannot directly buy or request more crawl budget. Google retired the crawl rate limiter setting in Search Console in early 2024. If Googlebot is overloading your server in an emergency, Google's documentation says returning 503 or 429 status codes temporarily slows crawling. Do not do this for more than a day or two, or Google may start dropping URLs.
To get more crawling, make pages faster, more useful and better linked, so demand rises.
Common crawl budget mistakes
- Worrying about crawl budget on a small site, when the real problem is thin or duplicate content.
- Noindexing thousands of filter pages instead of stopping them being crawled.
- Blocking CSS or JavaScript to "save budget", which stops Google rendering pages.
- Listing redirected or noindexed URLs in the sitemap.
How to check this with Spider
A Spider site crawl of up to 2,000 pages shows redirect chains, broken links, duplicate URLs and soft 404s, the usual sources of wasted crawling. Its Google Search Console reports show index status, so you can see whether important pages are being left out.
Check this on your own site
Spider's free check tests this and about 100 other things on any page, and links each problem to a guide like this one.
