Developers often talk about Googlebot as if it were a single program you could just run as “googlebot.exe”, but that is not how Google’s crawling actually works. In this episode of Search Off the Record, Martin and Gary from the Search Relations team unpack how Google’s crawling infrastructure is really built and operated. They cover why “Googlebot” is a misnomer and how it relates to a central crawling software-as-a-service used by many Google products, how crawl behavior is controlled centrally to avoid overwhelming sites (throttling, handling 503s, and “don’t break the internet” safeguards) and more! If you build for the web, work on SEO, or just want a more accurate mental model of how Google crawls pages, this behind‑the‑scenes discussion is for you.
Resources: Crawlers → https://developer.google.com/crawling
Episode transcript → https://goo.gle/sotr107-transcript
Listen to more Search Off the Record → https://goo.gle/sotr-yt
Subscribe to Google Search Channel → https://goo.gle/SearchCentral
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast #SEO #GoogleSearch
Speakers: Martin Splitt, Gary Illyes
source
I m getting redirect issues on my website
I am having hard times indexing the site.
Why don't you give us the feature to add site to Google.
Inside Googlebot
Search off the Record
I love Gary's vibe!
It's like my aunt's cooking – delightfully toxic 🙂
Next topic please: HTTP Caching
Jan and Adam (IFYKYK) are going to dissect this.
Can you settle an argument? Is there a separate AI crawler? There are vendors out there selling AI SEO to companies. From what I see, it's basic SEO wrapped in a new title.