Content revised 20 October 2026.
In SEO, a bot often means a crawler that requests web pages. Crawling, indexing and ranking are different steps, and not every automated request comes from a search engine.
What a crawler does
A crawler discovers URLs and requests accessible resources. Search engines then process information from pages they can access. A successful visit by Googlebot doesn't mean that the page will be indexed or appear in a particular position.
Other bots have different jobs
A monitoring service may check whether a page responds. A feed reader may fetch updates. Scrapers and unwanted automation can behave differently again. When investigating a traffic spike, examine the actual requests instead of counting every automated visit as a potential customer.
Check access without relying on the name alone
- Review server responses for the affected URLs.
- Check whether necessary CSS and JavaScript can load.
- Use appropriate verification when identifying Googlebot; a user-agent string alone can be copied.
- Compare server evidence with Search Console’s live inspection information.
Make the page understandable
Keep important content and links available to ordinary visitors as well as crawlers. If server protection blocks a legitimate fetch, investigate the rule rather than serving a special keyword-filled page to the bot. The aim is a reliable website that people and search systems can access consistently.
Official guidance: Google’s overview of search discovery.