What is robots.txt?
A plain-text file at the root of a domain that tells crawlers which paths they are allowed (or not allowed) to fetch.
robots.txt lives at https://example.com/robots.txt and uses the Robots Exclusion Protocol to publish rules per crawler. A typical file lists User-agent: lines naming a bot (or * for all bots), followed by Disallow: and Allow: rules that scope which URL paths that bot may visit.
It is a request, not an enforcement mechanism. Compliant crawlers (Googlebot, Bingbot, GPTBot, Claude-Web, your own well-behaved scraper) read robots.txt and obey it. Adversarial crawlers ignore it entirely. Sites still rely on it because the major search and AI players do follow it, and because it doubles as a clean signal of intent in any legal dispute.
In 2024 the file picked up a new neighbor: llms.txt, which signals which content a site is willing to expose to LLM crawlers and where the canonical Markdown lives. The two files solve different problems, but you will increasingly see them deployed together.
In the wild
User-agent: GPTBot\nDisallow: /, block OpenAI's training crawler from the entire siteSitemap: https://example.com/sitemap.xml, point any crawler at the sitemapUser-agent: *\nDisallow: /admin/, keep every bot out of the admin path