What is an HTTP status code?
A three-digit number returned by an HTTP server that summarizes the result of a request, grouped into 1xx informational, 2xx success, 3xx redirect, 4xx client error, and 5xx server error.
Every HTTP response starts with a status line: HTTP/1.1 200 OK. The number is the machine-readable summary, the reason phrase is decoration. The first digit puts the response in a class (200 successful, 301 permanent redirect, 404 not found, 500 server error) which is enough for most error handling. The remaining two digits add specificity (404 vs 410 vs 451 are all "client problem, the resource is not where you think").
For crawlers, the status code is the primary signal for what to do next. 2xx means parse and continue. 3xx means follow the Location header (or do not, if the redirect chain is suspicious). 4xx means stop and either log or skip; 429 specifically means back off. 5xx means retry with exponential backoff.
A few status codes deserve special handling. 451 ("Unavailable For Legal Reasons") signals legal blocking, often geographic. 418 ("I'm a teapot") is a joke RFC; some sites return it as a polite "stop scraping me." And servers sometimes return a 200 with an HTML error page, which is why response-body sniffing complements status-code logic.
In the wild
- A crawler treating 429 as "sleep 60 seconds, then retry once with backoff"
- An SEO audit flagging every 404 in an internal link graph
- A monitoring tool paging on-call when the 5xx rate exceeds 1% over 5 minutes