TL;DR
ROBOTS_DISALLOWEDis a WebFetch result. Your local permissions let the call run, and then the site's access rules refused it. Changing local permissions will not fix it.- The error has been reported for x.com, api.fxtwitter.com, and tiktok.com. On November 1, 2026, all three hosts served robots.txt rules that disallow agents in Claude-User's position.
- Before you change anything, find out which layer failed: tool availability, a local permission rule, the domain safety preflight, a robots refusal, a redirect, or lossy extraction.
- To recover, find the primary source the post points to (a repository, release notes, docs, or a publisher blog) and cite that URL. Label search snippets as snippets. If the site is yours, allow
Claude-Userin robots.txt.
What the ROBOTS_DISALLOWED error means
When Claude Code tries to read a link and gets this response, the request reached the retrieval step and was refused there:
{
"error_type": "ROBOTS_DISALLOWED",
"error_message": "Site disallows automated access. Use the search-result snippet, or find an alternative source."
}WebFetch sends a User-Agent header beginning with Claude-User. Anthropic describes Claude-User as its agent for user-initiated requests and says its bots honor robots.txt directives. Robots refusals are also an expected outcome rather than a crash. The Claude Code changelog entry for 2.1.307 says a robots.txt refusal is now "a normal result" in Cowork cloud sessions, "as in local sessions".
The error shows that the site's published policy did not allow this request. It does not prove that the page is private, that the site blocks every AI tool, or that your Claude Code installation is broken.
What prompted this guide
On October 28, 2026, a developer opened anthropics/claude-code issue #105618. They reported that WebFetch had started returning ROBOTS_DISALLOWED for x.com, api.fxtwitter.com, and tiktok.com that day, while GitHub and YouTube still worked. Their environment was Claude Code 2.1.312 with Opus on the Anthropic API, running in Windows Terminal. Their workflow was common: paste a post, ask Claude to read it, follow its links, and check its claims.
As of November 1, 2026, the issue was open with no maintainer reply. It is one report from one environment. It does not establish a global rollout, a version regression, or a Windows-only problem. None of the changelog entries for 2.1.312 through 2.1.316 mention robots.txt.
What robots.txt says on the reported hosts
We fetched each host's robots.txt on November 1, 2026, and checked which group applies to Claude-User.
| Host | Group that applies to Claude-User | Rule |
|---|---|---|
| x.com | User-agent: * (no Claude-User group) | Disallow: / |
| www.tiktok.com | Explicit group naming Claude-User | Disallow: / |
| api.fxtwitter.com | User-agent: * | Disallow: / |
| github.com | User-agent: * (no Claude-User group) | Allows most paths |
These rules are consistent with the reported pattern, but today's file cannot show when it changed. Nothing here shows whether the publishers changed their rules, Anthropic changed enforcement, or both. Proving either one would take dated before-and-after copies of the files or an explanation from a maintainer.
Note that api.fxtwitter.com is a separate host with its own robots.txt. Switching to a mirror domain moves you to another site's policy, and here that policy disallows everything.
Find the failure layer first
Several different failures can look alike in a chat transcript. The agent's explanation is a paraphrase, so read the raw tool result before choosing a fix.
| Symptom | Likely layer | How to confirm | Fix that applies |
|---|---|---|---|
| WebFetch never runs, or the tool is missing | Availability | Run /status and check the Organization policy line, or run claude doctor (docs) | Wait for the policy to load, or ask your admin |
| Claude Code prompts, or a denial names a rule | Local permissions | Check your deny, ask, and allow rules (docs) | Add a scoped WebFetch(domain:...) rule |
| Every fetch fails on a locked-down network | Domain safety preflight | Check whether your network can reach api.anthropic.com (docs) | Allowlist api.anthropic.com |
error_type: ROBOTS_DISALLOWED | Site policy | Read the host's robots.txt for a Claude-User or * group | Find another source, or change robots.txt if you run the site |
| The result names a different URL | Cross-host redirect | WebFetch returns the redirect target instead of following it | Fetch the target, which may have its own policy |
| A deadline error after a long wait | Timeout | The error mentions the five-minute deadline | Retry later, or adjust CLAUDE_CODE_WEBFETCH_DEADLINE_MS |
| The fetch succeeds but the answer is thin | Extraction | WebFetch runs your prompt in a separate model call, so the output is lossy by design | Ask a more specific question, or use curl for the raw page |
Permission rules are evaluated in the order deny, ask, allow. An allow rule only means the call is allowed to run. It cannot make a site answer. A short answer after a successful fetch is an extraction problem, not a robots block.
Run a minimal check
Before you change any settings, collect evidence in a single session:
- Baseline. Ask Claude to fetch a stable public documentation page, such as
https://code.claude.com/docs/en/changelog. If this fails too, the problem is availability, permissions, or the network, not the target site. - Exact URL. Ask Claude to fetch the link that failed and to paste the raw tool result verbatim.
- Robots file. Run
curl -s https://<host>/robots.txtand find the group that matchesClaude-User, or*if no group names it. - Redirects. Note whether the result points at a different host.
- Record the context. Save the timestamp, the URL,
claude --version, your model, your OS, your permission mode, and the complete error.
Do not keep retrying a refused host. Repeating the request will not change the site's policy and only adds noise to the comparison. If you file a report, include the raw result alongside your interpretation and keep the two separate.
Choose a recovery route
If a local rule blocked the call: add a rule scoped to the domain you intend, such as WebFetch(domain:docs.example.com). WebFetch(domain:*) and a bare WebFetch rule both cover every URL, but they behave differently. The domain:* form also opens sandboxed commands to every host. Neither rule overrides a site's robots policy.
If you operate the website: decide whether you want user-initiated agent access. Claude Code 2.1.83 changed WebFetch to identify as Claude-User so site operators can allowlist it in robots.txt. Anthropic lists Claude-User separately from ClaudeBot (training) and Claude-SearchBot (search indexing), so you can allow one without the others:
User-agent: Claude-User
Allow: /docs/
Disallow: /If you only need what the post claims: most posts about a tool or release link to something else. Find the repository, release notes, documentation page, or publisher announcement, and cite that URL. A primary source is usually more reliable than the post anyway.
If the content exists only on that platform: ask the person who shared it to paste the relevant text, or use an official API or integration you are authorized to use. Label pasted text as user-supplied.
If all you have is a search snippet: you can use it to locate a source, but don't present it as the page's content. A snippet is a fragment chosen by a search engine.
Don't fake the user agent, rotate IP addresses, or turn on bypassPermissions to get around a publisher's refusal. Turning off the preflight with skipWebFetchPreflight disables Anthropic's domain safety blocklist check. The documentation does not connect that setting to robots handling, so changing it is not a fix for this error.
Make your agent disclose inaccessible sources
The worst outcome is not the error. It is an agent that reads a snippet, writes a confident summary, and implies that it read the page. Add rules like these to your project's CLAUDE.md:
## Web sources
- If a fetch returns an error such as ROBOTS_DISALLOWED, say so and quote the error_type.
- Never summarize a page you could not read. Label search snippets as snippets.
- When a link cannot be read, look for the primary source it references and cite that URL instead.
- Keep each claim next to the URL it came from.Our guide to building a scraping tool that cites its sources covers the same discipline in code. For failures that return no error at all, see debugging an agent that reads the web.
Worked example: from an unreadable post to a primary source
Say someone shares an X post claiming that the latest Claude Code release added a way to disable the ultrathink keyword, and WebFetch refuses the link. The claim names a product and a release, so the official changelog is the source to check.
Context.dev's MCP server gives Claude Code separate web-search and web-scrape tools for this step. Setup is one command (docs):
claude mcp add --transport http --scope user context https://mcp.context.dev/mcpStart Claude Code, run /mcp, select context, and complete browser sign-in. User scope makes the server available in all of your projects. If your team shares project-scoped configuration, follow Anthropic's MCP scope guidance instead of changing it silently. Our MCP setup guide covers Cursor and Claude as well.
Then ask for the route by name:
Use Context.dev's web-search tool to find the official Claude Code changelog
on code.claude.com or github.com. Then use web-scrape on the best result and
tell me what the newest release added. Quote the version, date, and URL.We ran the same two tool calls through the Context.dev MCP server on November 1, 2026. The search, restricted to github.com and code.claude.com, returned these top results:
| Rank | URL | Relevance |
|---|---|---|
| 1 | https://code.claude.com/docs/en/changelog | high |
| 2 | https://github.com/anthropics/claude-code/releases | high |
| 3 | https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md | high |
Scraping the first result returned the newest entry, 2.1.316, dated October 31, 2026, which begins: "Added CLAUDE_CODE_DISABLE_ULTRATHINK, which turns the ultrathink keyword off." The claim is now verified against a primary source with a URL, and the answer doesn't depend on the post at all.
This is a separate retrieval route with its own behavior and limits. It doesn't repair WebFetch, and one successful page says nothing about coverage of any other site. Use it within each site's terms, and verify what comes back the way you would any fetch.
FAQ
Does ROBOTS_DISALLOWED mean the site blocks every AI agent?
No. It means the site's rules refused this request from this tool. Robots.txt rules are written per user agent, and a result from one tool in one environment says nothing about other providers.
Will allowing WebFetch in my permissions fix it?
Only if a local rule was the problem, and in that case you would see a prompt or a rule-based denial, not ROBOTS_DISALLOWED. Permission rules control whether Claude Code runs the call. They don't control whether the site answers.
Why can my browser open the page when Claude Code cannot?
Your browser is you visiting the site. WebFetch identifies itself as Claude-User, and sites can publish different rules for that agent. A page being public in a browser and being retrievable by a given tool are different questions.
Can I use the search snippet instead?
Use it to find a better source, not as a substitute for the page. If you quote a snippet, label it as one, and don't describe it as something the agent read.
Does installing an MCP server guarantee access to the page?
No. Check four things separately: the server connected, authentication succeeded, the tool ran, and the content it returned answers your question. An HTTP 200, a page title, a login wall, or empty Markdown does not count as reading the page.
Is there a Claude Code version that fixes this?
None has been announced. As of November 1, 2026, issue #105618 had no maintainer reply, and no changelog entry from 2.1.312 through 2.1.316 mentions robots.txt. Check the issue and the changelog again before you assume a change.
What remains unconfirmed
We have not reproduced the original report or compared behavior across Claude Code versions, platforms, or account types. We don't know whether the October 28 change came from publishers, from Anthropic, or from both. This guide reflects the issue, Anthropic's documentation, and the robots.txt files as they stood on November 1, 2026. We will update it if a maintainer explains the change or a release note addresses it.