Every failure mode you learn as an engineer comes with a signal. A stack trace, a non-200, a timeout, a failed assertion. You learn to read the signal and work backward from it, and after a few years the whole discipline feels like pattern matching against things that announced themselves.
Agents that read the web break that habit, because the interesting failures don't announce anything. The fetch succeeds. The schema validates. The model returns confident, well-formed, plausible output. And the answer is useless in a way that no status code will ever tell you about.
This guide uses an earlier extraction run against the Node.js 24.0.0 release notes to show how a successful request can produce a poor answer. The screenshots and multi-page trace are historical; the runnable examples now use Context.dev's unified Scrape API with JSON output and its current response fields. Respan records the inputs, evidence, and output so you can inspect the run.
What you're building: a changelog watcher that flags breaking changes
Upgrades break things, and the warning was almost always in the changelog. Nobody read it, because release notes for a major version are hundreds of lines of commit messages and the three that matter are not marked out from the ninety-three that don't.
So you build a watcher. It reads a release notes page, extracts the breaking changes into structured data, and posts a summary to Slack before anyone runs the upgrade. Small, useful, and the kind of thing a team actually adopts.
Two pieces make it work. Context.dev Scrape takes a URL and a JSON Schema in jsonParams.schema, reads that page, and returns structured fields in json.data, which means you skip the entire scrape-then-parse-then-prompt pipeline. Respan wraps the run in a trace, so every step records its input and output and you can go back and look at what happened rather than guessing.
Without the second piece, a wrong answer is a dead end. You have the output and the input, and nothing in between to explain how one became the other.
Building the changelog watcher with Context.dev
Step 1: Define the schema you want back
Context.dev is a web context API for teams building software and AI agents. Point it at a URL and it handles the crawling, rendering, and parsing, returning clean markdown or structured data matching a schema you define. It ships SDKs for TypeScript, Python, Ruby, Go, and PHP, plus an MCP server.
Use Scrape with formats.json: true for structured fields from a known release-notes page. Include formats.markdown: true in the same request to preserve readable evidence. The base request costs 1 credit, or 2 with browser actions; successful JSON extraction adds 4 credits. Validate the fields with Pydantic before sending a summary.
It's schema-first, which means the schema does most of the work. You describe the shape of the answer, and the descriptions you attach to each field are what tell the model what to look for. Pydantic generates the JSON Schema for you, so the schema and your runtime types stay the same object:
from pydantic import BaseModel, Field
class BreakingChange(BaseModel):
component: str = Field(description="The subsystem, e.g. 'fs', 'tls', 'http'.")
description: str = Field(description="What changed, in one sentence.")
class ReleaseReport(BaseModel):
version: str = Field(description="The released version, e.g. '24.0.0'.")
breaking_changes: list[BreakingChange] = Field(
description=(
"Every breaking change in this release. Node.js marks these with "
"SEMVER-MAJOR in the commit list. Return an empty list only if the "
"release genuinely contains none."
)
)Those description strings are doing more work than the field names are. breaking_changes on its own tells the model nothing; the description is where you say what counts. And nothing in this one defines what a breaking change actually is. It gestures at Node's SEMVER-MAJOR convention and leaves the rest to the model.
Step 2: Extract with unified Scrape
One call, with the schema and a plain-language instruction:
import os
from context.dev import ContextDev
context = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
response = context.web.scrape(
url="https://nodejs.org/en/blog/release/v24.0.0",
formats={"json": True, "markdown": True},
json_params={
"schema": ReleaseReport.model_json_schema(),
"instructions": "Collect every commit marked SEMVER-MAJOR as a breaking change.",
},
)
if response.json_.data is None:
raise RuntimeError("JSON extraction failed")
report = ReleaseReport.model_validate(response.json_.data)That is the current fetch layer: one page, a JSON Schema, and validation of response.json_.data. The screenshot below records the earlier API used for the original experiment; its timing and 10-credit cost are not a benchmark for this updated example.
The current response carries the fields worth tracing:
url: the final URL after redirects.json.data: extracted fields, exposed asjson_.datain Python.markdown.data: readable source evidence from the same request.isPartial: whether capture returned unfinished work (is_partialin Python).request_id: the identifier for diagnosing the API call.
At the REST boundary, check each requested format's success flag. SDK examples below check for non-null data, which also works with published SDK types that do not yet declare that flag. Keep historical crawler statistics separate from current Scrape diagnostics.
Step 3: Post the summary to Slack
Format the structured data into a message and send it:
def format_slack_message(report: dict) -> str:
version = report.get("version", "unknown")
changes = report.get("breaking_changes", [])
if not changes:
return f"*Node.js {version}*\nNo breaking changes in this release."
lines = [f"*Node.js {version}*", f"{len(changes)} breaking changes:"]
for change in changes[:10]:
lines.append(f" - `{change['component']}` {change['description']}")
if len(changes) > 10:
lines.append(f" ...and {len(changes) - 10} more")
return "\n".join(lines)Point it at a Slack webhook, put it on a schedule, and you're done. It works on the first try, which is the problem.
What a useless answer looks like
Here is what the earlier multi-page workflow posted for Node.js 24.0.0:
Node.js 24.0.0
96 breaking changes:
- src enable Float16Array on global object
- src enable explicit resource management
- src,test unregister the isolate after disposal and before freeing
- src use non-deprecated WriteUtf8V2() method
- src use non-deprecated Utf8LengthV2() method
- src use V8-owned CppHeap
- test fix test-fs-write for V8 13.6
- build update list of installed cppgc headers
- tools update V8 gypfiles for 13.6
- tools update V8 gypfiles for 13.5
...and 86 moreRead the entries. update V8 gypfiles for 13.6 is a build system change. fix test-fs-write is a test fix. update list of installed cppgc headers is internal plumbing. None of these will break anyone's application code, and none of them belong in a message whose entire job is telling a team whether it's safe to upgrade.
Node.js 24 does contain real breaking changes. tls.createSecurePair was removed. fs.truncate lost the ability to take a file descriptor. OutgoingMessage._headers is gone. Those are in there somewhere, inside the 86 the message didn't show, ranked no differently from a gypfile update.
Now the part that makes this a debugging problem rather than a bug report. Nothing failed:
- The original extraction returned HTTP 200, as did all six historical calls at their then-current 10-credit price.
- The response validated cleanly against the Pydantic schema.
- Every required field was populated.
- No retries, no timeouts, no exceptions.
numFailed: 0andnumBlocked: 0. Every page the crawler tried, it got.
A dashboard watching this agent shows green. An on-call alert never fires. The only signal that anything is wrong is a human reading the Slack message and finding it useless, and humans stop reading a channel that posts 96 items a week.
Debugging it with Respan
You can't debug this from the Slack message, because the message is the last step and it faithfully rendered what it was given. You need what happened before it.
Step 1: Trace the extraction as part of the run
Respan builds everything on spans, and its decorators group them into one tree per run. Initialize Respan once, then decorate the functions you want to see:
from respan import Respan
from respan.decorators import workflow, task
Respan()
@task(name="fetch_release_notes")
def fetch_release_notes(url: str):
response = context.web.scrape(
url=url,
formats={"json": True, "markdown": True},
json_params={
"schema": ReleaseReport.model_json_schema(),
"instructions": "Collect every commit marked SEMVER-MAJOR as a breaking change.",
},
)
return {
"data": response.json_.data,
"source_url": response.url,
"requested_url": url,
"markdown": response.markdown.data,
"json_returned": response.json_.data is not None,
"is_partial": bool(response.is_partial),
"request_id": response.request_id,
}
traced_format_slack_message = task(name="format_slack_message")(format_slack_message)
@workflow(name="changelog_watcher")
def changelog_watcher(url: str) -> str:
result = fetch_release_notes(url)
report = ReleaseReport.model_validate(result["data"])
return traced_format_slack_message(report.model_dump())One detail worth copying rather than skipping: fetch_release_notes returns a plain dict rather than the SDK's response object. Return the response object and the span records nothing, because the tracer can't serialize it, and you end up debugging with an empty Output panel. Pull the fields you care about into a dict on the way out.
Step 2: Read the data the extraction returned
Within Respan, you can open the changelog_watcher trace and expand fetch_release_notes. The original trace contains the version, the 96-entry array, and crawl statistics. The current Scrape wrapper records the extracted data, final source URL, returned Markdown, partial flag, and request ID instead.
The first thing to look at is not the entries. It's the count. 96 breaking changes in one release is not impossible, but it's implausible enough that it should have been the first thing anyone questioned, and it wasn't, because nothing in the pipeline was set up to have an opinion about plausibility.
As Respan is an observability tool, it gives you insight into exactly what happened, when, and what prompted it.
Step 3: Check which pages it actually read
This is where the historical run stops being a mystery. Its urls_analyzed field listed the following pages. That field belonged to the earlier crawler response; current Scrape returns the single final url:
[
"https://nodejs.org/en/blog/release/v24.0.0",
"https://nodejs.org/blog/release/v26.7.0",
"https://nodejs.org/en/blog/release/v22.15.0",
"https://nodejs.org/en/blog/release/v24.0.1",
"https://nodejs.org/dist/v24.0.0"
]
You asked about v24.0.0. The crawl read v26.7.0, v22.15.0, and v24.0.1 as well, then merged everything it found into one report labelled 24.0.0. Three of the five pages that produced this answer are about different releases.
The crawl statistics on the same span fill in the rest:
pages_analyzed: 5
pages_succeeded: 5
pages_skipped: 55
pages_blocked: 055 skipped is the interesting number. The crawler evaluated sixty pages, decided fifty-five were irrelevant to the schema, and kept five. The relevance filtering worked exactly as designed, and on most jobs that discovery is the reason to use a crawler rather than a page reader. It just meant five pages about Node releases all looked equally relevant to a schema about Node releases, and there was no signal anywhere that "wrong ones" was a possibility.
Step 4: Compare the schema you wrote to the answer you got
The last check is the one that isn't in any tool. Put your schema next to the output and ask whether the output is a correct answer to the question the schema asked.
The schema asked for breaking_changes, described as "every breaking change in this release," with a hint that Node marks them SEMVER-MAJOR. The output is every commit tagged SEMVER-MAJOR across four releases. That is, precisely and literally, what was requested. The extraction did not misunderstand anything.
What made the extraction over-report
Three causes. The first widened what got read, and the other two decided what came back, which is why fixing only the first would still have produced a bad answer.
The earlier workflow followed links across releases
The earlier extraction workflow followed internal links and selected several pages for its answer. That widened the evidence beyond the intended release. Current Scrape is a single-URL operation, so the examples above avoid this discovery step entirely. Use Answers when cross-page research is the intended task.
For lead enrichment, that default is exactly right. You point it at a company homepage and want it to find the about page and the careers page on its own. For a changelog watcher, it's wrong in a specific way, because a release notes page links to every other release notes page, and every one of them is highly relevant to a schema about releases. The crawler did what it was designed to do and the design didn't match the job.
SEMVER-MAJOR does not mean "will break your code"
Node.js tags a commit SEMVER-MAJOR when it can't ship in a minor release. That includes API removals, and it also includes V8 engine upgrades, build system changes, and test infrastructure that depends on them. It's a release engineering marker, not a compatibility warning.
The instruction told the model to treat SEMVER-MAJOR as the definition of a breaking change. It complied. The marker was never the right proxy.
A field named breaking_changes doesn't define breaking_changes
The schema said "every breaking change in this release" and left the rest to inference. Nothing in it says what a breaking change is from the perspective of someone deciding whether to upgrade: a public API that no longer exists, a signature that changed, a default that moved.
A field name is not a specification. When you hand a model a schema, the descriptions are the specification, and vague descriptions produce output that's technically responsive and practically useless.
How to catch the next one automatically
Step 1: Read only the page you meant
Use unified Scrape for the exact release URL, request fresh JSON and Markdown, and define what counts as a breaking change:
response = context.web.scrape(
url="https://nodejs.org/en/blog/release/v24.0.0",
formats={"json": True, "markdown": True},
max_age_ms=0,
json_params={
"schema": ReleaseReport.model_json_schema(),
"instructions": (
"Read only this release notes page. A breaking change is a removed or "
"renamed public API, a changed function signature, or a changed default "
"that requires application code to be updated. Exclude V8 engine "
"upgrades, build system changes, test infrastructure, and internal "
"refactors, even when they are tagged SEMVER-MAJOR. "
"Include only changes explicitly supported by the page."
),
},
)
if response.json_.data is None:
raise RuntimeError("JSON extraction failed")
report = ReleaseReport.model_validate(response.json_.data)The instruction now defines a breaking change and names what to exclude. Scrape has no fact_check option: ask for explicit evidence, keep the Markdown, and verify extracted claims against it. The response shape and a successful request cannot substitute for that check.
Then verify the constraint held rather than assuming it:
assert response.url == "https://nodejs.org/en/blog/release/v24.0.0"
assert response.markdown.data is not None
assert not response.is_partialStep 2: Score the extraction against what a developer would act on
Respan's online evals, which live under Evals then Automations, run an evaluator against live production spans, sample them, and write the score back onto the span. Point one at your fetch_release_notes span with a rubric that encodes the judgment the schema couldn't:
Here is a release report extracted from a changelog:
{{output}}
For each entry, decide whether a developer upgrading to this version would
need to change application code because of it.
Removed or renamed public APIs, changed signatures, and changed defaults
count. Engine upgrades, build tooling, test changes, and internal refactors
do not, even when the project tags them as major.
Return the proportion of entries that would require a code change, and list
any entry you excluded with a one-line reason.A score near 1.0 means the extraction is returning things worth acting on. Our run would have scored close to zero, and it would have scored close to zero on day one, before anyone had a chance to stop reading the channel.
Step 3: Alert when a run returns an implausible count
A model can be wrong in ways a rubric misses. An unusually large count deserves review; also flag redirects, missing evidence, failed extraction, and partial captures:
def sanity_check(result: dict) -> list[str]:
warnings = []
changes = (result["data"] or {}).get("breaking_changes", [])
if len(changes) > 20:
warnings.append(f"Review unusually large change count: {len(changes)}")
if result["source_url"] != result["requested_url"]:
warnings.append(f"URL redirected to {result['source_url']}")
if not result["json_returned"]:
warnings.append("JSON extraction failed")
if not result["markdown"]:
warnings.append("No Markdown evidence returned")
if result["is_partial"]:
warnings.append("Scrape returned partial content")
return warningsReturn those from the traced task and they land on the span alongside everything else the run recorded, so a failed sanity check is visible next to the output that triggered it rather than buried in application logs.
The complementary move is on the fetch side. A Context.dev monitor watching the release notes page on a schedule will tell you when the page itself changes, independently of your agent's opinion about it. Page monitors accept plain-language instructions, so you can tell it to report new breaking changes and removals while ignoring download counts and timestamps, and it judges each diff against that goal rather than firing on every cosmetic edit.
Subscribing to run.completed as well as change.detected is worth doing here. It fires after every run including the ones that found nothing, which gives you a heartbeat proving the watcher is alive. An agent that silently stopped running looks identical to a quiet week.
A debugging checklist for agents that read the web
When an agent that reads the web gives you a bad answer and nothing errored, work through these in order.
- Check the count before the content. Implausible volume is the cheapest signal you have and it takes one glance.
- Read the source actually used. Scrape returns the final
url; Answers returnssources. Compare them with the intended evidence. - Check extraction and evidence. Inspect the requested format results, returned Markdown, and partial marker. Crawl statistics belong to the separate Crawl response.
- Compare the output to the schema, not to your intent. Most of the time the output is a correct answer to the question you actually asked.
- Read your field descriptions as a stranger would. If a description doesn't define its term, the model supplied a definition and you don't know what it was.
- Check defaults. Review cache age, content filters, waits, and timeout behavior for the endpoint you actually call.
- Ask whether the source's own labels mean what you assumed. SEMVER-MAJOR, "deprecated", and "breaking" are project-specific conventions, not standards.
None of this is reachable from a status code, and all of it is reachable from a trace. The instrumentation is one initialization line, two decorators, and a dict, and it's the difference between knowing what your agent did and guessing.
Get a Context.dev API key to build the fetch layer, and try Respan for free to see what your agent actually did.