TL;DR
- A cron job, an HTTP request, and a text diff can monitor one static page in an afternoon.
- Production website change detection also needs overlap protection, missed-run recovery, JavaScript rendering, noise filtering, snapshot retention, retries, and reliable notifications.
- DIY monitoring makes sense for a few predictable pages or a highly custom comparison. It becomes expensive when third-party websites, larger URL sets, or business-critical alerts enter the picture.
- Context.dev Monitors turns the stack into one managed API resource. Use page monitors for visible-text changes, sitemap monitors for added or removed URLs, and extract monitors for meaningful changes across relevant pages.
What Is Website Change Detection?
Website change detection is the process of capturing a webpage or set of URLs on a schedule, comparing the new result with a saved baseline, and creating an event when relevant content changes.
Teams use it to track competitor pricing, terms of service, product launches, documentation, job listings, policy updates, and other web content that changes without notice. The comparison can be exact, such as detecting an edited price, or semantic, such as deciding whether a product's positioning meaningfully changed.
The basic idea is simple. Making it reliable across real websites is not.
Why Cron Plus a Scraper Looks Like the Obvious Answer
Suppose you want to know when a competitor changes its pricing page. The first version takes about an afternoon:
from pathlib import Path
import requests
URL = "https://example.com/pricing"
SNAPSHOT = Path("pricing.html")
response = requests.get(URL, timeout=30)
response.raise_for_status()
current = response.text
previous = SNAPSHOT.read_text() if SNAPSHOT.exists() else None
if previous is not None and current != previous:
notify_team(URL)
SNAPSHOT.write_text(current)Put that script in cron, connect notify_team to a Slack webhook, and you have a working proof of concept. It can catch a change in staging, and the whole system fits in a few dozen lines.
It feels finished because the difficult conditions have not appeared yet. The price happens to exist in the initial HTML. The site accepts your requests. One snapshot fits on disk. No run has timed out, overlapped another run, or compared an error page against a valid baseline.
Each assumption holds until it does not. That is when a small website change detector starts becoming production infrastructure.
1. Scheduling Becomes Job Orchestration
Standard cron starts commands on a clock. It does not know whether the previous run finished. If a slow fetch takes longer than the monitoring interval, the next run can start at the same time. Both processes may read and write the same snapshot, producing a race condition in a tool that was supposed to compare one page.
You can add a file lock, but missed runs are a separate problem. If the host is offline during the scheduled window, standard cron does not automatically replay the job. Daylight saving transitions can also skip or duplicate local-time schedules, depending on the cron implementation.
Scaling from one page to hundreds adds another requirement: distribute work instead of sending every request at once. Now you need a queue, per-domain rate limits, run status tracking, and a way to retry failures without creating duplicate alerts.
At that point, the scheduler is no longer just a scheduler. It is an orchestration layer.
2. Fetching Breaks on JavaScript and Bot Protection
A plain HTTP request only sees the response the server returns. If a React, Vue, or other client-rendered page loads its pricing data after JavaScript runs, the initial HTML may contain little more than an app shell and script tags.
Seeing what a user sees requires a browser renderer such as Playwright or Puppeteer. That introduces Chromium processes, browser lifecycle management, memory pressure, navigation timeouts, and new judgment calls:
- How long should the browser wait for asynchronous content?
- Which selector or network state means the page is ready?
- What happens when a cookie banner covers the content?
- How many browser sessions can run concurrently?
- How do you distinguish a real empty page from a failed render?
Third-party sites may also rate-limit automated traffic or place requests behind a web application firewall. Headers, cookies, proxy routing, and browser fingerprints become part of the fetch path. A scraper that worked last month can start returning a challenge page after the target changes its frontend or access rules.
The most dangerous failure is silent. If the fetcher returns an empty shell with a 200 status, a naive monitor may save it as the new baseline, fire a false alert, or miss the next real change.
3. Naive HTML Diffs Cry Wolf
Raw HTML changes constantly, even when the useful content stays the same. Common sources of noise include:
- Rendered timestamps
- Rotating ads and testimonials
- Session identifiers and CSRF tokens
- Cache-busting query parameters
- Analytics attributes
- Randomized element IDs
- A/B test variants
- Whitespace and attribute-order changes
A byte-for-byte comparison treats all of those as meaningful. If every scheduled run creates an alert, people mute the channel and stop trusting the monitor.
Filtering the noise is where much of the real engineering work lives. A reliable comparison layer needs to extract visible content, normalize harmless formatting, ignore volatile regions, and avoid replacing a healthy baseline with a failed fetch. Rules written for one site often break when that site is redesigned.
There is also a more important question: what should "changed" mean for this target?
For a pricing page or legal policy, a single edited number or sentence can matter. For competitive intelligence across a whole site, layout changes and paraphrased copy may be irrelevant. Those use cases need different detection strategies.
4. Snapshots Need Storage and Retention
Every comparison needs a trustworthy baseline. If you also want an audit trail, debugging context, or a human-readable before-and-after view, you need more than the latest file.
One hundred pages checked hourly produce 2,400 captures per day and about 72,000 in a 30-day month. Raw HTML, rendered text, screenshots, and diff metadata can make each capture much larger than expected.
Storage therefore needs:
- Atomic writes so a process never reads a partial snapshot
- Versioning so a bad run does not destroy the last healthy baseline
- Retention rules to control growth
- Metadata that connects each snapshot to a run
- Access controls if monitored pages contain sensitive data
Without those controls, a full disk, corrupted write, or premature cleanup can make future comparisons impossible.
5. Retries and Alerts Become Their Own Systems
Fetches fail for reasons unrelated to content changes. A timeout, temporary 503 response, DNS failure, or rate limit should not be compared with the last valid page. The monitor needs to classify the failure, retry transient errors with exponential backoff and jitter, and decide when a persistent failure deserves its own alert.
Retries also need idempotency. If the same successful run executes twice, it should not overwrite a newer baseline, consume work twice, or send duplicate notifications.
Alerting starts as one webhook and quickly grows into routing logic:
- Which changes go to Slack, email, or an internal event bus?
- Who owns each monitor?
- Should low-confidence changes be suppressed?
- How are duplicate events deduplicated?
- What context does a person need to understand the change?
- What happens when the notification endpoint is down?
A raw diff is rarely enough. Useful alerts include the monitored URL, the before and after content, a short summary, detection time, and enough metadata for downstream systems to process the event safely.
Exact vs Semantic Website Change Detection
Exact and semantic detection solve different problems. Choosing the right one is more important than tuning a universal sensitivity threshold.
| Monitor type | What it detects | Best for |
|---|---|---|
| Page with exact detection | Visible-text changes on one URL, with whitespace normalized by default | Pricing, terms, policies, documentation, and changelogs |
| Sitemap with exact detection | URLs added to or removed from a sitemap | New product pages, blog posts, job listings, and removed content |
| Extract with semantic detection | Meaning-level changes across relevant pages, judged against your instructions | Competitor intelligence, product positioning, packaging, and other site-wide questions |
In Context.dev, exact page detection means a visible-text diff, not a comparison of raw HTML bytes. That distinction removes much of the markup noise while preserving literal content changes.
Semantic monitoring goes further. You describe what matters in natural language, and an extract monitor discovers and tracks the relevant pages. Confirmed content changes are judged against those instructions, so cosmetic edits and paraphrase-only changes can stay quiet while a meaningful product or pricing update creates an event.
DIY Website Monitoring vs a Managed API
Here is what the build-versus-buy decision looks like once the full system is visible.
| Dimension | DIY website change detector | Managed monitoring API |
|---|---|---|
| Scheduling | Cron, locks, missed-run recovery, queues, and per-domain throttling | Provide an interval; scheduling and concurrency run in the service |
| Page fetching | HTTP clients, browser rendering, timeouts, cookies, and access handling | Managed crawling and rendered-page retrieval |
| Change detection | Custom parsers, normalization rules, selectors, and semantic judging | Exact page diffs, sitemap URL changes, or instruction-guided semantic detection |
| Storage | Snapshot database or object storage plus retention and cleanup | Managed baselines plus run and change history |
| Failure handling | Retry policy, backoff, health checks, and idempotency | Background retries and run status managed by the service |
| Notifications | Custom Slack, email, webhook, and deduplication code | Signed webhooks or changes retrieved through the API |
| Maintenance | Every layer can fail independently | One API contract to integrate and monitor |
The left column offers maximum control. It also makes your team responsible for every edge case. The right column trades that infrastructure ownership for a managed dependency.
How Context.dev Monitors Replaces the Stack
Context.dev Monitors lets you define what to watch, how to compare it, how often to run, and where to send events in one API request.
This example checks a pricing page every six hours:
curl -X POST https://api.context.dev/v1/monitors \
-H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Competitor pricing",
"target": {
"type": "page",
"url": "https://example.com/pricing"
},
"change_detection": {
"type": "exact"
},
"schedule": {
"type": "interval",
"frequency": 6,
"unit": "hours"
},
"webhook": {
"url": "https://your-app.com/webhooks/context"
}
}'Creating the monitor triggers an immediate baseline run. Future runs compare against that baseline on the interval you chose, from every 10 minutes up to once per year.
When a change is detected, you can retrieve it from the API or receive a signed webhook. The response includes a webhook signing secret, which lets your application verify the HMAC-SHA256 signature before acting on an event.
The managed service handles the crawling, diffing, semantic judging, scheduling, and retrying in the background. Your application keeps the part specific to your product: what to do after a relevant change arrives.
When DIY Website Change Detection Is the Right Choice
A managed API is not automatically the right answer. Building your own monitor can be reasonable when:
- You watch one or two pages that you control
- The content is server-rendered and structurally predictable
- A missed or delayed alert has little business impact
- Your comparison logic is highly specialized
- The project is a prototype or a way to learn scraping fundamentals
Even a small internal monitor should use a run lock, validate response status and content, write snapshots atomically, preserve the last healthy baseline, and alert when the monitor itself stops running.
DIY also makes sense when monitored content sits behind authentication or inside a private network that a third-party service cannot access. In that case, infrastructure ownership may be a requirement rather than a preference.
When to Use a Website Monitoring API
A managed website monitoring API becomes the stronger option when:
- You monitor third-party or JavaScript-heavy websites
- The URL count is growing beyond a handful
- False positives are causing alert fatigue
- You need semantic change detection
- Changes feed a production agent, workflow, or data pipeline
- Run history and signed webhooks matter
- The team would rather invest engineering time in the action after detection
The clearest signal is operational ownership. If the script needs dashboards, an on-call runbook, regular selector fixes, and someone responsible for missed alerts, it is already infrastructure.
Teams routinely use managed queues, logging platforms, and error trackers because those systems contain more edge cases than their first versions suggest. Website change detection belongs in the same category once it becomes part of a production workflow.
The Bottom Line
A cron job and a diff are a good prototype for website change detection. They are not the full production system.
The real stack includes scheduling, concurrency control, browser rendering, access handling, content normalization, semantic judgment, snapshot retention, retries, idempotency, event delivery, and observability. You can build those layers when the control is worth the cost.
If your goal is to react to web changes rather than maintain the machinery that finds them, use a managed monitor. Create a Context.dev monitor, choose exact or semantic detection for the target, and keep your engineering effort focused on what happens next.
FAQs
What is the best way to monitor a website for changes?
For one predictable page, a scheduled fetch and visible-text comparison may be enough. For third-party sites, multiple URLs, or production alerts, use a managed website monitoring API that handles rendering, baselines, retries, and event delivery.
What is the difference between exact and semantic change detection?
Exact detection reports literal visible-text changes on a page or URL additions and removals in a sitemap. Semantic detection evaluates whether changed content matters to the instructions you supplied. Use exact detection when every wording or price change matters, and semantic detection when you want to suppress cosmetic or irrelevant edits.
Can website change detection monitor JavaScript-rendered pages?
Yes, but a plain HTTP client may not see content loaded in the browser. A DIY system needs a browser renderer such as Playwright or Puppeteer. A managed monitoring API can put rendered-page retrieval behind the monitoring request.
How is website change monitoring different from uptime monitoring?
Uptime monitoring checks whether a page responds and often measures latency or status codes. Website change monitoring checks what the page says or which URLs exist. A page can be online and healthy while its price, policy, or product details have changed.
How often can Context.dev Monitors check a website?
Context.dev supports fixed intervals from every 10 minutes to once per year. Choose the interval based on how quickly you need to react, how often the source changes, and the cost of each run.
Can I monitor only one section of a website?
Yes. A sitemap monitor can include or exclude URL path patterns, such as watching /blog/ while excluding /legal/. An extract monitor can use instructions and page limits to focus semantic monitoring on the content relevant to your question.
How are website change alerts delivered?
Context.dev stores detected changes for retrieval through the API and can send signed webhooks for change.detected, run.completed, or both. Your webhook handler can then route verified events to Slack, email, a database, or another workflow.