Website to JSON: Extract Structured Data from Live Pages

TL;DR

• For a live website-to-JSON workflow, send a known URL to Context.dev’s Scrape endpoint with formats.json and a top-level object schema. Scrape returns schema-shaped JSON without requiring you to convert a saved HTML file.
• On JavaScript-heavy pages, rendering and any configured wait happen before output capture. Choose a wait based on the target page’s content, then check the response rather than assuming the content loaded.
• Scrape can return null for missing fields. Your application still needs to validate extracted values and apply its own business rules.
• Batches processes known URLs asynchronously, but it returns Markdown or HTML, not schema-extracted JSON. Use Scrape JSON per URL when you need that output.

Why website-to-JSON is different from HTML-to-JSON file conversion

A static HTML-to-JSON converter can only parse the markup it receives. On a JavaScript-rendered page, an initial fetch may return a loading container while the browser adds the content later. If a product price appears only after that script runs, converting the initial HTML cannot produce a price field.

For a live website, render the page before extracting its data. Context.dev’s Scrape rendering controls let you wait before output capture when a page loads content asynchronously. Its JSON extraction then uses the captured page’s Markdown, JSON-LD, and meta tags to fill fields you define. Unlike an offline file converter, this workflow retrieves a current page and returns named values for your application to validate.

Getting structured JSON from a live page with Scrape

Context.dev’s Scrape endpoint accepts a live page URL and can return extracted fields under json.data. For a known URL, send a POST request to /v1/web/scrape with JSON enabled and a top-level object schema. The following request uses the fields documented in the JSON extraction guide.

curl https://api.context.dev/v1/web/scrape \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": {"json": true},
    "jsonParams": {
      "schema": {
        "type": "object",
        "properties": {
          "title": {"type": "string"},
          "contact_email": {"type": ["string", "null"]}
        },
        "required": ["title", "contact_email"]
      },
      "instructions": "Use only facts stated on the page."
    }
  }'

Context.dev extracts JSON using page Markdown, JSON-LD, and meta tags. The schema names the fields you want, while instructions constrains what the extractor should use. The guide provides the following abridged illustrative response for https://example.com. We did not run this request or test a JavaScript-rendered target.

{
  "json": {
    "requested": true,
    "success": true,
    "data": {
      "title": "Example Domain",
      "contact_email": null
    }
  }
}

Within json, requested confirms that the call asked for JSON, success reports whether that output succeeded, and data holds the extracted fields. A null email shows how a missing fact can remain missing instead of becoming a guessed value. The full response also includes top-level isPartial, which flags incomplete results, and request_id, which identifies the request for troubleshooting. Check the output’s success value even when the HTTP response is 200, since an individual output can fail.

The serialized schema can be at most 50 KB, and jsonParams.instructions can be at most 2,000 characters. For field design and application-side parsing, use our schema-driven extraction tutorial.

How rendering affects what gets extracted

A JavaScript-heavy page may not contain the data you need when the browser first loads it. Context.dev waits after browser actions and before capturing the page for extraction, so sharedParams.waitFor controls which page state the JSON extractor sees. The rendering documentation lists a default wait of 500 milliseconds, or 2,000 milliseconds when frames or XML are involved. A wait of 0 skips that pause.

Use a selector wait when a specific element signals that the data has loaded. Choose the selector by inspecting the target page’s actual DOM. A selector copied from a generic example may never appear on your page. Use a millisecond wait when you cannot identify a dependable element, but test the duration against the page because a fixed delay cannot confirm that content appeared.

Before you use json.data, check isPartial and json.success in the Scrape response. An HTTP 200 can still contain a failed output. If a required field is missing, inspect the rendered page state and adjust the wait before treating the extracted JSON as complete.

Handling missing data and validating output

A missing field and a failed extraction require different handling. Make fields nullable or optional when the source page may not contain them. For example, if a page has no contact email, a nullable contact_email field can return null without making the JSON output a failure. The Scrape JSON documentation describes how schemas handle missing facts.

Before using a response, check json.success and isPartial. An HTTP 200 can still contain a failed output. In that case, json.data may be null, with error_code and message describing the failure. Your application should handle that record rather than pass it into a validator as extracted data. The Scrape overview documents these output checks.

A valid schema confirms the output’s shape, not the truth of its values. Your application remains responsible for normalizing values, enforcing business rules, and checking consequential facts against the source URL.

HTML vs Markdown vs JSON: choosing the right output format

Choose a Scrape format based on what will consume the page. A website-to-JSON request extracts fields you specify rather than converting every HTML element into a JSON property.

Format	Use case	Structure and parseability	Typical consumer
HTML	Inspect rendered markup or use DOM selectors	Preserves page elements, but requires your own parsing	Browser display or a custom parser
Markdown	Read page content without working through HTML markup	Provides text-oriented content, not fixed data fields	LLM context
JSON	Extract specified facts from a known URL	Returns fields shaped by your object schema, with missing facts potentially null	Downstream data pipeline

Request HTML when element structure matters. Choose Markdown for readable page content, or JSON when your application needs named fields. You still need to validate extracted values against your business rules.

Extracting JSON from many pages at once

For hundreds of known URLs, Context.dev Batches fetches up to 25,000 URLs in an asynchronous job. Each page result contains Markdown or HTML, not schema-extracted JSON. A JSON or NDJSON batch result file is a container for those page results, not a collection of extracted JSON fields.

For schema-shaped records, call Scrape JSON separately for each URL. Limit concurrent calls to your applicable rate limits, retry failures, and check json.success and isPartial before storing a record. Alternatively, use Batches to fetch pages, then transform the Markdown or HTML and validate records in your own code. Scrape JSON cannot take a Batch result as its input, so calling it after a Batch fetches the URL again.

Map can discover URLs before you submit a Batch or schedule Scrape calls. Batch results include an itemId and a per-record status or error. Results may arrive in a different order than the submitted URLs, so match records by ID rather than position. For other options at scale, see our structured-data API comparison.

When to use Answers instead of Scrape

Use Scrape JSON when you know the page URL and need its contents extracted into fields defined by a JSON Schema. Use Answers when your task requires an answer synthesized across multiple sources.

Answers accepts an example-shaped json_format, not JSON Schema. Its response puts the structured answer in json_content, identifies supporting pages in sources, and uses partial to flag an incomplete answer. You do not need to call Scrape before Answers.

FAQ

Does Scrape render JavaScript by default? Yes. Scrape uses a browser and waits 500 milliseconds by default before capturing output. For a page that loads content later, set sharedParams.waitFor to a selector observed on that page or to a longer wait in milliseconds. Rendering documentation explains both options.

How do I know whether extraction partially failed? Check isPartial and the JSON output’s success field even when the request returns HTTP 200. If the output failed, inspect its error_code and message, and retain request_id for debugging. Scrape documentation describes the response fields.

Can Batches return JSON directly? Batches accepts Markdown or HTML output, not schema-extracted JSON fields. Its results can arrive in a JSON container, but you must transform the page content separately or call Scrape JSON for each URL. Batches documentation lists the supported formats.

How do I validate extracted facts? Validate the returned object against your application schema, including permitted null values. Then check business rules and verify important claims against the source page. A schema can check shape, but it cannot establish factual accuracy. JSON extraction documentation covers missing values and output behavior.

Conclusion

For a live JavaScript-rendered website, getting usable JSON starts with rendering the page before extracting fields. Context.dev's Scrape endpoint handles that retrieval and extraction boundary, so you do not need to maintain your own headless browser and page parser. Your application still needs to normalize values and validate extracted data against its business rules.

For deeper schema design, read our schema-driven extraction tutorial. If you are evaluating providers, see our structured-data API comparison.