A customer asks your website: "Does the Pro plan include API access?" Your AI search finds a pricing article, generates a confident answer, and cites the page. There is only one problem: the pricing changed three days ago. The answer is grounded, the source is real, the retrieval worked, and the result is still wrong.
This is the uncomfortable part of building AI search from website content. Getting pages into an index doesn't stay the hard problem for long. Keeping the answers aligned with the website does.
A production AI website search system has to solve more than crawling. It needs clean content, sensible indexing, useful retrieval, source attribution, access boundaries, and a freshness strategy that reflects how quickly different pages change. The crawl gets information into the system. The real work is deciding what happens after that.
Crawling the website is only step one
The simplest RAG pipeline looks straightforward: crawl the site, extract the text, split it into chunks, create embeddings, store them, retrieve relevant chunks when someone asks a question, and generate the answer. That architecture can work well, but every step hides a decision. Which URLs should be crawled? Should navigation and footer content be indexed? What happens to duplicate product pages? How should tables be represented? What about JavaScript-rendered content? How often should the crawl run again? What happens when a page is deleted?
Our guide to building an AI web research agent starts with a useful principle: raw HTML is usually the wrong format to feed directly into an LLM. HTML contains the content you want, but it also contains layout markup, scripts, navigation, tracking elements, and cookie banners, material that adds tokens without adding useful context.
That's why the extraction layer matters. Search quality starts before the index exists.
Clean the content before you index it
Imagine indexing a documentation page as-is: a navigation menu, a sidebar, the article, a related-posts section, a legal footer, a cookie banner, and three calls to action. If all of that becomes searchable text, the retrieval system now has several competing versions of what the page is about. The article explains API authentication, the navigation mentions Pricing, Integrations, Security, and Contact, and the footer repeats the company name twenty times. Nothing in the index automatically knows which parts deserve attention.
Clean extraction reduces that noise. Markdown is particularly useful here because it preserves the structure that helps models interpret content while stripping most presentation markup. Headings, lists, links, tables, and code blocks survive; the surrounding HTML scaffolding disappears.
For an AI search system, this matters twice. Cleaner content improves the material being indexed, and cleaner retrieval gives the generation model less irrelevant context to reason through later. Extraction isn't plumbing that happens before the important work; it's part of retrieval quality.
Decide what actually belongs in search
A whole-site crawler can find hundreds or thousands of URLs. That doesn't mean all of them should appear in customer-facing AI search. A typical website carries product pages, help articles, blog posts, legal policies, campaign landing pages, old release announcements, author archives, tag pages, duplicate regional pages, archived product information, and the occasional internal-looking document exposed publicly. The crawler sees URLs. The search product needs a content policy.
Start by asking what the user expects to find. For customer-facing search, that usually means current product pages, documentation, help center content, pricing, policies, current integrations, and recent release information. It usually doesn't mean thin tag archives, expired campaigns, old comparison pages, duplicate print views, staging content accidentally exposed to crawlers, or historical information with no current relevance.
This is one reason the SiteGPT case study is instructive. SiteGPT uses whole-site crawling to turn customer websites into AI knowledge bases, but the business value comes from handing the downstream system usable business content, not from collecting every HTML document. Coverage matters, and so does selection.
Keyword search and semantic search solve different problems
It's tempting to treat keyword search as the old system that semantic search is supposed to replace. That's too simple. Keyword search is excellent when the user's words match the thing they're looking for: "API documentation", "refund policy", "Shopify integration", "Pro plan". Exact terms matter.
Semantic retrieval becomes more useful when the user's language differs from the wording on the page. A visitor searches "Can I connect my store?" while the page is titled "E-commerce Integrations". They ask "How do I stop my subscription?" when the relevant article says "Cancel your account".
The two solve different retrieval problems, and a practical AI search experience uses both. Fast keyword retrieval handles explicit matches, semantic retrieval bridges vocabulary gaps, and reranking decides which results deserve to appear first. The real architecture question is how the two cooperate.
Once generated answers sit above those results, the quality bar rises again: an answer is only as trustworthy as the content retrieved underneath it.
Freshness needs its own architecture
Suppose your website has 2,000 pages. Your About page might remain stable for a year. A technical guide may change once every few months. Your pricing page could change tomorrow, and a status page might change several times today. Treating all four with one crawl schedule is inefficient: crawl everything every ten minutes and you waste resources, crawl everything weekly and your AI may answer commercial questions from stale information.
Design freshness by content type instead. One useful approach is to classify pages by expected volatility:
- Low-change content: company history, evergreen educational articles, stable documentation. These tolerate longer cache windows and less frequent crawling.
- Medium-change content: product documentation, integration pages, help center articles. These may need scheduled refreshes or event-driven updates when the product changes.
- High-change content: pricing, availability, changelogs, promotions, policies undergoing active updates. These deserve tighter monitoring.
Context.dev Monitors are built around this problem. A monitor can watch a page, a sitemap, or a specific piece of extracted information on a schedule, and produce a change record when relevant content actually changes. That opens a better pattern than blindly re-crawling everything: detect the change, then refresh the part of the knowledge system affected by it.
Know when to re-crawl and when to fetch live
Not every freshness problem should be solved by indexing faster. Sometimes the correct answer is to skip the index entirely. If an agent is answering "What is your refund policy?", a recent indexed copy may be completely adequate. If it's answering "What price is listed on this product right now?", and the page is volatile and the answer affects a purchase decision, live retrieval may be the safer choice.
We make a similar distinction in our guide to real-time web scraping tools for AI agents. The core difference is whether the system can tolerate cached or batch-collected information, or needs the page as it exists at request time.
That suggests a hybrid model: indexed retrieval for content that benefits from fast semantic discovery, live fetching when freshness outweighs retrieval speed, with the system deciding based on the question or the source. "How does the API work?" searches the indexed documentation. "What plans are listed today?" should probably fetch the pricing page live. "What changed in the latest release?" reads the current changelog or recently refreshed release content.
There are two ways to get this wrong: making every question pay the cost of live retrieval, and pretending every cached answer is fresh enough.
Return sources, not just generated answers
Generative answers make search feel easier. They also hide retrieval mistakes beautifully. Traditional search shows the user several results, and if the first one looks wrong, they can choose the second. A generated answer compresses that decision into one response, which makes sources more important, not less. If the answer says "The Business plan includes unlimited projects", the user should have a reasonable path to the page supporting that statement.
This is why customer-facing implementations should treat citations as part of the product experience. Mando's AI search for your website, for example, combines keyword and semantic retrieval and generates answers with their sources attached, drawn from connected site content. The important part isn't the specific interface. It's the separation between retrieval and assertion: the system searches, finds supporting material, produces an answer, and then shows the user where that answer came from.
That doesn't guarantee the answer is correct. It does make the answer inspectable, which matters once the interface is a single answer instead of ten results.
Keep public search separate from internal knowledge
Once you've built a good retrieval pipeline, the temptation is to put everything into it: public documentation, internal docs, support procedures, sales playbooks, customer records, engineering notes, company policies. One giant knowledge system, with permissions layered on afterward.
Be careful. Customer-facing search has a different risk profile from an internal assistant, and a public visitor shouldn't be able to retrieve information simply because the content happens to be relevant. Content needs exposure rules before retrieval, so for a customer-facing index, define explicitly what's public, what's authenticated customer content, what's tenant-specific, and what's internal or restricted.
The safest retrieval result is one the user was allowed to see before the model ever received it. This also makes debugging easier. If a strange answer appears, you want to know which approved corpus produced it, not investigate an enormous mixed index and hope a prompt-level instruction prevented leakage.
Treat search failures as knowledge signals
AI search can reveal problems in your website that ordinary analytics miss. Imagine people repeatedly search "Does this work with QuickBooks?" and get no useful result. That can mean several things: the integration doesn't exist, the integration exists but nobody documented it, the page exists but the crawler missed it, the page is indexed but retrieval can't connect the user's wording to the content, or the answer exists but ranks too low. Those are five different failures, and a good AI search system should help you tell them apart.
Worth tracking:
- Queries that return no results.
- Queries that produce low-confidence retrieval.
- Generated answers users reject.
- Results users repeatedly ignore.
- Questions followed by support conversations.
- Common wording that differs from your site's terminology.
Search becomes a knowledge feedback loop. If fifty customers type the same question every week, that's not just search traffic; it's evidence about what your content architecture is failing to explain. This may be the most valuable difference between traditional site search and AI-assisted retrieval: the query itself becomes product feedback.
A practical crawl-to-answer architecture
A useful production architecture can be thought of as eight stages.
- Discover. Use the sitemap, internal links, or defined URL sets to determine which content exists.
- Extract. Render pages when necessary and convert useful content into clean structured text or Markdown.
- Filter. Remove duplicate, irrelevant, private, expired, or low-value URLs before indexing.
- Index. Create the lexical and semantic representations needed for retrieval, and store enough metadata to know the page, section, timestamp, and access level behind every chunk.
- Retrieve. Use keyword, semantic, or hybrid retrieval based on the query, and rerank when necessary.
- Verify freshness. Before generating a high-impact answer, ask whether the underlying information is fresh enough. Use scheduled refreshes, change detection, or live fetching based on the source.
- Answer with evidence. Generate only from retrieved information, and expose source links so the user can inspect the underlying material.
- Learn. Track failed searches, weak retrieval, stale answers, and common unanswered questions, and feed those failures back into crawling, documentation, and search tuning.
The pipeline isn't crawl once, embed, done. It's a loop: discover, extract, index, retrieve, answer, observe, refresh. That last step separates a search demo from something you can leave running while the website continues to change.
Freshness is a search feature
It's easy to think of crawling as infrastructure and search as a product. In practice, the customer experiences both at the same time. They don't know whether an answer came from an embedding generated this morning or a cached page from three weeks ago. They only know whether the answer is useful. That's why freshness can't sit entirely behind the scenes.
The quality of AI website search depends on the entire chain. The crawler has to find the right pages and the extractor has to preserve their structure. The index has to contain the right content, retrieval has to understand the query, and the source has to still be current. The generated answer has to stay inside the evidence, and the user needs a path back to that evidence when the answer matters.
Crawling gets your website into the machine. A production search system has to keep the machine aligned with the website after that. That's the harder problem, and it's where the useful engineering starts.
Get a Context.dev API key to build the crawling, extraction, and change-detection layer your search index sits on.