You ask an internal chatbot what a supplier charges this month. It answers in one confident sentence, and the number turns out to be from last spring. That happens more often than people expect, and it usually has little to do with the model itself. The model only knows its training data plus whatever documents your retrieval system hands it, so if those documents are old, the answer is old too.
Most teams building assistants now pull matching documents into the prompt before the model writes a reply, which works pretty well but quietly moves the hard part onto the data pipeline. Somebody has to keep the pages in that index current, and when the source is the public web, that job is a lot bigger than it looks.
Why stale sources cause so many wrong answers
When an assistant gets a figure wrong, the usual reason is that it found an older copy of the right page. Prices, plan limits, fees and policy pages change quietly and often keep the same URL, so an old crawl still looks perfectly valid to the retriever. The model then repeats the old number with the same confidence it would give a current one.
Refresh on a schedule that matches the source
Refreshing everything every day sounds like the fix, but it gets expensive quickly, and it puts extra load on the sites you depend on. A better approach is to sort sources by how often they really change, so a pricing page might get checked daily while an old explainer gets checked once a month.
The network side matters too, since a refresh job hitting the same domains from a shared, busy address tends to get slow or inconsistent responses. Many teams run these jobs through IPRoyal’s private isp proxies solution or something similar, which gives the crawler a small pool of fixed, dedicated addresses that look the same from one run to the next. That makes it much easier to tell a real content change apart from a network hiccup.
Check what changed before downloading it again
The web already has a polite way to ask whether a page changed. With conditional requests, the crawler sends the date or ETag it saw last time, and if nothing changed, the server sends back a short “not modified” reply instead of the full page. That saves bandwidth on both sides and gives you a clean signal for when a document needs to be embedded again.
When a page does change, keep the old version next to the new one with a timestamp. That way the chatbot can say a figure was checked on a given date, and you can look back later when someone asks why an answer moved.
Follow the rules of the sites you collect from
Public pages still come with rules, so the crawler should read each site’s robots.txt file and stick to what it says about which paths are open. It’s worth setting your own limits as well, with a sensible pause between requests to the same domain and a clear user agent that says who is crawling. Sites that offer an API or a data feed are usually the better source anyway, since the data comes cleaner and nobody has to guess at page layouts.
Keep the index and the answers in sync
A fresh page only helps once it is back in the index, so changed documents should be split and embedded again straight away rather than waiting for a weekly rebuild. Old chunks need to be removed properly too, because leftover copies of last year’s page will keep turning up in results. The whole idea behind retrieval-augmented generation is that the model answers from current sources, and this step is where that usually falls apart.
A small setup covers most of it
You don’t need a huge crawling operation to keep an assistant accurate. A list of trusted sources, a schedule based on how often each one changes, conditional requests, stable network settings and a quick test set of questions with known answers after each refresh will catch most problems before users do.
The model still matters, and some models are better than others at saying when they aren’t sure. But if the documents behind it are current and clearly dated, even an average model ends up giving answers people can rely on.