Both, and they are different systems. Indexing crawlers visit on their own schedule to build a searchable index. User-triggered fetches happen live when somebody asks a question that needs your page. So a change can appear within hours on one path and take weeks on another.
The short version
- Two mechanisms: scheduled indexing crawlers, and live fetches triggered by a user’s question.
- Indexing crawlers set the pace for most of what changes.
- User-triggered fetches can pick up a change within hours, but only for pages being asked about.
- Training data is a third thing entirely and updates on the provider’s schedule, not yours.
- This is why the window around a site launch matters far more than people expect.
Why this comes up
It usually surfaces as a reasonable objection during a migration: an assistant is not constantly spidering us, it only looks when somebody asks a question, so why does it matter what we fix before launch rather than after?
The premise is half right, and the half that is wrong is the expensive half.
Scheduled indexing crawlers. GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Bingbot
and Google’s crawlers visit on their own cadence, building indexes that answers are drawn from.
They decide when to come back, and that interval is not yours to set. For a mid-sized site it can
be days for important pages and weeks or months for the rest. This is the mechanism that governs
most of what an assistant knows about you, and it is slow to correct.
User-triggered fetches. ChatGPT-User, Perplexity-User, Claude-User and their equivalents
fetch a page live because somebody’s question required it. This path is fast, which is why a change
can occasionally show up the same day. But it only touches pages that somebody is actively asking
about, and it does not repair the index.
Training data is a third mechanism and the slowest. It is a fixed snapshot and updates when the provider retrains. Nothing you do reaches it quickly.
So the honest answer to the migration version of the question is: the pages nobody asks about today are exactly the ones that will sit wrong in the index for weeks, and the third-party sources pointing at your old URLs are re-crawled on their own schedule too. A problem that ships on launch day does not wait politely to be discovered. It propagates quietly at whatever pace each system happens to run at, and by the time it is visible, the trail is cold.
How to check it yourself
- Read your server logs, not your analytics. Filter by the crawler user agents above and look at actual fetch frequency for your key pages. This is the only place the real answer lives.
- Compare fetch rates across crawlers. They differ substantially, and knowing which ones neglect you is useful.
- Change one page and watch. Make a visible, factual change and ask assistants about it over the following fortnight to observe your own lag.
- Check whether a WAF is returning 403 to any of them. A permissive
robots.txtproves nothing if something upstream is blocking.
What to do about it
- Fix things before launch, not after. The asymmetry is the whole point of this page.
- Use IndexNow where it is supported. It is one of the few genuine accelerants available.
- Keep sitemaps accurate, because that is how scheduled crawlers prioritise.
- Do not assume silence means acceptance. No crawler traffic is a finding, not an absence of one.
- Plan for lag in your reporting. Judging a change two weeks in will usually mislead you.
If you have a launch coming, keeping your AI citations through a site migration covers the full sequence.