Not in the conventional sense, because you cannot hold a model constant or split its traffic. What you can do is run before-and-after measurement on a fixed query set, test structural variants across comparable pages, and use unchanged pages as a control. It is slower and less clean, but it is real evidence.
The short version
- No split testing. You cannot serve two versions of a page to one model and compare.
- Model outputs vary between identical runs, so a single comparison proves nothing.
- Before-and-after on a fixed query set does work, with enough runs to see through the noise.
- Cohort testing across comparable pages is the closest thing to a real experiment.
- Always keep an unchanged control group. Platforms shift on their own and will fool you.
Why the conventional method breaks
A/B testing needs three things: two variants live at once, traffic randomly split between them, and a stable measurement of what each produced. AI citation gives you none of them.
There is one version of your page. You cannot show a model variant A and a competitor’s browser variant B. There is no traffic to split, because the unit is a generated answer rather than a session. And the measurement is not stable: ask the same model the same question twice and you can get different sources cited, with no change on your side at all.
That last point defeats naive testing entirely. If you change a page, ask once before and once after, and see a difference, you have learned nothing. You may have observed ordinary run-to-run variance.
Three approaches that do work
Repeated before-and-after on a fixed query set. Fix the queries, run each several times to average out variance, record the baseline, make one clearly defined change, wait for re-crawling, then re-run identically. The multiple runs are what make this credible rather than anecdotal.
Cohort testing. Take a group of comparable pages. Apply a structural change to half of them, leave the other half alone, and compare movement across the two groups over a couple of months. This is the closest thing to a genuine experiment available, and it is how you learn which changes are worth rolling out across a large site.
Holdout controls. Whatever else you do, always keep some pages unchanged. Platforms update their retrieval and ranking behaviour independently of you. Without a control you will attribute a platform-wide shift to your own work, in whichever direction flatters or alarms you most.
How to check it yourself
Ask an assistant the same question five times in a fresh session each time and record which sources it names. The variation you see is your noise floor. Any test you design has to be able to detect a change larger than that, which is a useful and slightly humbling thing to establish before you start claiming results.
What to do about it
- Change one thing at a time per cohort. Bundled changes cannot be attributed.
- Run each query several times and record the proportion of runs you were cited in, not a yes or no.
- Allow weeks, not days. Re-crawling and re-indexing set the pace, and it is not yours.
- Keep a control group permanently.
- Write down the prediction before the test. It is the cheapest defence against reading the result you wanted.