Comparability breaks in AI-answer studies

Watch the whole observation system

The model is only one source of change. Interfaces can alter whether citations are visible, routes can add browsing behavior, locale settings can change available sources, and capture code can normalize URLs differently. Any of these may change the measure.

Maintain a change log beside every batch. Record provider announcements when available, but also record observed interface and response-shape changes because not every material change is announced.

• Surface, route or access-mode change.
• Visible model or version label change.
• Prompt, control set or question-class change.
• Locale, personalization or session-rule change.
• Citation UI or extraction change.
• Normalization, coding or metric change.

Classify the effect, not the size

A small implementation change can be a major methodological break if it changes the denominator. A large visual redesign may be immaterial if the captured fields remain identical. Review the effect on the observation unit, missingness, field meaning and measure.

Use three outcomes: annotation when meaning is preserved, segmentation when comparison is possible only within eras, and restart when the unit or measure is no longer equivalent.

Use overlap runs when possible

If a known change can be anticipated, run the old and new method over the same control subset. The overlap does not automatically repair comparability, but it provides evidence about the size and direction of the method effect.

Document any bridge calculation separately. Never backfill old values with a new method without preserving the original release and explaining the revision.

Make the break visible in the result

A chart should show the boundary, and the data release should carry the protocol version for every record. The narrative should state what changed and which comparisons remain valid.

A broken series is not a failed study. Hiding the break is the failure. Visible segmentation is evidence that the research design responded honestly to a changing system.