Missing data in AI-answer observations

Missing is not zero

An answer with zero visible citations is an observed value. An answer that failed to load has unknown citation count. A route that does not expose citations may be ineligible for a citation measure. Treating all three as zero changes the result and hides why.

Store the raw state and derive measure eligibility through code. This keeps the record reusable when a later analysis applies a different defensible rule.

Use a bounded reason taxonomy

Free-text notes are useful but difficult to aggregate. Define a small versioned set of machine-readable reasons, then allow an optional note for case detail. The taxonomy should match the actual collection process rather than copying a generic data-science list.

A useful starting set separates response unavailable, response interrupted, no visible source, source UI unavailable, access blocked, capture failure, field not exposed, retention not permitted and protocol exclusion.

Publish the eligibility flow

For every measure, show how the initial scheduled observations became the analyzed denominator. Report scheduled, attempted, completed, eligible, excluded and missing counts. Reasons should reconcile to the totals.

When missingness differs by surface or window, report that before comparing the main metric. A change in missingness may explain an apparent change in diversity or stability.

• Never delete a scheduled observation because it failed.
• Never use one denominator label for different eligibility rules.
• Never infer “no citations” from a missing capture.
• Version changes to the reason taxonomy.

Rights-limited is a data state, not a loophole

Provider terms or applicable rights may permit metadata while restricting raw output retention or publication. Record the permitted representation and the decision source. Do not imply that a hash or summary reproduces content that reviewers cannot inspect.

If the limitation prevents the research question from being audited, the correct outcome may be to withhold the claim or redesign the study.