How to measure source diversity in AI answers

Choose the diversity unit first

URL diversity, registered-domain diversity and source-type diversity answer different questions. Ten cited URLs from one publisher can be diverse at page level and concentrated at domain level. A study should report the unit explicitly rather than using “sources” as an undefined count.

Normalize hosts and URLs before counting. Decide how to handle www aliases, tracking parameters, fragments, syndicated copies, subdomains and redirects. Preserve the original visible URL beside the normalized value so the transformation remains reviewable.

• Unique visible URLs: breadth at document level.
• Unique registered domains: publisher concentration.
• Declared source classes: mix of documentation, research, editorial or other categories.
• Recurrence share: how much of the sample is occupied by repeating sources.

Publish the denominator

A result such as “24 domains appeared” cannot be interpreted without the number of questions, answers returned, answers with citations and total visible citation slots. The denominator also changes when an interface returns no answer or no links.

Report coverage before diversity: eligible observations, captured answers, answers with at least one visible source, and total source occurrences. Then report unique values and concentration. This prevents a sparse surface from looking diverse simply because only a few observations contained links.

Diversity is descriptive, not automatically good

A higher unique-domain count may reflect broader sourcing, noisy citations, repeated one-off domains or a different question mix. A lower count may reflect appropriate reliance on primary documentation. Diversity should therefore be interpreted beside question class, source relevance and source role.

Do not turn the measure into an unvalidated quality score. If quality, authority or correctness matters, define and review those constructs separately.

Comparing two windows

Use the same control questions, surface, locale, route, timing rule and normalization code. Compare both the set overlap and the distribution of occurrences. A stable unique count can hide a complete turnover in which domains appeared.

Annotate interface or provider changes. If a change affects citation availability, the safest result may be a comparability break rather than a trend line.