Why do AI fanout results change?

Results can change with model updates, time, language, country, available web pages, provider routing and normal generation variability. One run is an observation, not a permanent map.

Read this before comparing providers, screenshots or results from different days.

Create a dated comparisonRun the same short topic again with a controlled model, country or language change and save both results locally.Run the first observation

Two dated runs show a difference

Our approved example uses best-seo-tools-us and best-seo-tools-repeat from the 27 August 2026 OpenAI corpus. Their timestamps are 13:13:23.681 UTC and 13:13:45.771 UTC. Both record best SEO tools, language en, country US and the gpt-5.6-luna model label. Both have incomplete status.

For the next step, use the repetition protocol and run log.

Each run contains four exposed query strings and one search action. After NFKC normalization, trimming and lowercasing, exactly one string matches: Semrush SEO toolkit official features pricing. The other exact strings differ. This is set overlap under that rule, not a semantic-coverage assessment.

The normalized domain sets contain eleven and ten entries respectively. A domain set is neither a ranking nor a list of uniquely mapped query sources. Together with the strings, it demonstrates a difference between these saved observations; it does not identify the cause.

Matching recorded fields do not isolate a cause

The matrix separates input, locale, model label, tool version, time, status, queries and sources. Matching labels confirm only those stored metadata. They do not prove an unchanged internal model deployment, search-index version or provider-routing decision.

The observations are about 22 seconds apart. That interval is known; a time effect is not isolated. Generation, search routing or available context remain possible influences. Two incomplete runs establish neither a universal stability rate nor a reliable cause of changing search branches.

Country and language can affect search context. To investigate them, change only the intended field and repeat under documented conditions. Unobservable influences still remain. The existing country example is a separate dataset, not proof that every cross-country difference is caused by country.

See the boundaries in the dated country example.

Tie repetition to a concrete decision

State the question before another run. For a content section, it might be whether seat costs recur as a research task. For a parser, the question is whether exposed strings and statuses remain intact. These jobs need different acceptance criteria rather than one vague stability score.

Record input, locale, provider, model label and public tool version. Separate completed, incomplete, failed and completed-zero observations. A failure is not a successful observation with zero queries. Also note fields that were not exposed or were removed before saving.

For the next step, use the reporting template.

Another run is useful when its possible result would change the next action. A pricing section often needs a current official source before a tenth fanout repetition. Carry the control matrix and unresolved causes alongside any reported comparison.

For the next step, use the completed comparison exercise.

Common questions

These short answers cover recurring questions from tool use and source review. They are not claimed search-volume data.

Why can the same input differ seconds or days later?: Search indexes, model versions, tool routing and available sources can change. A single repeat is therefore an observation, not proof of a stable provider rule.
Which fields should stay fixed in a repeat?: Keep the exact input, provider, model, country, language and tool settings fixed. Record the timestamp and run status so a later difference can be interpreted honestly.

Sources used

These sources support the functions and limits described here. They do not prove claims beyond that scope.

Original annotated workflow, controls, prompts and export exercises
Original diagrams and editorial annotations; only the previously approved normalized OpenAI records. Synthetic selection files use actual producers and independent CSV readback, not new provider output or Excel/Sheets import. Own dated public-page checklist makes indexing and outcomes explicitly unproven.
Open source
NIST AI RMF Generative AI Profile
Supports explicit measurement, documentation, monitoring and limitations for generative-AI evaluations.
Open source
OpenAI web search tool guide
Documents web_search_call output and search actions that usually, but not always, include the query or queries searched. Search actions, exposed strings and cited sources therefore need separate interpretation. It does not expose chain of thought or guarantee that every route returns query strings. Rechecked for the original 2026-10-10 zero-query decision table; no new provider observation is published.
Open source
Gemini grounding with Google Search
Documents Gemini 3.7 Flash Google Search, google_search_call arguments.queries, cited URL annotations and per-query billing.
Open source
Dated OpenAI fanout example observations
Four owner-run OpenAI API observations record exact inputs, timestamps, exposed query strings, search-action source domains, usage, method versions and response status. Versioned JSON, normalized CSV and a JSON Schema are published. They are not an independent benchmark.
Open source