Designing a control question set

Controls need a job

A control set is not a list of popular queries. Its job is to provide repeated observation units that make change interpretable. Each question should connect to a declared class or stress condition in the protocol.

Include enough variety to expose the behavior under study, but keep the set small enough to repeat reliably within the cost ceiling. Unmaintainable controls are not controls.

Select for stability and diagnostic value

Prefer wording that has one clear task and does not depend on a private personal context. Flag time-sensitive questions, location-sensitive questions and questions whose correct answer changes. These can be useful, but they need their own interpretation.

Record why each question exists, which class it represents and what would make it invalid. This protects the set from becoming an unexplained prompt collection.

• Stable question ID and exact submitted text.
• Question class and inclusion rationale.
• Locale and route requirements.
• Volatility or safety flag.
• Retirement condition.

Freeze text and conditions together

The text alone is not the full control. Surface, route, locale, session rule, timing and available configuration belong to the protocol version. A stable prompt sent through a different route may be a different observation unit.

Hash or otherwise identify the frozen control file. Publish the version and effective date. Where the raw question cannot be shared, publish enough metadata to explain the limitation without inventing reproducibility.

Change through retirement and replacement

Do not edit a question in place. Retire the old ID with a reason and add a new ID. Decide whether the new question begins a new series or can be analyzed only as an adjacent exploratory track.

Periodic review is still necessary. A formerly stable question may become ambiguous, unsafe, obsolete or impossible to run under changed provider terms. Version control preserves the history without forcing a broken control to continue forever.