Sensitivity Analyses, Robustness, and Model Assumptions

Every statistical result depends on choices about the target question, data, model, analysis population, and assumptions. A sensitivity analysis asks whether the inference for the same target treatment effect remains credible when a specified assumption is changed. Robustness is the stability of the substantive conclusion across credible limitations and analytic alternatives, not the number of analyses that happen to produce p values below a threshold.[1][2][3]

A useful sensitivity analysis makes its challenge visible. It states the assumption being probed, the alternative assumption or method, the estimand that remains in view, the result, and what the difference means for interpretation.

Start with the estimand and main estimator

FDA’s May 2021 final Level 1 guidance implementing ICH E9(R1) separates three linked ideas.[1][2]

• The estimand specifies the treatment effect the objective seeks, including the population, variable, handling of intercurrent events, and population-level summary.
• The main estimator is the prespecified statistical method used to estimate that effect.
• A sensitivity analysis uses one or more alternative assumptions to assess robustness of inference from the main estimator while targeting the same estimand.

This ordering matters. Choosing a convenient model first and then letting it redefine the question can make apparently similar analyses answer different things. Readers should be able to state what is being estimated before comparing methods.

“Primary analysis” commonly refers to the prespecified main analysis supporting the primary objective. It should identify the outcome, comparison, analysis population, model, covariates, missing-data approach, effect measure, and uncertainty. A sensitivity result is interpretable only relative to that anchor.

Sensitivity and supplementary analyses have different jobs

ICH E9(R1) distinguishes sensitivity analyses from broader supplementary analyses.[2]

Changing how an intercurrent event is handled can change the estimand. Changing from a mean outcome to a responder definition may answer a different question. Those analyses can be valuable, but calling all of them “sensitivity analyses” hides whether the target stayed constant.

The distinction is conceptual, not a ranking that makes supplementary analyses unimportant. It simply prevents disagreement between two target effects from being misreported as instability of one estimator.

Name the assumption before changing the method

A sensitivity analysis should be traceable through five elements:

1. Assumption challenged: for example, a distributional form, covariance structure, proportional hazards condition, missing-outcome relationship, or classification rule.
2. Alternative: the credible departure or method used to represent it.
3. Target: confirmation that the same estimand remains under evaluation.
4. Result: effect estimate, uncertainty, and other relevant diagnostics on a comparable scale.
5. Interpretation: whether magnitude, direction, precision, or substantive conclusion changes.

ICH E9(R1) recommends documenting assumptions and, where possible, changing one assumption at a time so the reason for a difference remains interpretable.[2] Altering the analysis population, endpoint definition, missing-data model, covariates, and effect scale simultaneously may yield another result, but it becomes difficult to learn which premise mattered.

Some assumptions are testable; others concern unobserved outcomes and cannot be confirmed from the data. A good fit statistic does not prove an untestable assumption.

Robustness is broader than repeated significance

ICH E9 describes robustness as sensitivity of overall conclusions to limitations in the data, assumptions, and analytic approaches. It says robust conclusions are not substantially affected under relevant alternatives.[3] That assessment includes the size and direction of the effect, confidence interval, decision threshold, and scientific importance.

Consider four patterns:

• Estimates remain similar and intervals support the same practical interpretation.
• Estimates keep their direction but vary enough that the likely magnitude becomes uncertain.
• The point estimate is similar, but uncertainty expands and includes materially different conclusions.
• Direction or decision changes under a credible assumption.

Only the first pattern is straightforwardly reassuring. The second and third may preserve evidence of a signal while weakening claims about its size or certainty. The fourth exposes a conclusion that depends strongly on an assumption. Reducing all four patterns to “significant/not significant” discards essential information.

Missing-data sensitivity needs an explicit target

Missing outcomes are a common focus because the relationship between observed and unobserved data is often only partly testable. ICH E9(R1) says missing data must be defined relative to the estimand and that plausible departures from assumptions should be studied.[2]

For example, a main analysis might assume that, after accounting for observed information, unobserved outcomes follow a relationship represented by the model. A sensitivity analysis can ask how the estimate changes if the unobserved outcomes are plausibly better or worse than that assumption implies. The point is not to manufacture exact values for unseen data; it is to show how much departure would be needed to alter the conclusion.

This guide does not prescribe a calculation. The method depends on the outcome, missingness, intercurrent-event strategy, and estimand. A generic “worst case” is not automatically conservative.

Tipping points locate dependence on assumptions

A tipping-point analysis varies an unverified assumption across a plausible range and identifies where the inference changes. The “tip” might be loss of statistical significance, crossing a clinically relevant threshold, reversal of direction, or another prespecified decision boundary. ICH E9(R1) explicitly notes that both statistical and clinical-relevance thresholds can be considered.[2]

The useful question is not merely whether a tipping point exists; nearly any result can be changed under an extreme enough premise. Readers should ask:

• Is the assumption at the tipping point plausible?
• How does it compare with observed participants or external knowledge?
• Was the range and decision boundary prespecified?
• Does the result change gradually or abruptly?
• Are departures allowed to differ between randomized groups for a defensible reason?

A tipping point far outside credible values can support robustness. One reached under a modest, plausible departure should narrow the conclusion.

Prespecification prevents a garden of favorable analyses

Running many analyses after inspecting the data and reporting only favorable versions does not demonstrate robustness. It introduces selection: the analysis becomes partly determined by the observed result.

Strong planning names important assumptions, credible alternatives, priorities, and interpretation rules in the protocol or statistical analysis plan before unblinding. Unexpected problems can still justify post hoc analyses, but the report should label them, explain why they were added, and distinguish them from prespecified tests.

CONSORT 2025’s statistical-methods item asks reports to describe methods used to compare groups for primary and secondary outcomes, including harms.[4] Its broader checklist separates additional and ancillary analyses. Transparent reporting makes it possible to see which result was primary and how alternative analyses arose; the checklist does not turn post hoc selection into confirmation.

Discordant analyses should remain visible

A sensitivity result that changes magnitude, direction, precision, or conclusion is evidence about dependence on assumptions. It should not be dismissed solely because it is not the primary analysis. Nor should it automatically replace a well-justified primary analysis.

The report should compare the assumptions directly and explain which are more credible in context. If both are plausible, the conclusion should preserve that uncertainty. Language such as “results were robust” should be reserved for a defined set of analyses and supported by a table or figure showing estimates and intervals.

Concordance can also mislead if every analysis shares the same flaw. Repeating the same biased outcome definition across several models does not test measurement error. Using several missing-data methods that all assume no important departure from observed patterns does not probe the key untestable assumption. Diversity of method names is not the same as diversity of assumptions.

A robustness table is more useful than a vote count

A reader-level evidence table can include:

This format discourages counting favorable analyses as votes. Ten minor variations do not outweigh one credible sensitivity analysis that exposes strong dependence on an important assumption.

A sensitivity-analysis reading checklist

1. Write the primary objective and estimand in plain language.
2. Identify the main estimator and its important assumptions.
3. Separate sensitivity, supplementary, exploratory, and post hoc analyses.
4. For each sensitivity analysis, name the single main departure being probed.
5. Confirm that it targets the same estimand.
6. Compare effect magnitude, direction, precision, and substantive thresholds.
7. Judge whether the alternative assumption is credible, not merely extreme.
8. Check prespecification and explain additions made after data review.
9. Preserve discordant results in the conclusion.
10. Avoid transferring robustness from a study analysis to a different material or claim.

Important limitations

• A sensitivity analysis is meaningful only relative to a defined main analysis and estimand.
• An analysis answering a different question is not a same-estimand sensitivity test.
• Repeated statistical significance does not by itself demonstrate robustness.
• A large p value in an alternative analysis does not necessarily prove conflict.
• Extreme worst-case assumptions may be uninformative if they are implausible.
• Prespecification reduces selective analysis but does not make assumptions correct.
• Concordant analyses can share the same bias.
• Analytic robustness cannot authenticate a separately sourced material or extend a result beyond the study.

Frequently asked questions

What is a sensitivity analysis?

Under ICH E9(R1), it is a series of analyses targeting the same estimand under differing assumptions to examine robustness of inference from the main estimator.[2]

How is a supplementary analysis different?

A supplementary analysis adds insight and may target another estimand or use another estimator. It can answer an important related question, but it should not be mislabeled as testing the same assumption for the same target.

Does the sensitivity analysis have to be prespecified?

Important sensitivity analyses should generally be planned before unblinding. A justified post hoc analysis can still inform interpretation, but its timing and rationale must be disclosed.

Does robustness mean every analysis is statistically significant?

No. Robustness concerns the stability of the substantive conclusion across credible assumptions, including magnitude, direction, uncertainty, and scientific relevance, not a vote count of p values.

What does a tipping point show?

It shows how far an assumption must move before a prespecified inference or decision changes. Its value depends on whether that level of departure is plausible.

Should a discordant sensitivity result be ignored because it was secondary?

No. It should be interpreted according to its assumptions, credibility, and target. A credible discordance may require narrower or more uncertain conclusions even when the primary analysis remains the main reference.

Can sensitivity analyses prove that bias is absent?

No. They probe specified assumptions and limitations. Unexamined measurement, selection, conduct, or reporting problems can remain, and multiple analyses can share the same flaw.

Primary and official sources

1. FDA: ICH E9(R1) Estimands and Sensitivity Analysis, May 2021 Final
2. FDA: Final ICH E9(R1) Addendum PDF
3. FDA: ICH E9 Statistical Principles and Robustness Definition
4. CONSORT 2025 Item 21a: Statistical Methods for Outcome Comparisons