Statistical Significance, Meaningful Change, and Responders

A statistically significant result is not automatically large, important, or meaningful to participants. A change that one person experiences as important is not automatically evidence of a between-group effect. A responder threshold can make a continuous score easier to describe, but it also discards information and depends on a justified context.

These distinctions are easiest to preserve by reporting four layers separately: the prespecified comparison, the effect estimate and uncertainty, evidence about what score differences mean, and any individual-level responder analysis. A p value can contribute to the first layer. It cannot replace the other three.

What a p value does and does not say

The American Statistical Association’s March 2016 press release summarizing its statement explains that p values indicate how incompatible data are with a specified statistical model.[1] The linked three-page document reproduces the statement’s six principles; it is not the full journal article. The calculation depends on the tested hypothesis, model, design, data, and assumptions.

A p value does not directly measure:

• the probability that the tested hypothesis is true;
• the probability that the data arose from random chance alone;
• the size or direction of an effect;
• the importance of a result;
• the probability that an individual will benefit; or
• the quality of the study.

The ASA also states that scientific or policy conclusions should not rest only on whether a p value crosses a threshold and that proper inference requires complete reporting and transparency.[1] A result at p=0.049 and one at p=0.051 are not separated by a scientific cliff. Their estimates, intervals, design, multiplicity, and context may be nearly identical.

Sample size affects statistical significance. A very large study can produce a small p value for a tiny contrast. A small study can produce a large p value even when its interval remains compatible with an important effect. “Not statistically significant” does not prove no effect; it describes the test result under the specified analysis.

Start with the between-group effect

In a randomized trial, the central efficacy comparison is generally between assigned groups on a prespecified endpoint and time point. The report should provide the estimate, such as an adjusted mean difference, risk difference, risk ratio, or another defined contrast, along with a confidence interval.

That interval shows a range of effect values compatible with the data and model under the chosen confidence procedure. It does not repair bias, selective reporting, missing outcomes, or a poorly chosen endpoint. Use Confidence Intervals and P Values for those interpretation boundaries.

A within-person change is different. If a participant’s score moves from 70 to 60, the observed change is 10 points in the direction defined by that instrument. It does not show what would have happened without the studied assignment. Natural history, regression to the mean, repeated measurement, co-interventions, expectations, and other events can contribute. The comparator group is needed for a causal between-group contrast.

Quantities that should not be collapsed

A group mean difference is not “the average number of people who improved.” A responder proportion does not preserve how far above or below the threshold each person was. MSDs often concern within-person change, but FDA’s draft framework also allows other score comparisons. Neither MSDs nor MSRs are automatically responder endpoints: they can help interpret an original-scale result without replacing it with categories.[4]

Meaningful change is context-specific

FDA’s final 2009 patient-reported outcome guidance says an empirically supported responder definition can vary by target population or other trial characteristics and should be evaluated in the specific trial context.[2] FDA’s final October 2025 PFDD Guidance 3 emphasizes selecting, developing, or modifying clinical outcome assessments that are fit for their context of use.[3]

A clinical outcome assessment (COA) measures a defined aspect of how a person feels or functions, or another specified clinical concept. Its instructions, administration, scoring, and evidence must support the proposed use. Fit-for-purpose measurement is necessary but does not, by itself, justify every endpoint built from that measure.[3,4]

A meaningful-change threshold is therefore not a permanent universal property of a questionnaire score. Its interpretation can depend on:

• the exact instrument, version, language, scoring rule, and administration method;
• the concept measured, such as a particular symptom or aspect of function;
• the target population, condition, severity, and baseline score range;
• whether improvement, deterioration, or both are being interpreted;
• the endpoint construction and assessment time point;
• the anchor and evidence used to justify the threshold; and
• the trial objective and broader regulatory or research context.

Even people in one population may disagree about what change feels meaningful. A prespecified range can communicate uncertainty when interpreting a continuous endpoint. By contrast, a dichotomized responder endpoint needs a single prespecified threshold; an endpoint based on several score regions needs its prespecified set of boundaries. An interpretation range is not permission to choose the most favorable binary definition after seeing the results.[4]

Anchor-based and distribution-based evidence differ

An anchor is an external variable whose meaning is more directly interpretable than the score needing interpretation. An anchor-based analysis relates categories or changes on that external measure to changes on the outcome instrument. For an MSD analysis, the anchor should not simply be derived from the COA being interpreted. It should assess the same concept or one encompassing it, be understandable, have a justified meaningful distinction, and show an adequate association with the target score over comparable assessment and recall periods. These are quality requirements, not a claim that every method of interpreting score regions requires an external anchor.[4]

Distribution-based information uses statistical features such as variability or measurement error. It can show whether a proposed change is large relative to score variation or measurement precision. It does not directly show that participants regard the change as important.

FDA’s 2009 final guidance describes distribution-based approaches as supportive rather than an appropriate sole basis for a responder definition.[2] FDA’s April 2023 PFDD Guidance 4 makes the same distinction for meaningful score differences and recommends using multiple sources and conveying uncertainty.[4] Guidance 4 remains draft, non-binding, and not for implementation as of the review date; its recommendations are not final FDA policy.

Anchor quality matters. A retrospective rating can be influenced by current status, recall, knowledge of assignment, or expectations. A clinician measure may not represent a patient-experienced concept. Correlation alone does not establish that the selected anchor category is meaningful.

When can score-meaning evidence interpret a group difference?

It would be too strong to say that a between-group mean difference can never be considered alongside MSD evidence. Section III.C.1.a of FDA’s draft Guidance 4 expressly describes such a comparison when there is evidence that the MSD is relatively consistent across baseline scores. Its illustration combines a baseline-adjusted mean contrast and confidence interval with an independently derived, prespecified MSD range. The model example assumes an approximately constant treatment effect across baseline-score subgroups.[4]

When meaningful differences vary substantially with baseline status, the interpretation needs more care. An effect exceeding the largest supported MSD suggests meaningfulness across that range; an effect below that largest value could matter for some patients or for none, depending on their baseline scores. Baseline-specific comparisons and other well-justified analyses may be informative. One should not silently treat a threshold from a different population or baseline range as applicable.[4]

The draft’s illustrations also consider the confidence interval, not merely whether the point estimate crosses one value. A precisely estimated effect lying above a justified MSD range supports a different interpretation from an interval spanning smaller and larger differences. These are hypothetical guidance examples, not results of a real trial or a universal numerical rule.

MSRs offer another approach: connect regions of a scale with experiences such as different levels of functional difficulty. The regions need not have equal widths. Their interpretation and any comparison with a treatment effect require evidence about the relevant health states, baseline status, and score properties. A change of the same numerical size need not have the same meaning everywhere on a scale.[4]

Responder analyses trade detail for classification

A responder analysis classifies each participant according to a prespecified rule, such as reaching a score state or changing by at least a justified amount at a stated time. This describes an observed participant-level outcome, but it transforms an ordinal or continuous outcome into categories. Being classified as a responder does not establish that treatment caused that particular person’s improvement. A randomized between-group responder contrast addresses a different, population-level causal question under its design and analysis assumptions.

FDA’s draft PFDD Guidance 4 notes that using the original ordinal or continuous score retains all information and usually maximizes statistical power; dichotomization can be reasonable for a clearly defined important health state when the threshold is justified and prespecified.[4] Participants just above and just below the threshold may have nearly identical experiences but receive opposite labels. Participants far above the threshold and barely above it are both counted as responders.

The continuous effect should therefore remain visible. Showing the distribution of changes or results across several justified thresholds can reveal whether a claim depends on one cut point. For a dichotomized endpoint, the draft specifically recommends that the data used to derive its score threshold differ from the data used to demonstrate effectiveness. That recommendation is stronger than simply calling a favorable post-hoc threshold exploratory. Additional displays across thresholds can supplement a prespecified analysis; they do not retroactively create its endpoint or multiplicity control.[4]

Separately, the draft recognizes that advance evidence about score interpretation may sometimes be infeasible. It describes constrained exit-interview approaches, with attention to masking, timing, and trial integrity. This is not a general exemption to select a responder cutoff after examining which one produces the smallest p value.[4]

What a complete responder result needs

For each group, report:

1. the exact prespecified responder definition, direction, endpoint, and time point;
2. why the threshold is meaningful in this instrument and population;
3. the randomized and analyzed denominators;
4. how participants with missing outcomes were classified or modeled;
5. responder counts and percentages in each group;
6. a between-group effect such as a risk difference or risk ratio with uncertainty;
7. multiplicity status and whether the analysis was primary, secondary, or exploratory; and
8. the corresponding continuous result and distribution when available.

A p value alone omits most of that information. “Thirty percent responded” is also incomplete without the comparator proportion, denominator, time point, uncertainty, and missing-data rule. See Absolute vs Relative Effects and Baseline Risk for presenting binary contrasts on both scales.

Multiple endpoints, time points, thresholds, subgroups, and models create repeated testing. Prespecification and a suitable multiplicity strategy matter for confirmatory interpretation. The confidence-interval and p-value guide discusses multiplicity and why one favorable result among many needs cautious interpretation.

Meaningfulness is more than one cutoff

An interpretation should bring together the estimated between-group effect, interval, score meaning, distribution, duration, endpoint importance, harms, missingness, and consistency with other evidence. A statistically detectable change can be too small to matter. A potentially meaningful estimate can remain too uncertain for a firm conclusion. A responder contrast can be informative while still hiding variation.

Null and uncertain results deserve the same care. The Negative and Null Results guide explains why failure to cross a statistical threshold is not proof of equivalence and why intervals reveal what effects remain compatible with the data.

Important limitations

• A p value describes data–model incompatibility under assumptions; it is not a probability that a hypothesis is true.
• Statistical significance does not measure effect magnitude, importance, or individual benefit.
• Within-person change and a randomized between-group effect answer different questions.
• Meaningful-change thresholds are instrument-, concept-, population-, endpoint-, time-, and context-specific.
• Distribution-based information is supportive but does not directly establish patient-perceived meaning.
• Dichotomizing a continuous outcome loses information and can make conclusions depend on one threshold.
• FDA PFDD Guidance 4 is draft and non-binding; this page does not provide medical or treatment guidance.

Frequently asked questions

Does p<0.05 prove that an effect is real or important?

No. It is a model-dependent threshold result. Read the effect estimate, confidence interval, design, multiplicity, endpoint, and limitations.

Does p≥0.05 prove there is no effect?

No. The interval may remain compatible with benefit, harm, or little difference. A non-significant result is not proof of equality or equivalence.

Is within-person improvement the same as a treatment effect?

No. A treatment effect compares defined conditions, usually between randomized groups. A person’s observed change can have several causes.

Is one meaningful-change threshold valid for every study using an instrument?

No. Its support depends on the instrument version, concept, population, baseline range, endpoint, time point, direction, anchor, and context of use.

What does an anchor-based method add?

It links score differences to an external variable with interpretable meaning. Its usefulness depends on conceptual fit, measurement quality, association, timing, and bias.

Why not report only the percentage of responders?

Classification discards the size and distribution of changes. Readers also need group denominators, comparator results, uncertainty, missing-data handling, multiplicity, and the continuous analysis.

Can a mean difference be compared with a meaningful-change threshold?

Sometimes, with evidence and an appropriate interpretation plan. FDA’s April 2023 draft Guidance 4 describes this when MSDs are relatively consistent across baseline scores; substantial baseline dependence needs more careful interpretation. The guidance remains draft, non-binding, and not for implementation as of the review date.

Primary and official sources

1. American Statistical Association: March 7, 2016 press release reproducing the statement’s six principles
2. FDA: Patient-Reported Outcome Measures, Final Guidance
3. FDA: PFDD Guidance 3, Fit-for-Purpose Clinical Outcome Assessments, Final Guidance
4. FDA: PFDD Guidance 4, Incorporating Clinical Outcome Assessments Into Endpoints, Draft Guidance

Review and corrections

Editorial evidence review: September 3, 2026, in both America/Edmonton and UTC. Serum Research is responsible for this source-based educational guide. This is an editorial review of statistical and regulatory source material, not an independent clinical-outcome-assessment or statistical specialist review, a reanalysis of a named trial, or medical advice.

The review checked the ASA release, current FDA guidance status, relevant sections of the linked guidance PDFs, and material figures. It did not review every section of all three FDA documents or the full ASA journal article. FDA’s 2009 PRO guidance and October 2025 PFDD Guidance 3 are final; April 2023 PFDD Guidance 4 remains draft. None of these documents establishes the benefit, safety, or suitability of a particular intervention.

See how we review evidence, return to the Research Library or study records, or report a source discrepancy through Contact Us.