Systematic Reviews & Meta-Analysis

A systematic review uses explicit methods to find, select, appraise, and synthesize evidence for a defined question. A meta-analysis may statistically combine compatible estimates within that review. The terms are not synonyms: a rigorous review may reject pooling, while a numerical meta-analysis can be unreliable if the search or appraisal was weak.

Cochrane’s Handbook places question formulation, eligibility, searching, data collection, and risk-of-bias assessment before statistical synthesis.[1][2][3] PRISMA 2020 provides a reporting framework for explaining what reviewers planned, did, found, and excluded.[5][6] Neither a forest plot nor a checklist label can substitute for those methods.

Start with the review question

A useful review specifies the population or system, intervention or exposure, comparator, outcomes, eligible study designs, settings, and time frames. Broad questions may be legitimate, but they can create clinical diversity that prevents one summary effect from having a clear meaning.

The protocol should define eligibility and methods before study results drive decisions. Later amendments should be dated, justified, and distinguished from the original plan so favorable findings do not silently influence choices.

Studies and reports are not the same unit

One study may produce a registry entry, conference abstract, journal article, follow-up paper, regulatory review, and correction. Conversely, one article can describe more than one study. Cochrane identifies studies, not individual publications, as the unit of inclusion and recommends linking multiple reports.[2]

Counting every report as a separate study can double-count participants and create false precision. Reading only one report can miss outcomes, protocol deviations, follow-up, or harms. A strong evidence table maps identifiers and reports to a single underlying study record.

A systematic search extends beyond one database

A reproducible search states the databases, registers, regulatory sources, dates, full strategies, language or publication restrictions, and supplemental methods. Cochrane recommends sensitive searches and trial-register searches because journal publication alone may omit eligible studies or results.[2]

PRISMA’s flow diagram shows records identified, screened, excluded with reasons, and included.[7] The diagram improves traceability; it does not show whether the search terms were adequate or eligibility decisions were unbiased. Those details remain in the methods and supporting materials.

An old search can make a review incomplete even if its original methods were strong. The search date therefore belongs next to the conclusion, and time-sensitive reviews need updating.

Appraise before combining

Pooling does not remove bias within the included studies. If several biased estimates point in the same direction, a precise weighted average can still be wrong. Reviewers should assess risk of bias for each important result with a tool appropriate to the included study design, outcome, and effect estimate, and explain how those assessments affect the synthesis.

Reviewers should also check intervention comparability, outcome definitions, time points, and overlapping populations. A large sample cannot repair a systematically mismatched comparison.

Certainty of a body of evidence is broader than the p value or confidence interval from a pooled analysis. Cochrane’s summary-of-findings guidance considers domains such as risk of bias, inconsistency, indirectness, imprecision, and publication bias.[8]

What meta-analysis does

Meta-analysis combines estimates from two or more studies, usually with precision-related weights.[3] It can increase precision and organize variation, but only when studies address a coherent question with compatible effect measures.

A pooled estimate should retain:

the outcome definition and time point;
included study designs and populations;
intervention and comparator boundaries;
effect scale;
model and weighting method;
study-specific estimates and uncertainty;
heterogeneity assessment; and
risk-of-bias and missing-evidence judgments.

When these differ too much, a structured synthesis without meta-analysis can be more truthful than a single number. “Could not pool” is not “no evidence”; it means the available evidence was summarized by another method or was insufficient for a meaningful quantitative combination.

Fixed-effect and random-effects models answer different questions

A fixed-effect model commonly assumes the studies estimate one common underlying effect, with observed differences attributed to sampling error. A random-effects model assumes study effects vary around a distribution and estimates an average effect under that model.[3]

Random effects do not make heterogeneous studies interchangeable. The method incorporates an estimated between-study variance component into the weights. When heterogeneity is estimated above zero, its conventional interval around the average effect is wider than the corresponding fixed-effect interval, but it still does not explain why effects differ or remove bias. With few or small studies, estimation of between-study variance and uncertainty around the average can be unreliable.

Model choice should follow the question and assumptions, not a preferred p value or the result of a heterogeneity test. If smaller studies show larger effects, random-effects weighting can give them relatively more influence; small-study effects remain a concern.[3][4]

Heterogeneity has several forms

Clinical diversity refers to differences in participants, interventions, comparators, outcomes, or settings. Methodological diversity refers to differences in design, conduct, measurement, or risk of bias. Statistical heterogeneity is observed effect variation beyond what sampling error alone would predict.[3]

Common statistical summaries include:

Measure	What it describes	Important boundary
Chi-square test	Evidence that effects vary beyond sampling error	Low power with few or small studies; high power with many studies
I²	Proportion of observed variability attributed to heterogeneity rather than sampling error	Uncertain with few studies; no universal threshold determines importance
Tau-squared	Estimated between-study variance on the analysis scale	Scale-dependent and imprecise with limited evidence
Prediction interval	Range intended to describe where an underlying effect in a new similar study may lie, or the spread of underlying effects, under a random-effects model	Depends on distributional and model assumptions and may be spuriously wide or narrow with few studies

An I² of zero does not prove identical effects. A high I² does not automatically prohibit pooling. Direction, magnitude, clinical and methodological differences, number of studies, and uncertainty all matter. When effects point in opposing directions, an average can obscure rather than clarify.

Exploring heterogeneity can create new false positives

Subgroup analysis and meta-regression can explore why study estimates differ, but they are observational comparisons across studies and are vulnerable to confounding. Cochrane recommends prespecifying a small number of plausible characteristics and using formal interaction methods.[3]

When reviewers inspect many study features after seeing heterogeneity, an apparent explanation can arise by chance. With few studies, meta-regression is especially unreliable. Such results are generally hypothesis-generating unless supported by strong prespecification, adequate data, within-study evidence, and replication.

Multiple analyses and null results need context

A review may examine many outcomes, time points, effect measures, models, subgroups, and sensitivity analyses. Repeated testing and flexible analytical choices increase the opportunity for chance findings or selective emphasis. Readers should distinguish a prespecified primary synthesis from exploratory analyses, inspect how many analyses were conducted, and check whether any confirmatory error control was planned.[3][4]

A pooled estimate whose confidence interval includes the null, or whose p value is above a chosen threshold, does not prove no effect, equivalence, or consistency across studies. The estimate, interval, effect scale, and prespecified decision rule should be reported together; a wide interval may remain compatible with important benefit and harm. Conversely, one nominally significant pooled result is not decisive when many analyses were run or when bias, indirectness, inconsistency, or missing evidence remains.

Publication bias is part of a wider missing-evidence problem

Publication bias is one form of non-reporting bias: whether an entire study report becomes publicly available may depend on the p value, magnitude, or direction of its results. Faster publication or more prominent placement are related dissemination biases, not synonyms for publication bias. Cochrane’s broader concept of bias due to missing evidence covers entire studies or particular eligible study results that are unavailable selectively.[4]

Selecting one reported estimate from several measured or analyzed results based on its findings is related but distinct bias in selection of the reported result. Readers should identify which results are wholly unavailable and which available result may have been selected from multiple candidates rather than treating every selective-reporting mechanism as identical.[4]

Searching registries, regulatory documents, protocols, and multiple databases can reveal evidence absent from journal articles.[2][4] Even a comprehensive search cannot guarantee that every study or result is recoverable or eliminate bias due to missing evidence.

Funnel-plot asymmetry is not a direct test that publication bias exists. Asymmetry can reflect heterogeneity, different methods, chance, poor design in smaller studies, or selective reporting. Tests for asymmetry have low power with few studies; Cochrane gives a rule of thumb that they should be used only when at least 10 studies are included, and still requires cautious interpretation.[4]

“No evidence of publication bias” is therefore too strong when methods were underpowered or data were unavailable. A better statement describes what was assessed, how many studies were available, what patterns appeared, and what uncertainty remains.

PRISMA improves reporting, not truth by declaration

PRISMA 2020 asks authors to report objectives, eligibility, information sources, search strategies, selection, data collection, risk of bias, synthesis methods, results, excluded studies, limitations, registration, funding, and data availability.[5][6] Complete reporting lets readers inspect the work.

PRISMA compliance does not certify that the question was sensible, the search was complete, the studies were unbiased, or the synthesis was correct. Likewise, publication in a journal, use of “systematic” in a title, or inclusion of a forest plot does not determine evidentiary strength.

A review-reading checklist

Is the question defined closely enough for the claimed conclusion?
Was a dated protocol registered or made available before synthesis decisions could be influenced by results, and are later amendments declared?
Are search dates, sources, and complete strategies reported?
Were trial registries, regulatory sources, and multiple reports considered?
Are inclusion decisions and exclusion reasons traceable?
Was risk of bias assessed for each important result?
Were the studies similar enough for the chosen synthesis?
Are model, effect scale, heterogeneity, and sensitivity analyses transparent?
Was missing evidence assessed without overreading funnel plots?
Does the conclusion match the certainty and applicability of the evidence?

Important limitations

A systematic review can be rigorous without a meta-analysis, and a meta-analysis can be unreliable without a rigorous review.
Statistical pooling does not cancel bias, indirectness, or incompatible study questions.
Random-effects models incorporate an assumed distribution of effects but do not explain heterogeneity.
I², Tau-squared, and funnel plots are uncertain, especially with few studies.
A non-significant pooled result does not prove no effect or equivalence, and an isolated nominally significant result needs multiplicity and analysis-selection context.
Missing studies and selectively missing results may distort the available evidence.
PRISMA is a reporting guideline, not a quality score or proof that conclusions are correct.
A pooled result for defined study interventions does not authenticate or establish equivalence for a separately sourced catalog material.

Frequently asked questions

Is every systematic review a meta-analysis?

No. A systematic review uses explicit methods to identify and synthesize evidence. Meta-analysis is an optional statistical component used only when quantitative combination is meaningful.

Does a larger pooled sample remove study bias?

No. Pooling can increase precision around a biased estimate if the included evidence has systematic problems.

What does I² measure?

It estimates the proportion of observed variation in study effect estimates attributed to heterogeneity rather than sampling error. Its uncertainty and context matter.

Does a random-effects model solve heterogeneity?

No. It models a distribution of effects and estimates an average under assumptions, but it neither explains differences nor makes incompatible studies comparable.

Does funnel-plot symmetry prove there is no publication bias?

No. Funnel plots and related tests have limited power and can be affected by several processes. Missing evidence can exist without clear asymmetry.

Does PRISMA compliance mean a review is high quality?

Not automatically. PRISMA supports complete reporting. Readers must still assess the question, methods, bias, synthesis, and conclusions.

Can a meta-analysis establish that a catalog material matches studied interventions?

No. It synthesizes evidence about the interventions defined in included studies and does not verify another material’s identity, composition, quality, or equivalence.

Primary and official sources

Review and corrections

Sources checked: September 1, 2026. Questions or corrections can be sent through the Serum Research contact page.

Editorial responsibility: Serum Research prepared and maintains this educational page. No independent scientific, statistical, evidence-synthesis, regulatory, or clinical review is claimed. This methods guide does not provide medical advice or establish that any intervention or catalog material is safe, effective, or equivalent to a studied intervention. See how Serum Research reviews evidence or return to the Research Peptide and Compound Library.

Are you over 21?

Systematic Reviews, Meta-Analysis, Heterogeneity, and Publication Bias