Random-Effects vs Fixed-Effect Meta-Analysis: How to Choose

Choosing between a random-effects and a fixed-effect model is the first real statistical decision in a meta-analysis, and it is routinely made backwards: run the pooled analysis, look at the heterogeneity statistic, then switch models if it looks large. The methods literature is unambiguous that the choice is about assumptions, made before the data are combined - the two models answer different questions, and neither is a repair kit for the other [1].

This article lays out what each model actually assumes, how those assumptions change the weights and the intervals, and a practical way to make the call in your protocol.

What does each model assume?

The fixed-effect model assumes there is one true effect size shared by every included study, and that all observed differences between study results are sampling error. Under that assumption, pooling is simply an exercise in precision: combine the studies to estimate the one effect they all share [1].

The random-effects model assumes the true effects themselves differ from study to study - because populations, doses, delivery, and settings differ - and that the included studies are a sample from a distribution of true effects. The pooled result then estimates the mean of that distribution, not a single common effect [1].

That difference in target is the point most summaries miss. The fixed-effect estimate answers "what is the common effect in these studies?" The random-effects estimate answers "what is the average effect across the kinds of settings these studies represent?" [1].

How do the weights and intervals differ?

Under a fixed-effect model, studies are weighted by the inverse of their variance, so large trials dominate: in a pool containing one mega-trial and several small studies, the mega-trial can contribute the overwhelming majority of the weight [1]. A random-effects model adds the between-study variance, tau-squared, to every study's variance before weighting. That extra term pulls the weights toward equality, so smaller studies carry relatively more influence than they would under the fixed-effect model [1].

Intervals behave accordingly. Random-effects confidence intervals are wider whenever between-study variation exists, because they carry the extra uncertainty of effects that genuinely vary [1]. And the random-effects framework offers something the fixed-effect model cannot: a prediction interval, the range within which the true effect in a new setting is expected to fall. Riley and colleagues argue this is often the most clinically useful output of a random-effects meta-analysis, because a pooled average can look precise while the effects it averages range from harmful to strongly beneficial [3].

How should you actually choose?

Make the decision in the protocol, from the review question and the expected clinical diversity of the studies - not from a heterogeneity test after pooling. The Cochrane Handbook is explicit that basing the model choice on the statistical significance of a heterogeneity test is not recommended, in part because that test has low power when studies are few [2].

A practical rule: if the included studies deliberately span different populations, intervention variants, or care settings, the assumption of one common effect is untenable, and random-effects is the honest model. The narrow case for fixed-effect is a set of studies so operationally similar - same protocol, same population, same dose - that a single shared effect is plausible [1]. In clinical evidence synthesis that situation is the exception, which is why random-effects is the default posture of most modern reviews and of automated pipelines, including ours at Arakis.

Two cautions apply in the low-study regime. First, when the pool is small, tau-squared is estimated imprecisely, so random-effects outputs inherit that uncertainty and should be presented with it [3]. Second, do not treat a heterogeneity statistic near zero as permission to switch to fixed-effect after the fact - with few studies the statistic simply lacks the power to detect real variation [2].

Which estimator and adjustment should you use?

Random-effects is a family, not a single method, and the default in much legacy software - DerSimonian-Laird weights with a normal-approximation confidence interval - performs poorly exactly where meta-analysis is hardest. In simulations by IntHout and colleagues, the standard DerSimonian-Laird approach produced error rates well above the nominal 5% level when studies were few or unequally sized, while the Hartung-Knapp-Sidik-Jonkman adjustment held error rates close to 5% across those same scenarios [4].

The practical consequence: with a typical clinical meta-analysis of a handful of trials of varying size, an unadjusted random-effects interval can be too narrow and its p-value too optimistic [4]. The Hartung-Knapp adjustment - a t-distribution-based correction to the pooled standard error - is a straightforward fix and is the default in the Arakis analysis pipeline for exactly this reason.

Key takeaways

• Fixed-effect assumes one common true effect; random-effects assumes a distribution of true effects and estimates its mean [1].
• Random-effects weighting adds tau-squared to every study's variance, flattening weights and widening intervals relative to fixed-effect [1].
• Choose the model in the protocol from clinical reasoning; do not switch models based on a heterogeneity test's significance [2].
• Report a prediction interval with random-effects results - the average alone can hide a range spanning harm to benefit [3].
• Prefer the Hartung-Knapp adjustment over the standard DerSimonian-Laird interval; it keeps error rates near the nominal 5% when studies are few or uneven [4].

FAQ

Is random-effects always the safer choice?

It is the more defensible default in clinical evidence synthesis because included studies almost always differ in populations and delivery [1]. But it is not free: its outputs depend on a between-study variance that is poorly estimated from small pools, and that uncertainty should be reported rather than hidden [3].

Can I run both models and report the one that looks better?

No. The models answer different questions, and selecting between them after seeing the results is a form of outcome shopping. Prespecify the model, and if you present the other as a sensitivity analysis, label it as exactly that [2].

Does a random-effects model fix heterogeneity?

No. It incorporates between-study variation into the weights and intervals; it does not explain that variation. Considerable heterogeneity still calls for subgroup or sensitivity analysis, and sometimes for the judgment that a single pooled average is not a meaningful summary [2].

What is the Hartung-Knapp adjustment in one sentence?

It replaces the normal-approximation confidence interval of a random-effects meta-analysis with a t-based interval that accounts for the uncertainty in estimating tau-squared, keeping false-positive rates near the nominal 5% level in the small, unbalanced pools where the standard method fails [4].