Look beyond
the headline.
Three small experiments to help you read the next big claim. The numbers below are hypothetical teaching examples, not omega-3 treatment forecasts.
What kind of answer
can a study give?
Randomized controlled trial
People are assigned by chance to intervention or comparison groups. This helps balance known and unknown confounders and supports causal inference.
Still inspect blinding, missing data, adherence, study duration and whether the trial asked the question you care about.
The same percentage.
A different impact.
Imagine an intervention lowers relative risk by 25% over a fixed five-year period. Change the starting risk to see why the absolute difference matters.
Hypothetical constant relative risk ratio of 0.75. Real effects and baseline risks require actual trial data. This illustration does not model time-to-event hazard ratios.
20 of 100 versus 15 of 100 over five years.
The last figure is the number needed to treat (NNT): 1 ÷ absolute risk reduction expressed as a proportion. It is tied to a specific population, outcome and time period.
More data can change the confidence.
Not the size of the effect.
Two hypothetical groups differ by 2 points on a symptom scale. Each has a standard deviation of 10 points. Keep that effect fixed and change the sample size.
Illustrative two-sided normal z-test with independent groups and known equal population standard deviations. Approximate 95% confidence interval = difference ± 1.96 × standard error. This simplified model is not an analysis of a real trial.
Estimated improvement: 2 points
95% confidence interval: −0.77 to 4.77 points
Two-sided p-value: 0.157
The interval includes no difference. The data remain compatible with more than one meaningful interpretation.
What does a p-value actually mean?
Under a specified statistical model, assuming the null hypothesis and other model assumptions hold, a p-value is the probability of a test result at least as extreme as the one observed. A small value suggests incompatibility with that null model.
It is not the probability that the treatment works, the probability that the null hypothesis is true, or the probability that the findings happened “by chance.”
Does p < 0.05 mean an important benefit?
No. It is a conventional threshold for statistical significance, not a threshold for clinical importance. A very small effect may reach it in a large study. A potentially useful effect may not reach it in a small study. Values of 0.049 and 0.051 are not fundamentally different evidence.
What does a 95% confidence interval mean?
Under the model assumptions, a procedure for constructing 95% intervals would cover the fixed true effect in 95% of repeated samples. For this study, the interval shows a range of effects compatible with the data and model. It is not a 95% probability statement about this particular fixed parameter.
Wide intervals mean imprecision. They do not capture every source of error, including bias or poor applicability.
Why do researchers care about multiple testing?
Searching many outcomes or subgroups makes isolated low p-values easier to find. Prespecifying the primary outcome, reporting all analyses and using appropriate adjustments helps distinguish stronger evidence from exploratory signals.
Further reading: American Statistical Association statement on p-values · Cochrane: interpreting results.