Skip to content
All resources

Reading the evidence

P Value vs Confidence Interval: What Each One Tells You

AR

By

· 10 min read

A summary of published research, with citations. It states no amount, schedule or route, and makes no claim about what this compound does in a person. Supplied for laboratory research use only.

On this page · 8 sections
  1. What does a p value actually mean?
  2. What does a 95% confidence interval mean?
  3. Why is "statistically significant" not the same as important?
  4. What does a non-significant result mean?
  5. What is multiplicity, and why does it matter?
  6. Where do 0.05 and 95% come from?
  7. A checklist for any reported result
  8. Frequently asked questions

A p value is the probability of seeing data at least as extreme as the trial's if there were truly no effect and every assumption of the analysis held. A 95% confidence interval is the range of effect sizes reasonably compatible with the data. The p value answers "is this surprising under no effect?"; the interval answers "how big might the effect be?" Only the second tells you whether a result matters, and a small p value never shows that an effect is large, real or important.

A printed bell-shaped curve with shaded tails beside a ruler and pencil
Illustration: a p-value is an area in the tails of a distribution.

What does a p value actually mean?

Every p value is calculated against a reference point, usually the null hypothesis of no difference between groups. The calculation asks: if the null were true, and the data were collected and analysed exactly as planned, how often would chance alone produce a difference this large or larger? A p value of 0.03 means about 3% of the time.

That is a narrow statement, and most everyday readings of it are wrong. Greenland and colleagues listed 25 common misinterpretations of p values, confidence intervals and power, and stressed that analyses chosen because of the p values they produce can give small p values even when nothing is there. The commonest errors:

StatementTrue or false?
P = 0.03 means a 3% chance the null hypothesis is trueFalse. The p value assumes the null is true; it cannot also measure its probability
P = 0.03 means a 97% chance the treatment worksFalse, for the same reason
P < 0.05 means the effect is large or importantFalse. It says nothing about size
P > 0.05 means there is no effectFalse. It means the data could not rule out no effect
Two trials with P = 0.04 and P = 0.06 disagreeUsually false. Their estimates may be almost identical
The p value depends on the analysis plan being followedTrue. Flexible analysis invalidates it

What does a 95% confidence interval mean?

A confidence interval is built from the same data and the same assumptions, but it reports a range rather than a verdict. Roughly: if the study were repeated many times and an interval calculated each time, 95% of those intervals would contain the true effect. In practice, read it as the span of effects the data do not contradict.

The interval carries three pieces of information a p value hides. Its centre is the best estimate of the effect. Its width shows precision: narrow means the study pinned the effect down; wide means it did not. And its ends show the largest and smallest effects still compatible with the data. Whether an interval includes the no-effect value (zero for a difference, one for a ratio) tells you what the p value would have said. The rest is extra.

For ratios, hazard ratios and how the interval relates to absolute risk, see relative vs absolute risk, which works through SURPASS-CVOT's interval in detail.

Why is "statistically significant" not the same as important?

Because significance depends on two things at once: the size of the effect and the size of the study. A large trial can make a small effect highly significant. Two phase 3 programmes on compounds discussed on this site show the same p value attached to very different effects.

TRIUMPH-1, retatrutide. In 2,339 adults with obesity, mean weight change at the two higher doses was −23.7% and −25.0%, against −3.9% on placebo. The differences were −19.8 and −21.0 percentage points, P<0.001 for both. Retatrutide is investigational and approved nowhere.

RECONNECT, bremelanotide. In two phase 3 trials of 1,267 premenopausal women with hypoactive sexual desire disorder, bremelanotide raised the desire domain score of a standard questionnaire by 0.30 and 0.42 points more than placebo. The P values were below 0.001. Bremelanotide is approved in the United States, as Vyleesi, for that one indication; research-chemical PT-141 is not that medicine.

Both results are "highly significant". One is a difference of about a fifth of body weight. The other is a fraction of a point on a scale whose range, and whose smallest change that patients notice, the abstract does not state. Deciding whether 0.30 points matters needs exactly that external yardstick. The p value cannot supply it. The PT-141 research page covers what that trial programme measured.

What does a non-significant result mean?

It means the data could not exclude no effect. Whether that is informative depends entirely on the confidence interval.

An informative null: MMPOWER-3. This phase 3 trial randomised 218 people with primary mitochondrial myopathy to elamipretide or placebo. The difference in six-minute walk distance was −3.2 metres, with a 95% confidence interval of −18.7 to 12.3 metres (p = 0.69). That interval is narrow enough to say something: a benefit larger than about 12 metres is not compatible with the data. The trial did not meet either primary endpoint. Elamipretide holds a US accelerated approval for Barth syndrome only; it is not approved for mitochondrial myopathy, as the SS-31 overview explains.

An uninformative null: ipamorelin and postoperative ileus. A phase 2 proof-of-concept trial enrolled 117 patients after bowel resection. Median time to the first tolerated solid meal was 25.3 hours with ipamorelin and 32.6 hours with placebo (p = 0.15). A seven-hour difference was neither demonstrated nor excluded; the authors noted the study was small and the patients varied widely. The abstract reports no interval, so a reader cannot tell how large a benefit, or harm, the data still allow. Ipamorelin is not approved anywhere.

Same verdict, "not significant", and two very different states of knowledge. "No significant difference" is a description of the data, not evidence of no effect. Showing that two treatments are equivalent takes a trial designed and powered to test equivalence.

What is multiplicity, and why does it matter?

Every test at the 0.05 threshold carries a 5% chance of a false positive when nothing is happening. Run 20 independent tests on a useless intervention and the chance that at least one comes out significant is about 64% (one minus 0.95 to the twentieth power). Multiplicity arises from many outcomes, many time points, many dose groups, many subgroups and many ways of analysing the same data.

Bender and Lange set out the principle: confirmatory studies need adjustment for multiple testing whenever several tests feed one conclusion. Well-run trials handle this in advance. TRIUMPH-1 named its primary outcomes before unblinding, weight, knee pain and sleep apnoea events, each tested at two doses against placebo. That lets six comparisons be read as confirmatory rather than as six chances to win.

Subgroups are the most common route to a false finding. Sun and colleagues proposed five tests of a credible subgroup claim. Can chance explain it? Is it consistent across studies? Was it one of a few hypotheses specified in advance, with direction? Is there strong prior biological support? And does it rest on comparisons within studies rather than between them? Most "works especially well in…" claims fail several.

Undeclared flexibility is worse, because nobody can count the tests. Simmons, Nelson and Simonsohn showed by simulation and experiment that ordinary choices made after seeing data can push false-positive rates far above the nominal 5%. Those choices include when to stop collecting, which measures to report and which covariates to include. This is the reason registration matters, and the reason unregistered preclinical work deserves caution; see publication bias explained.

Where do 0.05 and 95% come from?

Convention, not mathematics. Nothing changes in nature between p = 0.049 and p = 0.051, yet the threshold decides which results are called findings, published and remembered. Amrhein, Greenland and McShane argued in Nature for retiring the dichotomy of "significant" and "non-significant" altogether, in favour of reporting and interpreting the interval. Whatever one thinks of that proposal, it changes how to read a paper. Treat the threshold as a rough flag and look at the estimate and its interval.

A checklist for any reported result

  • What is the effect estimate, in units you understand?
  • What is the confidence interval, and does it include effects too small to matter, or large enough to worry about?
  • Was this the prespecified primary outcome, or one of many?
  • How many comparisons were made, and were they adjusted?
  • Is a subgroup claim credible on Sun's five tests?
  • If not significant, does the interval exclude a meaningful effect, or is the study simply too small?

Whether a statistically sound result is also causal is a separate question, taken up in correlation vs causation. Whether a marker result means a clinical one is covered in surrogate endpoints explained. Where these numbers sit on a PubMed record is shown in how to read a PubMed abstract. And why small studies that reach significance overstate effects is covered in how to read a peptide study. Nothing here is medical, dosing, or therapeutic guidance.

Frequently asked questions

Is a p value of 0.001 stronger evidence than 0.04? It is stronger evidence against the null hypothesis, under the analysis's assumptions. It is not evidence of a bigger or more important effect. A very large trial can produce p = 0.001 for a trivial difference. Compare the effect estimates and their intervals, not the p values, when judging which result matters more.

Can a confidence interval replace a p value? Mostly. An interval shows everything a p value does, whether the no-effect value is excluded, plus the size and precision of the estimate. Many methodologists recommend leading with the interval. The p value adds little, except in formal hypothesis testing with a prespecified threshold, such as a trial's primary endpoint.

Does a 95% interval mean a 95% chance the true effect is inside it? Not strictly. The 95% describes the method: across many repetitions, 95% of intervals built this way would capture the true value, if the model is right. Any one interval either contains it or does not. Reading it as the range of effects the data comfortably allow is close enough for practical use.

What is p-hacking? Analysing data in many ways and reporting the version that crosses p < 0.05. Examples include adding participants until significance appears, dropping outliers selectively and switching outcomes. Each choice may look defensible alone; together they make the reported p value meaningless. Prespecified, registered analysis plans are the main defence, and comparing a paper against its registration exposes it.

Selected Research References

  1. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. pubmed.ncbi.nlm.nih.gov/27209009
  2. Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-307. pubmed.ncbi.nlm.nih.gov/30894741
  3. Bender R, Lange S. Adjusting for multiple testing--when and how? Journal of Clinical Epidemiology. 2001;54(4):343-349. pubmed.ncbi.nlm.nih.gov/11297884
  4. Sun X, Ioannidis JP, Agoritsas T, et al. How to use a subgroup analysis: users' guide to the medical literature. JAMA. 2014;311(4):405-411. pubmed.ncbi.nlm.nih.gov/24449319
  5. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011;22(11):1359-1366. pubmed.ncbi.nlm.nih.gov/22006061
  6. Jastreboff AM, Kaplan LM, Davies MJ, et al.; TRIUMPH-1 Investigators. Retatrutide, a Triple Hormone Receptor Agonist, for Treatment of Obesity. New England Journal of Medicine. 2026. pubmed.ncbi.nlm.nih.gov/42814954
  7. Kingsberg SA, Clayton AH, Portman D, et al. Bremelanotide for the Treatment of Hypoactive Sexual Desire Disorder: Two Randomized Phase 3 Trials. Obstetrics & Gynecology. 2019;134(5):899-908. pubmed.ncbi.nlm.nih.gov/31599840
  8. Karaa A, Bertini E, Carelli V, et al. Efficacy and Safety of Elamipretide in Individuals With Primary Mitochondrial Myopathy: The MMPOWER-3 Randomized Clinical Trial. Neurology. 2023;101(3):e238-e252. pubmed.ncbi.nlm.nih.gov/37268435
  9. Beck DE, Sweeney WB, McCarter MD. Prospective, randomized, controlled, proof-of-concept study of the Ghrelin mimetic ipamorelin for the management of postoperative ileus in bowel resection patients. International Journal of Colorectal Disease. 2014;29(12):1527-1534. pubmed.ncbi.nlm.nih.gov/25331030

All products sold by NovaEvum are for research and laboratory use only. They are not intended for human or animal consumption, diagnosis, treatment, cure, or prevention of any disease.