Reading the structure of a paper

Selection, not fabrication

When a manufacturer-funded trial reports a positive result, the trial itself is usually well-conducted. The bias sits one layer above: in which trials were run at all, which were stopped early, which were buried when they returned negative, which were spun up post-hoc to find a secondary endpoint that crossed significance. Reboxetine, Tamiflu, statins for primary prevention; same pattern each time. The published literature on any drug at any given moment is the survivor’s lottery; the unpublished trials are usually the ones that disagreed.

“Reboxetine is a drug I myself have prescribed. Other drugs had done nothing for this particular patient, so we wanted to try something new. I’d read the trial data before I wrote the prescription, and found only well-designed, fair tests, with overwhelmingly positive results. Reboxetine was better than placebo, and as good as any other antidepressant in head-to-head comparisons.”

“Seven trials had been conducted comparing reboxetine against placebo. Only one, conducted in 254 patients, had a neat, positive result, and that one was published in an academic journal, for doctors and researchers to read. But six more trials were conducted, in almost ten times as many patients. All of them showed that reboxetine was no better than a dummy sugar pill. None of these trials was published. I had no idea they existed.”

Source: Goldacre, Bad Pharma (2012), Chapter 1. The doctor who acted in good faith on the published literature acted on the survivor’s lottery. The seventh trial was the published one because it was the one that survived the bin.

The cover-the-options test

When a headline reads “X linked to Y”, ask what was measured (causal mechanism? correlation? hazard ratio? absolute risk reduction?). Most of the time the underlying paper supports a much weaker claim than the headline carries. Goldacre’s I Think You’ll Find is a column-by-column tour of this gap.

Relative vs absolute risk

A statin halves the risk of a second heart attack in a particular five-year window. The relative reduction is 50%. The absolute reduction is 2.4 percentage points (from, say, 4.8% to 2.4%). The relative number is shareable on social media; the absolute number is the one that matters to the person taking the medication. Reporting relative without absolute is, per Goldacre, the single most common journalist mistake in medical reporting.

The healthy-user bias

Trial cohorts are not the general population. STEP-1 (Wilding 2021) enrolled adults motivated enough to attend 17 clinic visits across 68 weeks; they were not the population at large. Real-world weight-loss outcomes from semaglutide outside the trial cohort are smaller, because the people taking it outside a trial are not the people who completed the trial. AID systems show this pattern starkly: the pivotal-trial 78% TIR drops to 56% TIR in real-world cohorts within twelve months, because real users miss carbs, run hot, sleep badly, take antibiotics, change cannulas at the wrong time, live their lives. The trial is real; the trial is not the user.