P-Hacking Enables Scientists To Publish False Results As True

Brilliance | Dec. 16, 2017

P-hacking: just the word sounds nefarious. But to understand just how nefarious it is, you first need to understand a statistical concept known as the p-value. Consider a simple scientific experiment, one to determine whether a certain drug helps lower blood pressure. All of the study participants take pills, but one group gets pills containing the drug and the other gets sugar pills—the placebo, or control group. Scientists record the results of the two groups, then put them through statistical analysis and come up with a p-value. This number, which is always between 0 and 1, helps the scientists determine whether any differences they see between the experimental and control groups are due to a real effect or random chance. The lower the number, the less likely it is that the result could have arisen in any way except for the drug actually working; the higher the number, the more likely it is that the drug isn't having a measurable effect on blood pressure. A low enough number—less than 0.05 or 0.01, depending on the field of study—is considered "statistically significant," and will give that study a better chance of being published.

The problem is that scientific experiments are rarely this simple: there are many different experimental groups with a huge amount of data and many possible statistical formulas to choose from. The most rock-solid studies set out with a plan for exactly what data and what statistical techniques will be used before the experiment begins. But whether due to poor training, sloppiness, or plain corruption, some scientists end up doing it a different way: after the study is complete, they pick and choose certain pieces of data and statistical techniques until they find the one that gets them to statistical significance. That's p-hacking. (Try it yourself with FiveThirtyEight's interactive p-hacking calculator). It's similar to a game of football where one team gets to decide when the game ends, instead of playing for the pre-determined amount of time.

Unfortunately, without expertise in statistics and a whole lot of time on your hands, it's virtually impossible to spot p-hacking in a scientific study. To fight this troubling trend, some journals are reducing their reliance on p-values, focusing instead on other statistical elements that tell more of the story. Delve further into the realm of scientific research with the videos below.

Hot Comments
You're the first to comment
Say something.
Open app to add comment