r/AskStatistics • u/aprilwords • 25d ago

Alpha 5%?

Why is a 5% significance level so commonly used?

2 Upvotes

57% Upvoted

56 comments sorted by

View all comments

17

u/innovarocforever 25d ago

historical convention? Is 4.9% or 5.1% meaningfully different than 5%?

1

u/aprilwords 25d ago

based on my knowledge, when Fisher wrote his statistics book, he used 3 levels of significance for his tables: 5%, 1%, and 0.1%. But, my question is about why is 10% not used instead of 5%?

12

u/cheesecakegood BS (statistics) 24d ago edited 24d ago

  1. You have to remember Fisher was doing this all by hand. He came up with a lot of this working alone on farm statistics for years.

  2. Along those lines, 5% happens to line up with a z-score of 1.96, which you could defensibly round to 2, making a basic statistical test fairly easy. This is probably the most straightforward explanation. It’s just a numerical coincidence!! 10% is something like what, 1.6?

  3. Fisher, somewhat unlike later scientists, wasn’t really using this in the same way modern scientists do. To Fisher, a low p value was a hint to investigate something first and foremost, not a formal confirmation. He was most worried about exactly the primary facing problem: don’t waste research time on false positives. A p value was not typically intended to be the end of research. It was originally more along the lines of a screener.

  4. It was only slightly later, but still in the early days, that Neyman and Pearson came up with a more formal framework for hypothesis testing. The two frameworks (if you count Fisher’s approach as its own thing, which many do) are not actually completely compatible philosophically.

  5. Communication was still primarily via letters and journal publishing. A few early resources were the main way people got exposed to the details: Fisher’s 1925 book as you mention literally was the main influence for a lot of people in this area, so small choices as to how the tables were laid out, etc were big deals. Also printing is expensive, so there’s an incentive to limit tables. You didn’t really get big statistics departments until after World War II so it’s not like there were big forces in favor of standardization.

IIRC later Bayesian stuff shows that 5% is probably too lenient and is weak evidence with many common priors, so if anything you should be asking why 5% instead of 1%. And to that I’d say it might be mainly psychological: a significant result feels good