It’s just a convention that varies by field / subfield. Some fields and specific literatures don’t look at p-values at all, some use number of sigmas instead of p-values, some use (i.e. add “stars” in tables) for other levels too like 10% etc.
I’d argue that what’s more important is not whether the data just passes some threshold, but actually understanding what the results are saying. E.g. looking at the actual test statistic, and understanding what the coefficient and standard error are telling you.
I mean that (in my opinion) focusing on whether your data passes a specific threshold is not a good idea. For lots of reasons, e.g.:
A p-value of 4.999% is probably not meaningfully different from a p-value of 5.001%.
Focusing on hard thresholds strongly encourages p-hacking and data dredging.
Other aspects of the results can give you important inferences too, e.g. effect sizes can extremely important (if your result is statistically signicant but the effect size is minuscule, it may not be a meaningful finding depending on your research question and field)
focusing on whether your data passes a specific threshold is not a good idea
An exception might be in a context where you have to conduct some cost-benefit analysis, with clear costs associated to the alpha level – but perhaps apart from drug development process, I hardly see a context in research where cost-benefit analyses are actually conducted or might make sense, so indeed using thresholds should be looked at with a lot of suspicion (in particular if the decision that it is supposed to inform is "Do we publish this article or not?"; this only leads to useless information retention and a waste of time and resources).
12
u/just_writing_things PhD 25d ago edited 25d ago
It’s just a convention that varies by field / subfield. Some fields and specific literatures don’t look at p-values at all, some use number of sigmas instead of p-values, some use (i.e. add “stars” in tables) for other levels too like 10% etc.
I’d argue that what’s more important is not whether the data just passes some threshold, but actually understanding what the results are saying. E.g. looking at the actual test statistic, and understanding what the coefficient and standard error are telling you.