This story by Christine Aschwanden from 538.com describes the limitations of a JAMA article. That JAMA article describes a research project that found signs of Chronic Traumatic Encephalopathy (CTE) in 110 out of 111 brains of former football players.
How to use in stats and research methods:
1) It is research, y'all.
2) One of the big limitations of this paper comes from sampling.
3) The 538 article includes a number of thought experiments that grapple with the sampling distribution for all possible football players.
4) Possible measurement errors in CTE detection.
5) Discussion of replication using a longitudinal design and a control group.
The research:
The JAMA article details a study of 111 brains donated by the families deceased football players. They found evidence of CTE in 110 of the brains.
Which sounds terrifying if you are a current football player, right? But does this actually mean that 110 out of 111 football players will develop CTE? Not necessarily, if you consider sampling bias and the sampling distribution of the sample mean.
Is this sample a random sample of football players? Why do you think these 111 former football players/their next of kin went out of their way to donate their brains to science? Were these men experiencing symptoms prior to death? From Aschwanden:
As such, we really need to consider this sample within the bigger idea of all possible samples that could be generated from the true population of football players. This article never, ever refers to the sampling distribution by name, but does write out some interesting thought experiments that are referring to the concept of the sampling distribution.
Beyond sampling bias, how are we diagnosing CTE? Some criticism has been levied at the original paper for being to lax in diagnosing CTE. This could be a good way to discuss operationalizing and measuring any dependent variable.
Finally, there is nothing like a good control group or longitudinal research design to truly establish a causal relationship:
