Monday, September 28, 2015

Aschwanden's "Science is broken, it is just a hell of a lot harder than we give it credit for"

Aschwanden (for did an extensive piece that summarizes that data/p-hacking/what's wrong with statistical significance crisis in statistics. There is a focus on the social sciences, including some quotes from Brian Nosek regarding his replication work. The report also draws attention to Retraction Watch and Center for Open Science as well as retractions of findings (as an indicator of fraud and data misuse). The article also describes our funny bias of sticking to early, big research findings even after those research findings are disproved (example used here is the breakfast eating:weight loss relationship).

The whole article could be used for a statistics or research methods class. I do think that the p-hacking interactive tool found in this report could be especially useful illustration of How to Lie with Statistics.

The "Hack your way to scientific glory" interactive piece demonstrates that if you fool around enough with your operationalized variables, you can probably, eventually, find a way to support your argument (AKA p-hacking). This example uses the question of whether to economy does better under Republican or Democratic leadership to prove a point. Within the tool, you can operationalize both leadership and measures of economic health in a number of different ways (via the check boxes on the left), resulting in a number of different outcomes.

 This is whole article is useful in the classroom simply as a way to discuss the shortcomings of p-values (with the interactive piece) but I think the whole article is accessible to undergraduates, especially as it is littered with embedded links that provide greater information to previous research scandals and debates.