My favorite examples of bad data visualizations are the ones that use accurate data that was actually collected through seemingly ethical means but totally malign the data. The numbers are correct, the data viz is...not very truthy ( I'm looking at you, Florida. ) Especially when you mess up the data viz in a way that appears to be deliberate AND doesn't really strengthen your point. I'm also looking at you, The White House. Here is a story of a deliberate but pointless massaging of a y-axis. A story in Three Tweets. 1. The Biden Administration is doing a good job of encouraging economic growth, right? Take a gander at this bar graph. 2021 was a success...just look at the chart.  2. BUT WAIT. What's this? That y-axis is shady. I...just can't think of any software/glitch that could make this mistake by accident. ALSO: If you like Twitter, follow Graph Crimes.  3. The White House issues a correction featuring a pretty good data put, I would say.  FIN

Stuart Ritchie wrote an excellent book about the problems (and solutions to those problems) in science called Science Fictions . Illustrator and author Zach Weinersmith summarized and illustrated those problems in science in the form of a short webcomic . Both the book and the comic are great and have a home somewhere in your psychology curriculum. The comic is a quick, digestible primer on the problems with science. Meanwhile, the book goes into great depth, including many of the problems related directly to p-hacking, fishing expeditions, etc.  For more, read Stuart's book, Science Fictions. Aside: I am American, and he is Scottish and reads the audiobook version to me while I walk my dog, and it is soothing. I'm creating a one-credit course for our Honors program based on this book.  Also, follow Stuart and Zach on The Bird App.

I teach many, many statistics classes. In addition to the core topics typically covered in Introductory Statistics, I think covering real-life controversies involving statistics is vital. Usually, these are stories of large organizations that attempted to bias/PR attack/skew/p-hack/cherry-pick data to serve their own purposes.  I believe that these examples serve to show why data literacy is so critical because data is used in so many fields, AND our students must prepare themselves to evaluate data-based claims throughout their lives. I put out a call on Twitter , and my friends there helped me generate a great list of such controversies. I put this list into a spreadsheet with links to primers on each topic. This isn't an in-depth study of any of these topics, but the links should get you going in the right direction if you would like to use them in class. I hope this helps my fellow stats teachers integrate more applied examples into their classes. If you h...

The Knot, a wedding planning website, collected data on the amount of money that brides and grooms spend on items for their weddings. They shared this information, as well as the average cost of a wedding in 2017. See the infographic below: BUT WAIT! If you dig into this data and the methodology, you'll find out that they only collected price points from couples who ACTUALLY PAID FOR THOSE ITEMS. https://xogroupinc.com/press-releases/the-knot-2017-real-weddings-study-wedding-spend/ Problems with this data to discuss with your students: 1) No one who got stuff for free/traded for stuff would have their $0 counted towards the average. For example, one of my cousins is a tattoo artist and he traded tattoos for use of a drone for photos of their outdoor wedding. 2) AND...if you didn't USE a service, your $0 wasn't added to their ol' mean value. For example, we had our wedding and reception at the same location, so we spent $0 on a ceremony site. 3) As poi...

Geckoboard created a list of common statistical fallacies , including cherry picking, Simpson's paradox, gerrymandering, and many more. Each fallacy comes with a brief description of the fallacy, references, a printable card for review/display, and drawing. They are kind of gorgeous and to the point and helpful. https://www.geckoboard.com/learn/data-literacy/statistical-fallacies/sampling-bias/ Here is the downloadable card for the Regression Toward the Mean: https://www.geckoboard.com/assets/regression-toward-the-mean.pdf They even present all of their graphics as  a free, downloadable poster . My only peeve is that they use the term "Data Dredging" where I would have said "HARKing" or "Going on a fishing expedition". And that is just the tiniest of peeves, I think this is a good check list filled with images and concise descriptions that would look beautiful in a college professor's office, a stats class room, or anonymously ...

NPR, as part of their series about discrimination in America , talked about how it is difficult to obtain a diverse research sample when your diverse research sample doesn't trust scientists. This story by Rob Stein is about public outreach attempts in order to gather a representative sample for a large scale genetic research study. The story is also about how historical occurrences of research violations live on in the memory of the affected communities. The National Institutes for Health is trying to collect a robust, diverse sampling of Americans as part of the All of Us initiative. NIH wants to build a giant, representative database of Americans and information about their health and genetics. As of the air date for this story, African Americans were underepresented in the sample, and the reason behind this is historical. Due to terrible violation of African American research participant rights (Tuskeegee, Henrietta Lacks), many African Americans are unwilling to partic...

I love good graphic design and lists. This guide to spotting bad science embraces both. And many of the science of bad science are statistical in nature, or involve sketchy methods. Honestly, this could be easily turned into a  homework assignment for research evaluation. This comes from the Compound Interest ( @compoundchem ), which has all sorts of beautiful visualizations of chemistry topics, if that is your jam. 

This NPR story discusses research  detective work published JAMA . The JAMA article looked at a very influential NEJM review article that investigated the link between diet and Coronary Heart Disease. Specifically, whether sugar or fat contribute more to CHD. The article, written by Harvard researchers decades ago, pinned CHD on fatty diets. But the researchers took money from Big Sugar (which sounds like...a drag queen or CB handle) and communicated with Big Sugar while writing the review article. This piece discusses how conflict of interest shaped food research and our beliefs about the causes of CHD for decades. And how conflict of interest and institutional/journal prestige shaped this narrative. It also touches on how industry, namely sugar interests, discounted research that finds a sugar:CHD link while promoting and funding research that finds a fat:CHD link. How to use in a Research Methods class: -Conflict of interest. The funding received by the researchers from th...

This is a nice, concise hand out from APA that touches on the main points for evaluating research. In particular, research that has been distilled by science reporters. It may be a bit light for a traditional research methods class, but I think it would be good for the research methods section of most psychology electives, especially if your students working through source materials. The article mostly focuses on evaluating for proper sampling techniques. They also have a good list of questions to ask yourself when evaluating research: This also has an implicit lesson of introducing the APA website to psychology undergraduates and the type of information shared at APA.org. (including, but not limited to, this glossary of psychology terms .)

Brenner, writing for Distractify , has compiled a very healthy list of terrible, terrible graphs and charts . How to use in class: 1) Once you know how NOT to do something, you know how to do it. 2) Bonus points for pointing out the flaws in these charts...double bonus points for creating new charts that correct the incorrect charts. A few of my favorites:

This NPR story is a summary of the decisively titled " The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses " authored by Dr. John Ioannidis. The NPR story provides a very brief explanation of meta-analysis and systematic reviews. It explains that they were originally used as a way to make sense of many conflicting research findings coming from a variety of different researchers. But these very influential publications are now being sponsored and possibly influenced by Big Pharma. This example explains conflicts of interest and how they can influence research outcomes. In addition to financial relationships, the author also cites ideological allegiances as a source of bias in meta-analysis. In addition to Dr. Ioannidis, Dr. Peter Kramer was interviewed. He is a psychiatrist who defends the efficacy of antidepressants. He suggests that researchers who believe that placebos are just as effective as anti-depressants tend to analy...

This syllabus for  PSY 607: Everything is Fucked ,  made the rounds last week. The syllabus is for a course that  purports  that science is fucked. The course readings are a list of articles and books that hit on the limitations of statistics and research psychology ( p -values, shortcomings of meta-analysis, misuse of mediation, replication crisis, etc.). PSY 607 isn't an actual class ( as author/psychologist/blogger Srivastava explains in this piece from The Chronicle ) but it does provide a fine reading list for understanding some of the current debates and changes in statistics and psychology.  Most of articles are probably too advanced for undergraduates but perfectly appropriate for teaching graduat e students about our field and staying up to date as instructors of statistics. Here is a link to the original blog post/syllabus. 

I love research that is counterintuitive. It is interesting to me and makes a strong, memorable example for the classroom. That's why I'm recommending Carroll's piece  from the NYT. It questions the conventional wisdom that breakfast is the most important meal of the day. As Carroll details, there is a long standing and strong belief in nutrition research claiming that breakfast reduces obesity and leads to numerous healthy outcomes. But most nutrition research is correlational, not causal. AND there seems to be an echo-chamber effect, such that folks are miss-citing previous nutrition research to bring it in line with the breakfast research. Reasons to use this article as a discussion piece in your statistics or research methods course: -Highlights the difference between correlation and causation -Provides an easy to understand example of publication bias ("no breakfast = obesity" is considered a fact, studies that found the opposite were less likely to...

In my Honors Statistics course, we have discussion days over the course of a semester. One of the discussion topics involves instances when the media has skewered research results (for another example, see this story about  fitness trackers ,) Jezebel writer Caroline Weinberg   describes a  modest study  that found that people who have at least one previous tattoo experience a boost in their immunity when they get subsequent tattoos, as demonstrated via saliva samples of Immunoglobulin A. This is attributed to the fact that compared to tattoo newbies, tattoo veterans don't experience a cortisol reaction following the tattoo. Small sample size but a pretty big effect. So, as expected, the media exaggerated these effects...but mostly because the researcher's university's marketing department did so first. Various new outlets stated things like  "Sorry, Mom: Getting lots of tattoos could have surprising health benefits"  and  "Getting multip...

Annie Minoff delves into the sins of ad hoc hypotheses using several examples from evolutionary science (including evolutionary psychology) . I think this is a fun way to introduce this issue in science and explain WHY a hypothesis is important for good research. This article provides three ways of conveying that ad hoc hypotheses are bad science. 1) This video of a speaker lecturing about absurd logic behind ad hoc testing (here, evolutionary explanations for the mid-life "spare tire" that many men struggle with). NOTE: This video is from an annual event at MIT, BAHFest (Bad Ad Hoc Fest) if you want more bad ad hoc hypotheses to share with students. 2) A quiz in which you need to guess which of the ad hoc explanations for an evolutionary finding is the real explanation. 3) A more serious reading to accompany this video is Kerr's HARKing: Hypothesizing after results are known (1998), a comprehensive take down of this practice.

 Here is another example of how data visualizations can be accurate AND misleading. I Fucking Love Science broke down a brief Twitter war that started after National Review tweeted the following post in order to argue that global climate change isn't a thing. Note: The y-axis ranged from 110 - -10 degrees Fahrenheit. True, such a temperature range is experienced on planet Earth, but using such an axis distracts from the slow, scary march that is global climate change and doesn't do a very good job of illustrating how discrete changes in temperature map onto increased use of fossil fuels in the increasingly industrialized world. Twitter-verse responded thusly:

This headline is a good example of a) journalists misrepresenting statistics as well as b) confidence intervals/margin of error more broadly. See the headline below: In actuality, Bernie didn't exactly take the lead over Hillary Clinton. Instead, a Quinnipiac poll showed that 41% of likely Democratic primary voters in Iowa indicated that they would vote for Sanders, while 40% reported that they would vote for Clinton. If you go to the original Quinnipiac poll , you can read that the actual data has a margin of error of +/- 3.4%, which means that the candidates are running neck and neck. Which, I think, would have still been a compelling headline.  I used this as an example just last week to explain applied confidence intervals. I also used this as a round-about way of explaining how confidence intervals are now being used as an alternative/compliment to p -values. 

Aschwanden (for fivethirtyeight.com) did an extensive piece that summarizes that data/p-hacking/what's wrong with statistical significance crisis in statistics. There is a focus on the social sciences, including some quotes from Brian Nosek regarding his replication work. The report also draws attention to  Retraction Watch  and Center for Open Science as well as retractions of findings (as an indicator of fraud and data misuse). The article also describes our funny bias of sticking to early, big research findings even after those research findings are disproved (example used here is the breakfast eating:weight loss relationship). The whole article could be used for a statistics or research methods class. I do think that the p-hacking interactive tool found in this report could be especially useful illustration of How to Lie with Statistics. The "Hack your way to scientific glory" interactive piece demonstrates that if you fool around enough with your operationalized...

Mersereau, writing for Gawker website The Vane, provides  another example of How Not To Graph. Or How To Graph As To Not Lie About Data But Make Your Data Look More Impressive Than Is Ethical. Weather Underground (AKA Wunderground, weather forecasting service/website) was bragging about it's accuracy compared to the competition. At first glance (see below), this graph seems to reinforce the argument...until you take a look at the scale being used. The beginning point on the X axis is 70, while the high point is 80. So, really, the differences listed probably don't even approach statistical significance. This story, somewhat randomly, also includes some shady graphs created by Fox News. I don't understand the need for the extra Fox News graphs, but they also illustrate how one can create graphs that have accurate numbers but still manage to twist the truth.

The original idea for using this article this way comes from Dr. Susan Nolan 's presentation at NITOP 2015, entitled " Thinking Like a Scientist: Critical Thinking in Introductory Psychology."  I think that Dr. Nolan's idea is worth sharing, and I'll reflect a bit on how I've used this resource in the classroom. (For more good ideas from Dr. Nolan, check out her books, Psychology , Statistics for the Behavioral Sciences , and The Horse that Won't Go Away (about critical thinking)). Last summer, the National Academy of Sciences Proceedings published an article entitled "Experimental evidence of massive-scale emotional contagion through social networks ." The gist: Facebook manipulated participants' Newsfeeds to increase the number of positive or negative status updates that each participant viewed. The researchers subsequently measured the number of positive and negative words that the participants used in their own status updates. They fou...