This example starts with a chi-square but ends with a lesson on how even well-written prompts can result in hallucinations.

A research study counted how often ChatGPT made up citations for three different categories of mental disorders (binge eating, body dysmorphic, and major depressive). They used a chi-square to determine if rates of made up citations differed by disorder (they do).

If ever there was an article that belonged on this blog, this is it. You can use it in your stats class as an example of chi-square and/or as a warning to students if you ask them to perform literature reviews for your class.

The original paper, Influence of topic familiarity and prompt specificity on citation fabrication in mental health research using large language models: Experimental Study was published in December 2025, and summarized by PsyPost shortly after publishing.

What the researchers did:

Methods: In June 2025, GPT-4o was prompted to generate 6 literature reviews (~2000 words; ≥20 citations) on 3 disorders representing different levels of public awareness and research coverage: major depressive disorder (high), binge eating disorder (moderate), and body dysmorphic disorder (low). Each disorder was reviewed at 2 levels of specificity: a general overview (symptoms, impacts, and treatments) and a specialized review (evidence for digital interventions). All citations were extracted (N=176) and systematically verified using Google Scholar, Scopus, PubMed, WorldCat, and publisher databases. Citations were classified as fabricated (no identifiable source), real with errors, or fully accurate. Fabrication and accuracy rates were compared by disorder and review type by using chi-square tests.

What the researchers found:

Results: Across the 6 reviews, GPT-4o generated 176 citations; 35 (19.9%) were fabricated. Among the 141 real citations, 64 (45.4%) contained errors, most frequently incorrect or invalid digital object identifiers. Fabrication rates differed significantly by disorder (χ22=13.7; P=.001), with higher rates for binge eating disorder (17/60, 28%) and body dysmorphic disorder (14/48, 29%) than for major depressive disorder (4/68, 6%). While fabrication did not differ overall by review type, stratified analyses showed higher fabrication for specialized versus general reviews of binge eating disorder (11/24, 46% vs 6/36, 17%; P=.01). Accuracy rates also varied by disorder (χ22=11.6; P=.003), being lowest for body dysmorphic disorder (20/34, 59%) and highest for major depressive disorder (41/64, 64%). Accuracy rates differed by review type within some disorders, including higher accuracy for general reviews of major depressive disorder (26/34, 77% vs 15/30, 50%; P=.03)

How to use in class:

1. This is a good chi-square results section. They shared the test value and the p value, of course, but I like how they shared the varying rates of inaccuracy as absolute data and percentages throughout. Chi-squares can be tricky to present in text (versus a table) and the authors did a good job here.

2. If you are talking to your students about proper use of AI: These researchers shared their exact prompts in their supplemental material. This demonstrates a) proper, ethical citations of prompts when using AI in research and that b) the well-written prompts still resulted in bogus data.

Do you like this example? Imagine how much you would love a whole textbook filled with engaging, timely examples like this one: https://wwnorton.com/books/9781324102052

Comments

Ways to use funny meme scales in your stats classes

Have you ever heard of the theory that there are multiple people worldwide thinking about the same novel thing at the same time? It is the multiple discovery hypothesis of invention . Like, multiple great minds around the world were working on calculus at the same time. Well, I think a bunch of super-duper psychology professors were all thinking about scale memes and pedagogy at the same time. Clearly, this is just as impressive as calculus. Who were some of these great minds? 1) Dr. Molly Metz maintains a curated list of hilarious "How you doing?" scales. 2) Dr. Esther Lindenström posted about using these scales as student check-ins. 3) I was working on a blog post about using such scales to teach the basics of variables. So, I decided to create a post about three ways to use these scales in your stats classes: 1) Teaching the basics of variables. 2) Nominal vs. ordinal scales. 3) Daily check-in with your students. 1. Teach your students the basics...

Leo DiCaprio Romantic Age Gap Data: UPDATE

Does anyone else teach correlation and regression together at the end of the semester? Here is a treat for you: Updated data on Leonardo DiCaprio, his age, and his romantic partner's age when they started dating. A few years ago, there was a dust-up when a clever Redditor r/TrustLittleBrother realized that DiCaprio had never dated anyone over 25. I blogged about this when it happened. But the old data was from 2022. Inspired by this sleuthing, I created a wee data set, including up-to-date information on his current relationship with Vittoria Ceretti, so your students can suss out the patterns that exist in this data.

If your students get the joke, they get statistics.

Gleaned from multiple sources (FB, Pinterest, Twitter, none of these belong to me, etc.). Remember, if your students can explain why a stats funny is funny, they are demonstrating statistical knowledge. I like to ask students to explain the humor in such examples for extra credit points (see below for an example from my FA14 final exam). Using xkcd.com for bonus points/assessing if students understand that correlation =/= causation What are the numerical thresholds for probability? How does this refer to alpha? What type of error is being described, Type I or Type II? What measure of central tendency is being described? Dilbert: http://search.dilbert.com/comic/Kill%20Anyone Sampling, CLT http://foulmouthedbaker.com/2013/10/03/graphs-belong-on-cakes/ Because control vs. sample, standard deviations, normal curves. Also,"skewed" pun. If you go to the original website , the story behind this cakes has to do w...

Search This Blog