Can You Trust Research on Psychology Topics?
If you have ever looked up research on psychology topics, you may have come across the 1971 Stanford Prison Experiment, in which a mock prison with participants taking on the roles of guards and prisoners was reportedly shut down early due to cruel behavior from the guards. Or perhaps you saw a more contemporary study on how to increase honesty in filling documents, titled “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end.”
What do these two studies have in common? Well, they were exposed for fraud. It has since come out that the behaviors of the guards in the Stanford Prison Experiment were pretty much scripted, therefore rendering it a theater rather than science. The other study was retracted for data fabrication.
In my work here, I reference many research studies. While I don’t only use research on psychology topics, my main focus is, of course, mental health. It is true, though, that fraud occurs in every discipline, not just psychology.
If you have read my previous article, Amotivation and Cannabis, I had decided to first look at personal anecdotes exploring the connection between cannabis and motivation and then review the research. Personal anecdotes were (not surprisingly) all over the place, and research studies (more disturbingly) were just as varied in their findings. This is a serious problem, because scientific studies done right should all arrive at the same conclusion. This brings me to…
The replication crisis in psychology research
In the 2010s, psychology was hit with a bomb: the replication crisis. In short, researchers tried repeating earlier studies and found out that they were not getting the same results. The Open Science Collaboration analyzed 100 studies that replicated prior psychology research and discovered that only a little over a third of them had the same results as the original studies! Moreover, the effect size went down by about 50%. They decided to include 207 additional replicated studies, and this boosted the successful replication rate to 64%, with an average effect size of 68%. This might seem like a major improvement… except at least 89% of the studies would have been successfully replicated had all the original studies been scientifically sound.
Why is this happening? There’s always the reality of good faith mistakes, shoddy research, and inconsistent or poor definitions of a concept being studied. Going back to the examples of cannabis-and-motivation studies, the third issue could apply, as many of the studies defined motivation differently. One could argue that, in the end, were they all really studying the same thing?
Another issue is tiny sample sizes. In university, they taught that sample size of 30 is sufficient for statistical significance. For a long time, I found the notion pretty dubious, and my suspicion has been justified. In a video on psychology’s replication crisis, Professor Eric Vanman noted that a study that has a small effect size (which is normally the case for psychology) actually needs a minimum of 788 participants. Note that a number of the cannabis-and-motivation studies had under 50 participants.
Then, there is fraud. Unfortunately, not all researchers are fueled by a desire for the truth. Some want to bend the facts to support an ideology. Others simply want a prestigious career in research and will do anything they can to produce studies to satisfy the steep demands of scientific journals.
When it comes to fraud, people generally find solace in safeguards such as peer review and double-blinding. These do not suffice to stamp deception out of science. Dutch psychologist Diederik Stapel, for example, literally made up data for a whopping 58 studies and possibly would have gotten away indefinitely if his graduate students had not gotten suspicious about him “collecting” all the data himself.
In addition to outright fraud, there is also a grayer area encompassing what is known as researcher degrees of freedom. Researcher degrees of freedom is also called questionable research practices, or, to quote Professor Eric Vanman, “statistical jiggery.” Think of researcher degrees of freedom as basically artistic license for scientists. A lot of what researchers do to make their findings look significant can be likened to using filters on pictures for a dating profile. With filters, you can make the nose smaller, the skin clearer, the waist smaller, and so forth, thereby bumping a dating candidate from unremarkable to highly attractive. With researcher degrees of freedom, you can switcheroo a discounted hypothesis and therefore look like you were proven right after all, delete any outliers that “ruin” an otherwise neat data set, and tweak data until insignificant findings become significant. You can also run trials again and again until chance presents you with the desired data set and simply delete all the undesirable data sets – apparently, this tactic is popular with drug companies.
And then, there is another issue, which is that psychology is not a real science.
Psychology is not a real science
It’s been said many times before that psychology is not a real science like chemistry and physics. This has truth to it.
What happens if you take a bowl of baking soda and pour vinegar into it? The two substances will undergo an immediate chemical reaction, foaming and bubbling up and over the sides of the bowl and onto the table. It doesn’t matter if you perform this experiment in America or China or Brazil, or if you did it back in 1970 or now in 2026; the two chemicals will still react the same.
Human behavior, however, is so subjective. Our choices can be determined by culture, social pressure (or lack thereof), personal preferences, or even our mood at a given moment. Furthermore, while you can measure a substance like vinegar in milliliters, you cannot do that with emotions.
How can we wean out the fraudulent research on psychology (or any other) topics?
The sad state of things is that research needs to be approached with a healthy dose of skepticism.
Personally, I do reference research studies on both psychology and non-psychology topics, and I will continue to do so. That said, I also like to review anecdotes, because I believe a combination of the two (research studies plus anecdotes) offer a clearer picture of a given subject. Yes, it is true that anecdotes tend to be viewed dismissively, and this attitude likely originates from the scientific realm in order to cement their position as the foremost authority. I strongly believe if what a particular research study claims clashes against what I can observe in the real world, it is suspect.
Can you spot bad research? Well, if you are a statistics pro and have the time on your hands to assess every single study, then go for it. Keep in mind that this won’t always work. Famous cases of researchers who got caught, like Diederik Stapel and Francesca Gino, were actually quite sloppy with their fabrications. I suspect a lot of researchers are a lot more subtle with their data manipulation.
On the positive side, eventually, time always tells what is true and what is false.
References:
1. Evans, D. (n.d.) Epic fails! What can we learn from recently debunked psychology studies? Mindlab. https://themindlab.co.uk/academy/epic-fails-what-can-we-learn-from-recently-debunked-psychology-studies/
2. Marcus, A. (Sep. 14, 2021). Highly criticized paper on dishonesty retracted. Retraction Watch. https://retractionwatch.com/2021/09/14/highly-criticized-paper-on-dishonesty-retracted/
3. Nobel, C. (Feb. 9, 2015). Professional Networking Makes People Feel Dirty. Harvard Business School. https://www.library.hbs.edu/working-knowledge/professional-networking-makes-people-feel-dirty
4. Replication Crisis. (reviewed date Nov. 17, 2025). Psychology Today. https://www.psychologytoday.com/us/basics/replication-crisis
5. Korbmacher, M., Azevedo, F., Pennington, C.R., Hartmann, H., et al. (Jul. 25, 2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology https://pmc.ncbi.nlm.nih.gov/articles/PMC11290608/
6. Oransky, I. (Sep. 6, 2016). He’s back: Data faker Diederik Stapel will support research at vocational university. Retraction Watch. https://retractionwatch.com/2016/09/06/hes-back-data-faker-diederik-stapel-will-support-research-at-vocational-university/
7. SocialNeuro. (Feb. 5, 2023). Questioning Psychology’s Findings: A Real Crisis. Youtube. https://www.youtube.com/watch?v=FXIFtWH0GQM
More articles!