There’s a lot of “science-backed” advice in books, articles, newspapers, and other places we get our advice from. Honestly, I also regularly refer to scientific studies.
My goal for writing articles is to combine my personal experience with evidence. Sometimes I share mostly my experience; sometimes I share stories about other people’s experiences and research. But I always think, “How can we trust these sources?”
Fortunately, I’m not the only person who thinks that. Even scientists say we should remain critical of the findings of scientific experiments. David Chester, Ph.D., a psychology professor at Virginia Commonwealth University, recently did a study of studies. Together with his colleague, Emily Lasko, they wrote:
“The findings of experimental psychology likely rest on an untested foundation.”
They did a study of scientific experiments in the field of psychology. Chester and Lasko investigated 348 psychological experiments. Their study showed that roughly 42% of the experiments lacked validity.
Validity is an important concept in scientific research because it’s an indication of how sound research is. In other words: How solid are the claims scientists make? Chester and Lasko are basically saying we don’t know.
How to interpret psychology research
One of the biggest critics of scientific experiments is Nassim Nicholas Taleb, author of Fooled by Randomness:
“(M)any psychological “biases” are errors by researchers missing a layer of uncertainty in the model.”
Taleb argues that scientists also make thinking errors. We shouldn’t take findings as the truth. We should remain critical at all times. But how do we know which experiments lack validity?
Chester and Lasko didn’t disclose that information, which I understand. They didn’t want to point fingers at their colleagues. But we have to recognize that there are a lot of experiments. This Wikipedia lists 76 popular ones.
You might have heard of “Learned Helplessness.” It’s an experiment I’ve referred to in the past. It’s also included in my productivity course. So does this mean we can’t trust science at all? No.
If a study isn’t valid, it doesn’t automatically mean it’s invalid. David Chester explains this well: “Almost all of the manipulations we examined failed to provide the necessary evidence that they were valid, which does not mean they are invalid — their validity is just unknown.”
A lack of proof is not proof. Pseudo intellectuals love to use the absence of evidence as evidence. The 17th century philosopher John Locke was probably the first who wrote about this thinking fallacy. The aphorism goes as follows:
“Absence of evidence is not evidence of absence.”
It’s ignorant to dismiss science. And yet, we see this all the time. That’s why we can all benefit from having basic knowledge about how to interpret scientific studies. Here are a few things to keep in mind when you read about scientific studies:
- Scientists are humans, and humans make mistakes
- We don’t know everything
- Avoid claims that come from a single source
- Don’t pick sides
- If possible, test the ideas in your own life
- Proof doesn’t mean it’s true
- Lack of proof doesn’t mean it’s false
- Understand that there’s no one size fits all
Remain critical of theories
One of my favorite parts in Berkshire Hathaway’s annual meeting includes Warren Buffett and Charlie Munger taking subtle digs at economists who believe in the Efficient Market Hypothesis. EMH states that share prices always reflect all available information and are efficiently priced. According to that theory, it’s impossible for investors to purchase undervalued stocks.
That means it should be impossible to outperform the overall market. Tell that to Buffett and Munger, who consistently outperformed the market for decades. $1000 invested with Warren Buffett since 1965 would be worth more than $27 million last year. The comparable amount for the S&P 500 was roughly $200,000 (source: Barron’s):
Understanding that theories are not true for everyone is particularly important when it comes to psychology experiments. Let me give you an example. In the personal development world, many writers love to refer to the Zeigarnik effect.
The idea is that people remember unfinished tasks better than completed tasks. Some experts claim that certain tasks remain in your mind because you haven’t finished them.
But here’s the thing. Several other studies failed to replicate Zeigarnik’s experiment (Van Bergen, 1968). Does that mean the theory is false? Not necessarily. It means the concept might be true for some people, but not all people.
We must realize that experiments show results from a small group of participants. If a bunch of students kept on thinking about their unfinished tasks, it doesn’t mean we’re all built that way. Every person is different. Most psychology experiments are situational.
Maybe an idea is true for a certain group of people or during a particular situation. For example, I have no issue with unfinished tasks myself. That stuff doesn’t linger on my mind because I practice mindfulness.
In his book, Range, David Epstein argues against several popular beliefs that claim scientific evidence. He breaks down the research behind popular ideas like the 10,000 hours rule and Grit. He demonstrates we must remain critical because these studies only look at a very specific group of people. We can’t extrapolate these ideas to everything in life.
Universal truths are sparse
The problem with information is the way it’s presented. When we have an idea or learn something new, we’re quick to think it’s 100% true. We love to present ideas as “this is the way it is!”
The Coronavirus pandemic is a perfect example. I’ve read and watched interviews with dozens of virologists and epidemiologists. And nearly all of them spoke with full confidence. It seems like they’re describing hard science. But very few things in life are as straightforward as basic math.
Ask 100 teachers to show a kid how to do basic math and you’ll see a unified approach. Basic math is a universal truth. We can’t argue with the fact that 5 x 5 is 25.
But ask 100 virologists whether we should close schools during a pandemic and you get an intellectual war. Who knows it best? We don’t know.
There are very few universal truths in life; things we can say with absolute certainty. I personally live my life according to those truths. Most of them are common sense; things that have been passed from one generation to the next.
One of those truths is something I learned from my mother, who in turn learned it from her father, and so forth. My grandfather firmly believes in the power of waking up every day at the same time. He’s a highly orderly person and that strategy works great for him. He’s still independent and lives on his own.
And I follow his advice as well. I simply tried it and it worked for me. The times I’m most consistent in life are when I wake up at the same time. You can always do your own experiments to see what works for you. We don’t have to be a scientist to try things.
That’s the biggest lesson I’ve learned. What if Buffett and Munger simply accepted the Efficient Market Hypothesis? What if we all drove ourselves nuts if we didn’t finish a task? We must keep challenging ourselves and the status quo.
But I’m glad that people like Chester and Lasko keep doing scientific research with an open mind. Findings like these show us that we never stop learning.