The scientific method is a prime example of human ingenuity. In fact, the modern scientific method is often regarded as the greatest invention—one that has given rise to all other inventions.
Why? Because science helps us better understand the world in which we live by allowing us to test (and retest) important questions through a continuous process of observation and experimentation:
Ask a Question – Does a specific ingredient help people lose weight?
Do Background Research – Have similar experiments been conducted previously? Can you build upon this knowledge?
Construct a Hypothesis – The hypothesis is an educated guess based on your background research; e.g. existing studies show that an ingredient may have a measurable effect on weight loss.
Experiment – In order to test your hypothesis, you need to devise an experiment (we’ll talk more about this later).
Draw a Conclusion Based on Your Data – After you’ve completed your experiment and have all the data available, you can come to an educated conclusion about whether or not your hypothesis was correct. Depending on the results, you may need to start back at square 1.
Communicate Your Results – Finally, you can submit your findings to scientific journals to have your work peer reviewed (again, we’ll revisit this in a moment).
Despite the power harnessed using the scientific method though, the truth is that not all studies are created equal. As such, thousands of less-than-stellar companies knowingly use highly flawed clinical studies to support their otherwise absurd claims and to sell more of their products, often concentrated within the nutritional supplements and anti-aging niches.
In fact, as we’ll discuss shortly, a whole industry exists that’s intended to give credibility to these otherwise useless studies and to mislead and misinform consumers.
From fake customer reviews and free trials to high-pressure sales tactics and bloated, misleading scientific “studies,” with so much standing in the way between a company’s marketing hype and the truth, it can often seem overwhelming to the average consumer.
But the good news is that in this article, we’ll talk about the process that perpetuates this junk science, and some useful tips you can use to avoid falling victim.
One Journalist’s Journey into the Underbelly of Clinical Studies
If you need proof that clinical trials can be massaged to produce just about any result necessary, look no further than journalist John Bohannon.
Creating Fake Science 101
Beginning in 2014, he and a team of German journalists set out to show, first-hand, how much of the diet industry relies on junk science to promote their products, and to demonstrate “just how easy it is to turn bad science into the big headlines.”
To accomplish this, John and his team set up a clinical trial involving 16 random individuals aged 19 to 67. Then, these participants were divided into three groups: Low carbohydrate diet, low-carb diet plus a 1.5oz bar of dark chocolate per day, and another that made no changes to their diet.
By the time everything was said and done—and using a financial analyst to crunch the numbers—he ultimately used his clinical studies to “prove” that chocolate helps weight loss.
Spreading Like Wildfire
Then, once John and his team had the data they needed, they decided to see how easy it would be to “share [their] scientific breakthrough with the world.”
To do this, John submitted his research to 20 different fake scientific journal publishers (many of which come with official-sounding names) that publish findings without subjecting them to peer review, which is one of the primary methods of self-regulation within the scientific and academic communities.
After their study was accepted, John’s team could boast that their findings had been posted in a scientific journal, so they enlisted the help of a public relations professional to get the word out. And it worked—almost too well. According to John:
“We landed big fish before we even knew they were biting. Bild rushed their story out—“Those who eat chocolate stay slim!”—without contacting me at all. Soon we were in the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show.
“When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None is quoted.”
According to John, someone who has a great deal of experience in the field, even he was taken aback by how relatively simple it was to promote “terrible, meaningless science” and have it spread around the globe.
But what’s going on here? Why is it so easy to create and popularize fake science? Let’s take a closer look.
Why Isn’t the Government Doing Something About Fake Clinical Studies?
The technical term for what John and his team did in their experiment is something called scientific misconduct, or the "Intention[al] distortion of the research process by fabrication of data, text, hypothesis, or methods from another researcher's manuscript form or publication; or distortion of the research process in other ways."
With this said, weeding out fake clinical data is probably more difficult than you’d imagine, and involves a multi-pronged problem (whether by the government or the scientific community itself), for several different reasons. Here are some of the most important:
What is “Science?”
First, there’s no tangible, defining line between “good” and “bad” science (we’ll come back around to this in the next section). In fact, even the definition of science itself is extraordinarily broad and all encompassing.
To outline this fact, as John noted at the beginning of his article, “Other than those fibs [his name and professional title], the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.”
In other words, what John and his team did often passes for “science” thousands of times daily all across the globe, regardless of whether or not it’s intentional.
Everyone is Culpable
Second, when fake science like John’s makes its way into mainstream media, it’s often because of a systematic breakdown in every step of the process.
First, it’s the scientists who are (knowingly or otherwise) engaging in poor scientific practices. Then, it’s the “scientific journals,” with official-sounding names, who will publish research—sight unseen and often containing horrendous errors—without peer review, simply for a fee.
Journalists often don’t vet their sources as thoroughly as they should, which allows fake science to quickly make its way to the general public.
Finally, because most publications have moved online—a space where generating traffic is king, meaning that new, fresh content must be constantly created—journalists often don’t vet their sources as thoroughly as they should, which allows fake science to quickly make its way to the general public.
The Government’s Role in Oversight
Lastly, as we detailed in our Nutritional Supplements Buyer’s Guide and Anti-Aging Free Trials articles, unless these types of products specifically claim to treat a condition, they’re subjected to zero governmental oversight before hitting store shelves.
Ultimately, this means that manufacturers can make essentially any claims they wish, without having to back them up with a single shred of evidence. And to compound the problem even further, many times when they do decide to provide evidence, it’s often based on junk science, so the process comes full circle.
So what does all this mean? When it comes down to it, it’s up to you to learn how to identify junk clinical trials from legitimate science.
4 Easy Steps for Spotting a Less-than-Stellar Clinical Study
The next time you’re reviewing a clinical study related to a nutritional supplement or anti-aging product, use these actionable tips to help you figure out if it’s legitimate or fake.
Step 1: Who Conducted & Funded the Study?
At the top of any legitimate clinical study (all examples below were taken from the National Institutes of Health’s PubMed), you’ll find the date the clinical study was conducted, who conducted it, and often who funded it (such as a research institution or a company).
An example of the author information found at the top of any legitimate clinical study.
In the garcinia cambogia study noted above, we’re informed that it was conducted by Downs BW, Bagchi M, Subbaraju GV, Shara MA, Preuss HG, Bagchi D, and that it was funded by InterHealth Research Center in Benicia, CA.
While you can generally click on each of the author’s names to learn more about them, as well as uncover other studies they’ve been involved with, it may not be necessary in this instance.
Why? Because InterHealth Research Center is a nutraceuticals (e.g. supplements) company that manufactures a variety of weight management products. And as you might imagine, if the company funding the research has a direct relationship with the outcome of a study, they may have an incentive to fudge the numbers in their favor.
Note: Keep in mind that we’re not saying this is the case here. Only that it should raise a red flag any time a directly related company is funding clinical research.
Step 2: Where is the Study Published?
Continuing from the example above, scrolling down the page and looking under Publication types, we see that this study was originally published in Elsevier Science, a well known academic publishing company with offices in several countries all over the globe.
In this example, this study was published by Elsevier Science.
However, just because Elsevier is an entrenched, well known company, this doesn’t mean that they’re above posting junk science.
To this end, Elsevier was actually part of John Bohannon’s (the author noted at the beginning of this article) Who’s Afraid of Peer Review? sting that uncovered one of the publisher’s journals “accepted an obviously bogus paper made-up by Bohannon that should have been rejected by any good peer review system. Instead Drug Invention Today was among many open access journals that accepted the fake paper for publication.”
As such, it’s important that you always find out who published a clinical study and then Google their name to find out about any controversies surrounding them.
Pro tip: If you’re not looking at a study directly through PubMed (although most published research ends up in PubMed), but instead through a excerpt on a product’s website, look in the following areas:
- In the upper left-hand corner (e.g. 579(1-2):149-62)
- Below the abstract (e.g. PMID: 16055158)
As long as you have either of these pieces of information, you can copy/paste into the search engine of your choice to locate the original transcript.
Step 3: How Many Participants Were Involved?
Next, find out how many participants were included in the study (known as a sample size), as this number is directly proportional to the results it uncovers.
This is because, as John puts it his chocolate article:
“Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.
“Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.”
Step 4: Bringing It All Together
As straightforward as it may seem, the reality is that using the above 3 steps can go a long way toward helping you discern between junk and “legitimate” clinical trials. Sure, it might take a little bit of time to research each of these aspects, but they can end up helping you save a ton of your hard-earned money on products that are full of nothing but empty promises and marketing hype—especially if you’re in the market for a new anti-aging product or nutritional supplement.
With this said, although you’re now armed with all the information you need, keep in mind that unless you’re a medical professional (which we’re certainly not here at HighYa, although we are passionate about helping you become more informed and avoid scams), it’s always important that you discuss any of these products with your physician.
On top of this, they might even be able to give you some additional tips that can help you differentiate between good and bad science, while helping you to avoid scams.