Chocolate causes weight loss academic paper was a hoax

Seen something interesting in the news or on the intertubes? Discuss it here.

Moderators: Zamfir, Hawknc, Moderators General, Prelates

KnightExemplar
Posts: 5494
Joined: Sun Dec 26, 2010 1:58 pm UTC

Chocolate causes weight loss academic paper was a hoax

Postby KnightExemplar » Thu May 28, 2015 1:58 pm UTC

http://io9.com/i-fooled-millions-into-t ... 1707251800

Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.


The TL;DR of the article is that "p-hacking" is a very easy way to lie with statistics. He shapes the data to favor the 'fact' "Eating Chocolate causes weight loss", publishes the data in allegedly "peer-reviewed journals", creates a snazzy website that supports the claim. A few months later, he finds his data across the diet industry.

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.


With our 18 measurements, we had a 60% chance of getting some“significant” result with p < 0.05. (The measurements weren’t independent, so it could be even higher.) The game was stacked in our favor.


The article is more about how "fee-charging open access journals" are ruining academic journalism. And how easy it is to get a "scientific-sounding" website make absurd claims that get gobbled up by the media. Furthermore, his fake institution was not checked (he only made a website called the Institute of Diet[/url], and he was never contacted about his paper (leading him to believe that the "peer review" process is broken in those journals)
First Strike +1/+1 and Indestructible.

elasto
Posts: 3757
Joined: Mon May 10, 2010 1:53 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby elasto » Thu May 28, 2015 2:24 pm UTC

It's basically this writ large, right?

KnightExemplar
Posts: 5494
Joined: Sun Dec 26, 2010 1:58 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby KnightExemplar » Thu May 28, 2015 2:42 pm UTC

elasto wrote:It's basically this writ large, right?


Yes. Although the hope was that the scientific -> lay people "journalists" would have a mechanism for rejecting that kind of crappy science.

This guy seemed to have published fake science in various journals to test how good various "scientific" journals are. Another article is here about how many journals today are letting bad science through.
First Strike +1/+1 and Indestructible.

Tirian
Posts: 1891
Joined: Fri Feb 15, 2008 6:03 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tirian » Thu May 28, 2015 4:21 pm UTC

elasto wrote:It's basically this writ large, right?


What Randall is describing is a thing that we need to consider -- that we should suspect that 5% of authentically generated significant results are actually false positives. But what this guy was doing was worse.

Let me give you an actual example. I just rolled 100 dice and tracked the values that I got. The frequencies of the values turned out to be (10, 17, 16, 19, 21, 16). Now I plug these numbers into a binomial calculator and find that the probability of rolling 10 or fewer ones out of 100 rolls is 0.021+. So I am I justified in claiming that this data is a significant finding that my dice are biased against ones with p<0.05?

Absolutely not. The flaw is that I collected my data before deciding what my hypothesis was. Secretly, I was simultaneously prepared to report on any of twelve possible significant "abnormal" results, depending on what the data revealed. If any of the values are less than 12 or more than 22, I will have a significant result. If none of them were significant, I would have repeated the experiment after lunch and not told you about my first strike-out.

To do this rigorously, I need to roll the dice 100 more times and only report on whether those results (alone) justify the conclusion that 1's come up less often than they should. (And Randall's point is that 5% of the time, lightning would strike twice anyways even if my dice are fair.)
Last edited by Tirian on Thu May 28, 2015 4:27 pm UTC, edited 1 time in total.

Puppyclaws
Posts: 391
Joined: Fri Jul 15, 2011 7:08 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Puppyclaws » Thu May 28, 2015 4:24 pm UTC

Science depends in part on credulity, which is why I usually look askance at these "hoax" publications. It is well known that there are shitty journals that will publish shitty science, that science journalism is terrible for a number of reasons, and that p-hacking is a problem (which at least in my field we are actively trying to solve).

This study may have damaged millions of lives. Retractions, even splashy retractions, almost never make it as far as the original source does. I'm certainly seeing that now with the LaCour retraction (which was fraud, not hoax). And of course, the article KnightExemplar links to was published as part of work by Science, who published the LaCour paper.

Tyndmyr
Posts: 11443
Joined: Wed Jul 25, 2012 8:38 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tyndmyr » Thu May 28, 2015 5:52 pm UTC

Puppyclaws wrote:This study may have damaged millions of lives. Retractions, even splashy retractions, almost never make it as far as the original source does. I'm certainly seeing that now with the LaCour retraction (which was fraud, not hoax). And of course, the article KnightExemplar links to was published as part of work by Science, who published the LaCour paper.


In fairness, if your life was significantly damaged by hearing that chocolate = diet fad, and then chasing the shit out of that, you probably would have chased some other stupid fad that was equally poorly justified anyway. These exist basically constantly.

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Thu May 28, 2015 6:02 pm UTC

People look at things like this and assume the worst. That all science is unreliable. What it kills is the requisite trust that you must have. It makes it the realm of hucksterism and dog and pony shows.

User avatar
Quercus
Posts: 1806
Joined: Thu Sep 19, 2013 12:22 pm UTC
Location: London, UK
Contact:

Re: Chocolate causes weight loss academic paper was a hoax

Postby Quercus » Thu May 28, 2015 6:06 pm UTC

Tirian wrote:
elasto wrote:It's basically this writ large, right?


What Randall is describing is a thing that we need to consider -- that we should suspect that 5% of authentically generated significant results are actually false positives. But what this guy was doing was worse.

Let me give you an actual example. I just rolled 100 dice and tracked the values that I got. The frequencies of the values turned out to be (10, 17, 16, 19, 21, 16). Now I plug these numbers into a binomial calculator and find that the probability of rolling 10 or fewer ones out of 100 rolls is 0.021+. So I am I justified in claiming that this data is a significant finding that my dice are biased against ones with p<0.05?

Absolutely not. The flaw is that I collected my data before deciding what my hypothesis was. Secretly, I was simultaneously prepared to report on any of twelve possible significant "abnormal" results, depending on what the data revealed. If any of the values are less than 12 or more than 22, I will have a significant result. If none of them were significant, I would have repeated the experiment after lunch and not told you about my first strike-out.

To do this rigorously, I need to roll the dice 100 more times and only report on whether those results (alone) justify the conclusion that 1's come up less often than they should. (And Randall's point is that 5% of the time, lightning would strike twice anyways even if my dice are fair.)


Incidentally this is why genome wide association studies (at least for human genomes) typically have a P-value threshold of 5x10-8 and are only considered more than provisional after independent replication.

Tirian
Posts: 1891
Joined: Fri Feb 15, 2008 6:03 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tirian » Thu May 28, 2015 6:24 pm UTC

Puppyclaws wrote:This study may have damaged millions of lives.


LOL. Andrew Wakefield's autism-vaccine study fraud hasn't even damaged millions of lives yet. And that's damage as in children getting measles, not people gaining five pounds before realizing that an incredible diet plan is, well, not credible. (And they got to eat chocolate in the meantime, so I have even more trouble feeling sorry for them.)

LaCour's study is a good story about the way science should work (almost), IMO. He published a remarkable finding, the next researcher in line replicated the study with a few tweaks and didn't get the same results, LaCour couldn't produce his data, and the remarkable result is discounted. The only real loser is Donald Green for being a legitimate researcher who risked his career long track record by co-authoring a study where he had no information about the data or methodology or funding. But we still need to keep in mind that 5% of the breaking research is wrong even if they're doing it right. (Assuming p<0.05 studies, Quercus is right that fields with more stringent confidence intervals may well be more trustworthy at first blush.)

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Thu May 28, 2015 6:43 pm UTC

Tirian wrote:OL. Andrew Wakefield's autism-vaccine study fraud hasn't even damaged millions of lives yet.
Yet being the operative term. It is a gift that keeps on giving.

User avatar
Quercus
Posts: 1806
Joined: Thu Sep 19, 2013 12:22 pm UTC
Location: London, UK
Contact:

Re: Chocolate causes weight loss academic paper was a hoax

Postby Quercus » Thu May 28, 2015 6:56 pm UTC

Tirian wrote:But we still need to keep in mind that 5% of the breaking research is wrong even if they're doing it right. (Assuming p<0.05 studies, Quercus is right that fields with more stringent confidence intervals may well be more trustworthy at first blush.)


That wasn't quite the point of my aside - the point was in support of the notion that p<0.05 only implies a 5% false-positive rate if you're looking at a single pre-chosen outcome. In GWAS you are looking at >1 million single nucleotide polymorphisms, therefore you need to drastically reduce the p value in order to bring the false-positive rate under control. A p-value of 1x10-8 would imply around a 1% false-positive rate in this scenario, which is why you still don't trust GWAS until someone's done independent replication.

My point was that the large number of outcomes you are testing in GWAS makes p-hacking blindingly obvious (no-one would accept a study with ~200,000 apparent genetic associations) and it therefore acts as a good instructive example of why you can't just use raw p-values unless your study design is extremely simple.

Tirian
Posts: 1891
Joined: Fri Feb 15, 2008 6:03 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tirian » Thu May 28, 2015 7:12 pm UTC

Quercus wrote:which is why you still don't trust GWAS until someone's done independent replication.


I grant your point, but I was figuring that a replicated finding with the same confidence interval but with with a known hypothesis should be extremely persuasive. But perhaps your replicated findings don't need to be quite so stringent.

Puppyclaws
Posts: 391
Joined: Fri Jul 15, 2011 7:08 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Puppyclaws » Fri May 29, 2015 12:58 am UTC

Tirian wrote:
Puppyclaws wrote:This study may have damaged millions of lives.


LOL. Andrew Wakefield's autism-vaccine study fraud hasn't even damaged millions of lives yet. And that's damage as in children getting measles, not people gaining five pounds before realizing that an incredible diet plan is, well, not credible. (And they got to eat chocolate in the meantime, so I have even more trouble feeling sorry for them.)

LaCour's study is a good story about the way science should work (almost), IMO. He published a remarkable finding, the next researcher in line replicated the study with a few tweaks and didn't get the same results, LaCour couldn't produce his data, and the remarkable result is discounted. The only real loser is Donald Green for being a legitimate researcher who risked his career long track record by co-authoring a study where he had no information about the data or methodology or funding. But we still need to keep in mind that 5% of the breaking research is wrong even if they're doing it right. (Assuming p<0.05 studies, Quercus is right that fields with more stringent confidence intervals may well be more trustworthy at first blush.)


I was being a bit facetious and borrowing the author's title, "I fooled millions," but I actually do think it is unethical to loudly and widely spread misinformation in such a way. Maybe I'm oversensitive about it because I have to work with the broad (at times seemingly ridiculous) definition of harm that the average IRB uses. I think it's probable that Wakefield has damaged millions of lives, using a criterion like that (e.g. how many people have not been vaccinated since the study was published?).

In some ways the LaCour study is an example of science done mostly right, given the speed of the retraction; however, given a remarkably strong and contradictory finding, I do wish there had been a little more checking in. People have suggested before this that you should at least have to submit the dataset when going to publish. It's starting to look concerning that nobody in his graduate program was aware of the falsehoods he was spreading about fictional awards and grants he received, which likely had a significant impact on his hiring at Princeton.

User avatar
sardia
Posts: 6800
Joined: Sat Apr 03, 2010 3:39 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby sardia » Fri May 29, 2015 3:19 am UTC

On top of the p-level fishing, we really should improve upon the placebo studies. Right now the standard study compares a treatment vs nothing (the control/placebo) which tells us if something works. But we already have stuff that works, what we need now is stuff that works better than what we currently have. What's the point in spending(and subsequently charging) millions in R&D money to develop products that aren't any better than the super cheap patent expired products we already have? Well, other than to fish for money by deceiving consumers into thinking newer = better.

User avatar
Angua
Don't call her Delphine.
Posts: 5933
Joined: Tue Sep 16, 2008 12:42 pm UTC
Location: UK/[St. Kitts and] Nevis Occasionally, I migrate to the US for a bit

Re: Chocolate causes weight loss academic paper was a hoax

Postby Angua » Fri May 29, 2015 6:45 am UTC

Most studies these day are against the current treatment, and have to show that they are better (either better at treating the disease, or better sideeffects).

It would be unethical to give placebos to people for a condition for which treatment already exists.
Crabtree's bludgeon: “no set of mutually inconsistent observations can exist for which some human intellect cannot conceive a coherent explanation, however complicated”
GNU Terry Pratchett

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Fri May 29, 2015 6:53 pm UTC

From Ars Technia an op ed called "Is it OK to generate a fake news story to make a point? No".
In terms of ethical analysis, this is an experiment that did not tell us anything that wasn't known already. On that score alone, the experiment fails to pass muster. Then there are the downsides. The reputation of science journals and science communicators just got a slight additional tarnish. Worse yet, there are people out there who have been taken in by the false reporting, and many of them will never know that the story was false from the beginning.
Making muddy water muddier

You might say that such people are gullible anyway and will get taken in by the next false story. True, but at least most false stories aren't done for the purpose of making some point of principle or to make a story entertaining enough to get readers to click. Just because a situation is bad doesn't mean you should actively make it worse.

It wouldn't have been that hard for the team to comb through news reports to pick up a bunch of really badly covered health and diet stories. They could have confronted researchers. They could have filmed public reaction when people were shown how they had been fooled. They could have trapped reporters in their own laziness with these stories.

But then they would have had to work harder to make the documentary interesting. Now, as I see it, we've been left with a lazy reporter that has made a documentary about lazy reporters.

User avatar
cjmcjmcjmcjm
Posts: 1158
Joined: Tue Jan 05, 2010 5:15 am UTC
Location: Anywhere the internet is strong

Re: Chocolate causes weight loss academic paper was a hoax

Postby cjmcjmcjmcjm » Sun Jun 07, 2015 8:45 pm UTC

morriswalters wrote:From Ars Technia an op ed called "Is it OK to generate a fake news story to make a point? No".
In terms of ethical analysis, this is an experiment that did not tell us anything that wasn't known already. On that score alone, the experiment fails to pass muster. Then there are the downsides. The reputation of science journals and science communicators just got a slight additional tarnish. Worse yet, there are people out there who have been taken in by the false reporting, and many of them will never know that the story was false from the beginning.
Making muddy water muddier

You might say that such people are gullible anyway and will get taken in by the next false story. True, but at least most false stories aren't done for the purpose of making some point of principle or to make a story entertaining enough to get readers to click. Just because a situation is bad doesn't mean you should actively make it worse.

It wouldn't have been that hard for the team to comb through news reports to pick up a bunch of really badly covered health and diet stories. They could have confronted researchers. They could have filmed public reaction when people were shown how they had been fooled. They could have trapped reporters in their own laziness with these stories.

But then they would have had to work harder to make the documentary interesting. Now, as I see it, we've been left with a lazy reporter that has made a documentary about lazy reporters.

That was a hilariously over-the-top article and felt way out of place on Ars.
frezik wrote:Anti-photons move at the speed of dark

DemonDeluxe wrote:Paying to have laws written that allow you to do what you want, is a lot cheaper than paying off the judge every time you want to get away with something shady.

User avatar
Copper Bezel
Posts: 2426
Joined: Wed Oct 12, 2011 6:35 am UTC
Location: Web exclusive!

Re: Chocolate causes weight loss academic paper was a hoax

Postby Copper Bezel » Mon Jun 08, 2015 9:10 am UTC

Yeah, wow. "Reputations tarnished" was entirely the point, so ... that's a feature, not a bug, and the hope is that we'll continue to see improvements in how these things are vetted as people continue to press the issue, whether through more traditional investigative reporting or guerrilla tactics like this.

If the story picks up as well as the hoax did, some of the people who get into these fad diets might actually learn to be a bit more skeptical. The benefits and costs are going to be marginal and well below measurement, but, hey, there might actually be more people eating more healthy as a result of the hoax. That's obviously the hope, beyond the shaming-and-advocacy side of things.

And by doing this, they indeed "trapped" a lot of people "in their own laziness," the people who ... apparently have tarnished reputations now, which the AT article is also saying is a bad thing?

But if there's a funny story going around that makes a nice example case of something, and you come out with a spiteful and seemingly logical attack on it, you can probably get some clickthroughs on that. For instance, when people perform scientific studies on the seemingly obvious, say, a study showing that men pay less attention to the biographies on dating sites, as opposed to a social experiment on a relatively well-known but not entirely explored social phenomenon; it's very easy to convince readers that they already knew a thing for certain and that scientists are wasting their money or journalists are wasting their time by confirming it to be the case. People also like to be upset and love to upset their Facebook friends with things like this. Perhaps the Ars Technica piece is itself a zero-effort satire sort of performance piece as well?

Edit: See also this post, I suppose.
So much depends upon a red wheel barrow (>= XXII) but it is not going to be installed.

she / her / her

User avatar
Thesh
Made to Fuck Dinosaurs
Posts: 6579
Joined: Tue Jan 12, 2010 1:55 am UTC
Location: Colorado

Re: Chocolate causes weight loss academic paper was a hoax

Postby Thesh » Mon Jun 08, 2015 9:18 am UTC

Well, worst case scenario, every doctor in the world loses all faith in science, and goes back to bloodletting and exorcisms and trillions of people die over the next century.
Summum ius, summa iniuria.

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Mon Jun 08, 2015 3:06 pm UTC

Well in truth billions will die anyway despite anything that doctors might do to help them along. And op eds are about opinions. However it must be slow, I thought this thread was dead.

cphite
Posts: 1362
Joined: Wed Mar 30, 2011 5:27 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby cphite » Tue Jun 09, 2015 2:50 pm UTC

Thesh wrote:Well, worst case scenario, every doctor in the world loses all faith in science, and goes back to bloodletting and exorcisms and trillions of people die over the next century.


Well, clearly we cannot believe anything that scientists say anymore. We were willing to forgive the thing about the zombie cat in the box; and we let you slide on the stuff about warmer weather meaning more snow... but this time you messed with chocolate, and that simply cannot be tolerated.

Begone, science!

User avatar
Minerva
Posts: 947
Joined: Sat Nov 11, 2006 2:58 pm UTC
Location: Australia
Contact:

Re: Chocolate causes weight loss academic paper was a hoax

Postby Minerva » Fri Jun 12, 2015 6:00 pm UTC

It's just like the famous Sokal "experiment".

If it highlights very clearly, for everybody to understand, where the problems are and where we can improve things, and how - for journalists, for science communicators, for scientists - isn't that a good thing?
...suffer from the computer disease that anybody who works with computers now knows about. It's a very serious disease and it interferes completely with the work. The trouble with computers is you play with them. They are so wonderful. - Richard Feynman

User avatar
bigglesworth
I feel like Biggles should have a title
Posts: 7461
Joined: Sat Apr 07, 2007 9:29 pm UTC
Location: Airstrip One

Re: Chocolate causes weight loss academic paper was a hoax

Postby bigglesworth » Fri Jun 12, 2015 6:25 pm UTC

No, because journalists, even exactly the same ones that were "burned" will continue to do this. And there's nothing that scientists and science communicators can do about it.
Generation Y. I don't remember the First Gulf War, but do remember floppy disks.

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Fri Jun 12, 2015 6:28 pm UTC

Minerva wrote:It's just like the famous Sokal "experiment".

If it highlights very clearly, for everybody to understand, where the problems are and where we can improve things, and how - for journalists, for science communicators, for scientists - isn't that a good thing?
Yes and no. Had they done the deed and then reported the outcome as what is was, then yes. But they gamed journalists by first letting them publish. Which I admit made me all warm and fuzzy. But this will be showing up as fact on my Facebook feed from now on. In other words I don't mind that he baited the journalists, I do mind that he took the public for a ride. Most people in the general public could care less about what science magazines publish, they don't understand them or the science behind them. And more critically they don't want to. They do want to be told idiocy like chocolate causes weight loss. It suits their bias and they won't see the followup.

By the way here is the headline from the original link.
I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How.

Puppyclaws
Posts: 391
Joined: Fri Jul 15, 2011 7:08 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Puppyclaws » Sat Jun 13, 2015 1:35 am UTC

Minerva wrote:It's just like the famous Sokal "experiment".

If it highlights very clearly, for everybody to understand, where the problems are and where we can improve things, and how - for journalists, for science communicators, for scientists - isn't that a good thing?


Except for the fact that it is about an already proven problem, and nobody is going to change what they do. As it is Sokal set the humanities back fifty years by publishing one BS paper in one BS journal and has been an actual disaster from overgeneralization, but what do STEM people care?

User avatar
Copper Bezel
Posts: 2426
Joined: Wed Oct 12, 2011 6:35 am UTC
Location: Web exclusive!

Re: Chocolate causes weight loss academic paper was a hoax

Postby Copper Bezel » Sat Jun 13, 2015 11:03 am UTC

I kinda take a "shut up and eat your spinach" approach to both here. Or "blame your own cheap-shit peer review process." Frankly I'm not at all concerned that there will ever be any shortage of ammunition for anti-intellectualists broadly, because they and the people who listen to them are happy to take whatever presents itself out of any context and misread it into something absurd that supports their existing biases.

Critics can call a stunt sensationalism, or say that it's never going to get the message out to enough people to influence anything. Those are reasonable, plausible criticisms. You can only be sure they're arguing from a position of bad faith when the same people say both at once.
So much depends upon a red wheel barrow (>= XXII) but it is not going to be installed.

she / her / her

Willl
Posts: 34
Joined: Mon Dec 17, 2007 8:44 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Willl » Sat Jun 13, 2015 2:53 pm UTC

bigglesworth wrote:No, because journalists, even exactly the same ones that were "burned" will continue to do this. And there's nothing that scientists and science communicators can do about it.
Not true. They can educate the consumers of this media about these practices. A better-informed reader is less vulnerable to harmful influence by bad science, more receptive to good science and just might even demand less of the bad stuff. This hoax communicated insight into those practices more effectively to a wider audience than anything else I've seen.

User avatar
sardia
Posts: 6800
Joined: Sat Apr 03, 2010 3:39 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby sardia » Sat Jun 13, 2015 3:09 pm UTC

Willl wrote:
bigglesworth wrote:No, because journalists, even exactly the same ones that were "burned" will continue to do this. And there's nothing that scientists and science communicators can do about it.
Not true. They can educate the consumers of this media about these practices. A better-informed reader is less vulnerable to harmful influence by bad science, more receptive to good science and just might even demand less of the bad stuff. This hoax communicated insight into those practices more effectively to a wider audience than anything else I've seen.

That's inefficient. Is easier to just teach journalists what good or bad science studies are. Then they could be the have keepers of knowledge

Tirian
Posts: 1891
Joined: Fri Feb 15, 2008 6:03 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tirian » Sat Jun 13, 2015 3:59 pm UTC

There's nothing that science reporters (and far less that readers) can do to detect p-hacking in a study. That would require labs to document that they had settled on a single hypothesis before starting a trial or else report that they can not be statistically confident of any result that arises from the result.

I also think that news and science are not compatible fields. Take nutrition science. The dominant paradigm is that you can lose weight by burning slightly more calories per day than you ingest. But you'll rarely see a newspaper article with that headline because information that everyone already knows is not newsworthy. What is newsworthy is a finding that challenges the dominant paradigm ("Dark chocolate contains a unique blend of vitamins (or whatever) that (two) scientists have discovered to be a fat-busting powerhouse!") , but these are exactly the findings that are unlikely to be replicated because paradigm shifts happen on a generational timeframe that is not conducive to the news cycle.

morriswalters
Posts: 7073
Joined: Thu Jun 03, 2010 12:21 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby morriswalters » Sat Jun 13, 2015 5:03 pm UTC

This wasn't poor science. This was deliberate fraud. Poor science will happen. What doesn't need to happen is for somebody to abuse their credentials to make a point. Doing a piece where he submits a false paper and then uses his platform to call attention to it is one thing. Doing what he did is something else. To quote from the original article.
And Onneken wanted to do it gonzo style: Reveal the corruption of the diet research-media complex by taking part.
That smacks of proving bank vaults are weak by robbing bank vaults. The only difference that I can see is that he can't give back what he took. Once released into the wild it is by definition out of his control.

User avatar
Copper Bezel
Posts: 2426
Joined: Wed Oct 12, 2011 6:35 am UTC
Location: Web exclusive!

Re: Chocolate causes weight loss academic paper was a hoax

Postby Copper Bezel » Sat Jun 13, 2015 6:46 pm UTC

Tirian wrote:There's nothing that science reporters (and far less that readers) can do to detect p-hacking in a study. That would require labs to document that they had settled on a single hypothesis before starting a trial or else report that they can not be statistically confident of any result that arises from the result.

They might even have to e-mail the study authors for clarification!

sardia wrote:
Willl wrote:
bigglesworth wrote:No, because journalists, even exactly the same ones that were "burned" will continue to do this. And there's nothing that scientists and science communicators can do about it.
Not true. They can educate the consumers of this media about these practices. A better-informed reader is less vulnerable to harmful influence by bad science, more receptive to good science and just might even demand less of the bad stuff. This hoax communicated insight into those practices more effectively to a wider audience than anything else I've seen.
That's inefficient. Is easier to just teach journalists what good or bad science studies are. Then they could be the have keepers of knowledge

I disagree with the premise - that if only we had trustworthy news, consumers wouldn't have to read it critically. But I imagine that at least some of those writers and editors who picked up the story might think twice next time they consider uncritically reblogging someone's press release.

There are a lot of advocacy campaigns in the world. Most people are not very receptive to a random person in a lab coat's attempt to "educate" them. This was the right way.

And no, this isn't "fraud," either. Not unless every other fad diet in the world is also fraud. Courts are the last bodies in the world I want to see vested with deciding what is and isn't good science.
So much depends upon a red wheel barrow (>= XXII) but it is not going to be installed.

she / her / her

schapel
Posts: 244
Joined: Fri Jun 13, 2014 1:33 am UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby schapel » Sat Jun 13, 2015 7:46 pm UTC

Tirian wrote:
elasto wrote:It's basically this writ large, right?


What Randall is describing is a thing that we need to consider -- that we should suspect that 5% of authentically generated significant results are actually false positives.

Far more than 5%! This is because negative results are generally not published. If you ask 100 teams to test if green jelly beans are linked to acne, you would expect 5 teams to get a positive result. The 95 teams that found a negative result do not publish (because of course green jelly beans aren't linked to acne, duh!), and you get 5 papers that say green jelly beans are linked to acne and none that refute it. It's only when someone else tries to reproduce the result and fail that the negative results get published. Most scientists are understandably reluctant to try to reproduce already published results, because who is going to give them grant money to do that?

When you hear a news story that begins with "A new research study has just concluded ..." feel free to ignore it. One published study means almost nothing, even if it was a completely legitimate and honest result. This is why reproducability is one of the main principles of the scientific method.

Tyndmyr
Posts: 11443
Joined: Wed Jul 25, 2012 8:38 pm UTC

Re: Chocolate causes weight loss academic paper was a hoax

Postby Tyndmyr » Mon Jun 15, 2015 3:16 pm UTC

sardia wrote:
Willl wrote:
bigglesworth wrote:No, because journalists, even exactly the same ones that were "burned" will continue to do this. And there's nothing that scientists and science communicators can do about it.
Not true. They can educate the consumers of this media about these practices. A better-informed reader is less vulnerable to harmful influence by bad science, more receptive to good science and just might even demand less of the bad stuff. This hoax communicated insight into those practices more effectively to a wider audience than anything else I've seen.

That's inefficient. Is easier to just teach journalists what good or bad science studies are. Then they could be the have keepers of knowledge


Anything that relies on journalists being the keepers of knowledge is going to fail. Journalism is mostly the art of clickbaiting, etc these days. This is like relying on email spammers to be better informed about how pills make various bits larger, and believing this will solve spam.

They are not the cure, they are the problem.


Return to “News & Articles”

Who is online

Users browsing this forum: No registered users and 10 guests