In Woody Allen’s movie Sleeper, which is now 40 years old , a man (Woody Allen) time travels to the future, where (of course), they find him very strange. In this sequence, two doctors discuss their time traveler’s food preferences:
Dr. Melik: This morning for breakfast he requested something called “wheat germ, organic honey and tiger’s milk.”
Dr. Aragon: [chuckling] Oh, yes. Those are the charmed substances that some years ago were thought to contain life-preserving properties.
Dr. Melik: You mean there was no deep fat? No steak or cream pies or… hot fudge?
Dr. Aragon: Those were thought to be unhealthy… precisely the opposite of what we now know to be true.
Dr. Melik: Incredible.
When I read about nutrition science, I often think about this scene. Now, Edward Archer, a research fellow at the Arnold School of Public Health at the University of South Carolina, who makes somewhat less of the fact that he is also the Chief Science Officer of a South Carolina bio-informatics company whose business is helping people track their activity to improve their health, has recently been all over the nutrition news with this article:
Recently, I was the lead author on a paper demonstrating that about 40 years and many millions of dollars of US nutritional surveillance data were fatally flawed. In most research domains, such a finding might be monumental; yet in nutrition epidemiology—the study of the impact of diet on health, hereafter referred to simply as “nutrition”—these results are commonplace.
Nutrition has had many colossal and costly failures. The list of dietary components claimed to reduce cardiovascular disease (CVD), prevent cognitive decline, and/or fight cancer that were later refuted via clinical trials is extensive. And while the self-correcting nature of science necessitates failure, the vast majority of nutrition’s failures were engendered by a complete lack of familiarity with the scientific method.
So far, Archer and I are as one, except that he has more data. But then he starts to place blame, and first he blames the subjects of the studies:
This deficit is most apparent in the field’s reliance on self-reports of diet. Such information, to which nutrition researchers assign numeric caloric values, is rife with bias, and without the ability to corroborate or falsify the reports, the data should be considered pseudoscientific—outside the realm of scientific research.
Then he brings in his own vested interest, which he does not disclose.
Moreover, nutrition research fails to control for well-known, empirically supported, and in many cases obvious confounders. For example, the Food and Agriculture Organization of the United Nations and World Health Organization have repeatedly determined that human food energy requirements should be estimated using total daily energy expenditure, and that physical activity and basal energy expenditure are the primary determinants of this measure. Yet nutrition research rarely measures any form of energy expenditure or quantifies physical activity.
Not only does he keep quiet about his company, which makes energy expenditure tracking equipment, he doesn’t say that the major study he’s referring to (linked above) is based largely on “physically credible intake values,” or, in English, what quantifying scientists believe could be true about what study subjects say they eat. Finally, doesn’t acknowledge that self-reporting exercise and activity are not going to be any different from self-reporting food intake (perhaps because his company’s machinery provides “objective” evidence of energy expenditure).
And here’s the meat:
The subjective data yielded by poorly formulated nutrition studies are also the perfect vehicle to perpetuate a never-ending cycle of ambiguous findings leading to ever-more federal funding. The National Institutes of Health spent an estimated $2.2 billion on nutrition and obesity research in the 2012 fiscal year, a significant proportion of which was spent on research that used the pseudoscientific methods described above. The fact that nutrition researchers have known for decades that these techniques are invalid implies that the field has been perpetrating fraud against the US taxpayers for more than 40 years—far greater than any fraud perpetrated in the private sector (e.g., the Enron and Madoff scandals).
So, nutrition science sucks because:
1) People lie;
2) We count calories but not activity, and if we counted activity that would be reliable in a way that counting calories isn’t; and
3) Nutrition scientists are taught to keep their jobs and keep getting funding.
So, okay, people lie about what they eat. This is no different than any other kind of self-reporting; would he say that, for example, AIDS research into sexual behavior is equally pseudo-scientific? That jury pool questionnaires about past experiences are equally unreliable? Probably he would. What he doesn’t say, of course, is that one reason people lie about what they eat is that we know we will be either shamed (if we eat more than the researchers think we should) or disbelieved (if we eat less than the researchers believe we might), which is not any kind of motivation to tell the truth. And even if researchers are completely without judgment, we have an entire culture of food-shaming and disbelief experiences behind us, so it would be hard to recognize their nonjudgmental position.
For the record, I completely distrust this “physically credible intake” concept. Here’s one story, which April Miller told in Women En Large. Half the readers of this blog probably have similar ones:
When I was in ninth grade, I entered the hospital for a month on a strict five-hundred-calorie-a-day diet. I weighed in every morning, and one morning I’d gained three pounds. My doctor screamed at me. He said I could only have gained weight by sneaking candy. That diet was my last-ditch attempt to be “normal.” I knew that I had done everything perfectly and that those three pounds were not my fault. Maybe none of it–my weight, the way people treated me–was my fault.
April’s story illustrates at least two problems with Archer’s approach. First, the strictest hospital controls (just like the strictest prison controls) will never prevent some people finding ways to break the rules, so there’s no alternative to some level of self-reporting. Second, it is possible to gain three pounds (overnight) without sneaking candy. It isn’t plausible that a person could eat enough candy to gain three pounds overnight, unless something else is going on, which April’s doctor (and nutrition scientists) should realize instantly.
In this context, notice this recent reversal of common nutrition wisdom. This very large study tracked milk drinking in teenage years against hip fractures in older adults, and found that additional milk was neutral in women and actually risky in men, especially taller men. The study by its nature called for self-reporting of behaviors from 30-40 years prior to the study, which would tend to reduce the quality of self-reporting, but milk intake is less likely to be a shaming/disbelieving area, which would tend to increase the quality of self-reporting. By Archer’s standards, should we believe it? To be consistent, he would have to throw it out, but I suspect that a) it’s probably reasonably good science because of the large number of people (which also tends to balance out bad self-reporting, unless that bad self-reporting is completely consistent) and because of the confirming indicators (like tracking to men’s height).
Tracking energy expenditure is probably a good idea. In this age of Fitbits and pedometers and Wii’s (and expensive bioinformatics equipment like the things Archer manufactures), we can do some of this. And if we do it carefully and respectfully and well, we will probably learn something about how exercise and diet and weight work together for (if we’re lucky) most people most of the time. But that will still not explain the outliers, and it’s very tricky to set standards for most people without shaming the outliers.
Getting scientists to design carefully scientific experiments rather than experiments that will help them keep their jobs and get funding is hardly specific to nutrition scientists. Laurie and I tend to stay away from pure political blogging, but any good anti-corporate analysis will deal with this point. Unsurprisingly, Archer (who is, after all, pursuing the agenda of his own small corporation) sidesteps it completely in his conclusion.
The solution to this dilemma is quite simple: funding agencies must stop funding flawed nutrition research, and the editors of nutrition journals need to stop publishing the results. Given the immense amount of money invested in this field each year, this goal is much easier to state than to accomplish. Nevertheless, the health of our nation depends on nutrition finding a scientist that can disperse the wolves and lead the overly credulous nutrition flock to more productive pastures and empirically supported public health policies.
You have to hand it to him. I even agree with him. If funding agencies stopped funding flawed research (even assuming any kind of consensus on what’s “flawed”), the money would stop flowing even more than it has in the last five years. And if journal editors stopped publishing results of flawed research, the scientific journals business would be thrown into chaos. But to talk about these things without mentioning the flow of corporate money into science is to engage in flawed reasoning (which is where flawed research comes from).