Zoe Harcombe and Adele Hite's Hyper-System(at)ic Meta-Statistical Bovine Fecalemia

Alternate Title:  Baffling with Bull $#!†

This post discusses two "research articles" recently published in peer review journals, and how their use of statistical terminology:

• imparts undue scientific seriousness to the content of the paper
• obscures the fact that there is nothing new in the paper to make it even worthy of publication, and
• allows the authors to assign scientific significance to editorial opinions that are at best not supported by the statistical analysis in question, at worst directly contradicted by it.

These two "studies" both boil down to exploiting statistical jargon and methodology to further an agenda.  It's a darned shame the quality of the peer-review process has declined so precipitously in recent times as to allow such obvious examples to get through.  

---

Introduction

The past couple-few years have seen a spate of meta-analyses and such looking at dietary patterns, etc.  I'm not a huge fan.  While the meta-analysis holds such great promise for the scientific research community as a whole, in the wrong hands -- the hands of someone with an agenda -- it can be misused to great harm.   The terms meta, systematic and/or statistical in the title of a paper impart a seriousness to the effort that many bearing the word(s) in the title fail to deliver.  In contrast to the "mere" review paper, these terms imply that the authors have brought the math (or maths) to the science table.    At the very least, they didn't just conduct a few random searches,  or rely on their hard drive or file cabinet for material.  No ... they exhaustively searched on every conceivable string to amass a giant collection of scientific works, to which they then applied a strict filtering process for inclusion while taking the utmost care not to exclude research based on outcomes inconsistent with personal bias.  And then ... they analyzed ... and analyzed some more.

Alas, most meta-analyses, systematic reviews and statistical reviews of late have been a lot like some prescription strength Motrin:  more expensive, but no more effective than two or three generic OTC ibuprofen ... all the while less accessible and with the added risk of gagging on the horse pill.

It's 2015, the US Dietary Guidelines Advisory Committee (DGAC) just came out with its new report, and sometime later this year new Dietary Guidelines will be brought forth from on high.  This is big business time for all of the counter-advocacy groups out there!  The timing of two recent "peer review" papers is no exception.

The One About There Being
No Evidence for the Dietary Guidelines

First up, Zoe Harcombe.  Diet book author and self-styled obesity "researcher" managed to get a "scientific study" published in Open Heart:

Evidence from randomised controlled trials did not support the introduction of dietary fat guidelines in 1977 and 1983: a systematic review and meta-analysis

This online pay-to-publish journal of the British Medical Journal Group is the place to make a splash in the nutrition world these days.  For a mere $3000, if you can stir up enough controversy to get the attention of the press office, you'll see publicity galore!  Apparently it helps to get with the press office if an Associate Editor by the name of James DiNicolantonio is also an author.  OH essentially advertises for submissions with this publicity carrot (with lashings of butter of course), and the response to this paper in the mass media rags demonstrated their abilities to generate empty hype.

As previously mentioned, Harcombe et. al. get their timelines wrong as regards the US guidelines, but putting that aside they state:

Although a number of reviews of RCTs have been undertaken,8–10 no review has examined the RCT evidence available at the time dietary fat guidelines were introduced. Furthermore, these guidelines have not been changed since they had been announced; correspondingly, the validity of their evidence base remains relevant.  This systematic review and meta-analysis will assess if  the published RCTs available to the dietary committees supported their recommendations on dietary fat. With this in mind, we hypothesised that RCT evidence available to the dietary committees at the time of issuing recommendations did not support the contention that reducing dietary fat intake would contribute to a reduction in CHD risk or related mortality.

Why a Meta-Analysis to Begin With?

Kevin Klatt over on Nutrevolve, after looking at the studies analyzed, made the following comment:

I'm curious why the authors didn't just skip the meta-analysis and say "there wasn't a significant body of primary prevention trials from this period demonstrating an effect of low or modified fat diets on CVD outcomes" .

EXACTLY!!!  Before even looking at the trials they discussed or the whole meta-analysis mumbo jumbo, I was shaking my head thinking the very same thing!  The recommendations from McGovern's committee in 1977 were not based on RCTs,  admittedly so.  But this isn't news, nor does it sound very sciencey to just point this out.  No, we have to do a meta-analysis so that some journal will publish it as a scientific research article instead of yet another editorial or review (really, they are practically all the same in Open Heart anyway).  That's the only point to the excercise in futility that I can think of.    The six trials simply did not qualify to be considered together in any sort of "meta".  Period.  

If not clear from the authors' descriptions, a little reading further and/or checking of abstracts would make it clear in short order.

• None were reduced fat
• Only one was reduced sat fat
• The Rose oil experiments, in particular, had very poor compliance
• Weight loss/gain differed therefore caloric balance differed
• Most were in those who already had a history of heart disease and/or of a myocardial "event".
• Other lifestyle interventions and/or medication varied widely
• Sample size and duration alone pretty much nullify comparisons for hard outcomes.

As they wrap up the discussion:  

In the absence of epidemiological evidence from whole-populations, large-scale RCTs of longer duration (with adequate follow-up), which accounted for known confounding variables and included primary participants of both males and females, may have supported the introduction of dietary fat guidelines in 1977 and 1983. However, this opportunity expired when universal pharmacological treatment became the accepted norm.

Which, to echo Kevin, pretty much begs the question, why do the meta-analysis at all.   To divert attention, that's what.  All the discussions of homogeneity, PEDro scales, forest plots, tunnels, etc.  It was all meaningless fluff, after which the reader received a litany of one study this, and one study that, and a few criticisms of LA Veterans, and no mention of issues with Rose that may have played a role in the control group besting the corn oil intervention.

The Guidelines (issued in 1980 in the US, and that did not include specific 30%/10% recs until 1990) were not based on a lot of clinical data.  I don't see where they ever claimed they were.  Harcombe's "expired opportunity" fatalism is nothing but pure editorial comment.  Slide it in with some meta-analysis and perhaps it will be taken as the serious conclusions  of well informed investigators?

The summarizing continues:

From the literature available, it is clear that at the time dietary advice was introduced, 2467 men had been observed in RCTs. No women had been studied; no primary prevention study had been undertaken; no RCT had tested the dietary fat recommendations; no RCT concluded that dietary guidelines should be introduced.  It seems incomprehensible that dietary advice was introduced for 220 million Americans and 56 million UK citizens, given the contrary results from a small number of unhealthy men.

Again, this could have been said and debated in editorial fashion, but without the seemingly serious sounding meta-statistics, it sounds, as it is, to be nothing more than argumentative editorializing.  There's nothing about the meta-analysis that adds to this argument.  You don't need a meta-analysis to say that studies were not done in enough representative samples of the population as a whole.  And again, "It seems incomprehensible ..." is opinion and is not supported by the statistics, however meaningless they may ultimately be.

The Mega-Meta Error

By specifying the number of men involved -- 2467 -- Harcombe exposed just how poor the statistical analysis performed was.  Additionally, this exposed the "rubber-stamping" by co-authors and the less-than-diligent external peer review (if it really happened at all) process at OH.    Here is a screen-shot of the tabulated data used for the analysis of all-cause deaths.

There is a similar one for CHD.  It doesn't really matter which table I use as an example.  What stuck out at me was that the Rose oil study was "itemized" for the two study groups ... basically listed as two studies.  This was a side-by-side comparison involving two intervention/study groups and a single control group.  Nothing wrong with that, but this makes the numbers a bit tricky when it comes to a meta-analysis.   There was no indication of how this was handled in the text, so I didn't know if Harcombe had even realized this until I read first sentence of the previous paragraph above:

From the literature available, it is clear that at the time dietary advice was introduced, 2467 men had been observed in RCTs.

That 2467 is wrong.  No, what is clear is that these studies included 1240 men in intervention groups and 1201 men in control groups for a total of 2441 men.  The 26 men in the Rose control group are counted twice in the 2467 figure used by Harcombe.   It is probably safe to assume that all of Harcombe et.al.'s statistics are incorrect.  While the difference is likely negligible, the only thing worse than needless statistical analysis would have to be needless statistical analysis that is wrong.

Double counting like this is a big no-no.  The way the table was itemized, it highlighted something that might have otherwise gone unnoticed.  So this brought it to my (though apparently not anyone else involved in writing and reviewing the article's) attention, and nowhere amidst all of the highfalutin statistical nomenclature thrown around in the methods was this addressed.  None of the co-authors, not even PharmDr. James DiNicolantonio, king of the secondary statistical analysis himself?  Nope, apparently not.  This is shoddy analysis at best, and should not have made it into publication.  Don't believe me?  From the Cochrane Handbook for Systematic Reviews of Interventions:

16.5.4 How to include multiple groups from one study

There are several possible approaches to including a study with multiple intervention groups in a particular meta-analysis. One approach that must be avoided is simply to enter several comparisons into the meta-analysis when these have one or more intervention groups in common. This ‘double-counts’ the participants in the ‘shared’ intervention group(s), and creates a unit-of-analysis error due to the unaddressed correlation between the estimated intervention effects from multiple comparisons (see Chapter 9, Section 9.3).

Harcombe boasts a Cambridge degree in math and economics.  

In the end, this error wouldn't have altered the numbers much anyway.  That isn't the point, but this reality in the analysis is instructive as well.  You had six studies, seven comparisons.  The two oil comparisons contributed roughly 1% each towards the meta for CHD deaths and less than half-a-percent to the all-cause meta according to the authors.  Meanwhile one study (LA Vets) dominated all-cause at just over 40% and another (Oslo) almost 35% for CHD.  When you have such a small number of studies to pool to begin with, it helps if the sizes are at least somewhat similar between them.  Otherwise you risk augmenting the results of a large study by tacking on a bunch of small "me too" studies that were insignificant in their own rights, or diluting the results of a large study with meaningless outcomes.  But omitting Rose means you can't point out one intervention as being bested by the control, and six-looks-like-seven becomes five.  Ditch the 1978 study that wasn't available to the 1977 Committee, and your lame meta-analysis becomes tragically so.

Systematic Editorializing ...

This paper is becoming quite the black mark in the short history of Open Heart.   The final paragraphs were as follows:

There was best practice, randomised controlled trial, evidence available to the dietary committees, which was not considered and should have been. The results of the present meta-analysis support the hypothesis that the available RCTs did not support the introduction of dietary fat recommendations in order to reduce CHD risk or related mortality. 

Two recent publications have questioned the alleged relationship between saturated fat and CHD and called for dietary guidelines to be reconsidered.31 32

The present review concludes that dietary advice not merely needs review; it should not have been introduced.

The first paragraph above is strange.  Are they insinuating that there was other RCT evidence the committees ignored?  Why no citations?  Sigh.  But the ultimate conclusion is nonsense.  The statistical review --  methodological errors aside -- went nowhere towards discussing the validity of the dietary advice and all of the evidence such advice was based on.  In that regard, an actual review of all of the data, along with discussion of the nuances of the trials -- e.g. as was provided for LA Vets and the smoking issue, but not for LA Vets and the fake food/potential hyrogenated fats/trans fats issue -- would have been the appropriate approach.  Instead we got a hyper duper case of bovine fecalemia .

The One Claiming Americans 

Followed 

Dietary Guidelines and This Made Us Fat

There was another recent shot made across the DGAC bow, this time by Adele Hite of the Healthy Nation Coalition -- an organization pretty much dedicated to advocacy against current USDA/HHS Guidelines.  HNC and Hite are staunch low carbers, masquerading as moderates.  You cannot have Robert "Carbohydrates Can Kill" Su (deceased, 2013) on your advisory board, and Adam "Caloriegate @nocarbsgo" Kosloff  as your Outreach Chair, and continue to claim otherwise.   

The article in question is:  Statistical review of U.S. macronutrient consumption data, 1965–2011:  Americans have been following dietary guidelines, coincident with the rise in obesity 

Go ahead and laugh at the subtitle there.  This is authored by Adele Hite and members of an economics analysis firm (more on that perhaps another day).

Background:  For almost 50 years, the U.S. National Health and Nutrition Examination Survey [NHANES]  has measured Americans’ caloric consumption, and body heights and weight. This paper presents the first comprehensive analysis of that data, documenting how macronutrient consumption patterns and the weight and body mass index in the U.S. adult population evolved since the 1960s.

Hite goes on to explain that she is not using this

graph to imply causation.  Oh no!  She wouldn't 

do that!  link to post

You're kidding, right?  What new can they bring to analysis of NHANES data?  It has been analyzed extensively.  But I guess the claim is that this is the first effort to see how this correlates with obesity.  Didn't Adele already do that?  

The Major Findings:

The major findings of the study are repeated a few times in the text, I'll use a numbered list (to replace first, second, etc.) from the Introduction:

1. On average, adults in the U.S. have adhered to the government’s dietary recommendations in shifting their proportional consumption of total calories from fat to carbohydrate. Fat consumption dropped from 44.7% in 1965 to 33.6% in 2011, while carbohydrate intake increased from 39.0% to 50.5% over the same period. 
2. The reduction in the proportion of fat and cholesterol consumption arose largely from an increase in the daily intake of carbohydrate by about 260 calories from 1965 to 2011. 
3. There is a high correlation (close to 90%) between the change in dietary pattern and the rise of obesity over the last 40 years. From 1971 to 2011, average weight and body mass index (BMI) have increased dramatically, with the percentage of overweight Americans increasing from 42.3% in 1971 to 66.1% in 2011.

The Data & Re-Sampling:

The novel statistical analysis that was purportedly done by the authors involved "resampling" of the data to adjust for demographic shifts in race, gender (?) and socioeconomic strata in the population, as well as intentional over-sampling in various NHANES of certain subsections of the population.  But NHANES began in 1971, and no amount of statistical massage could transform the NHANES data to make it substantively different as you'll see here in a moment.  Further, the obesity data used also dates from 1971.  The Dietary Goals didn't come until 1977, and the Guidelines until 1980.  The analysis purports to demonstrate adherence to the Guidelines, so there is no apparent reason to go back further than 1971 for a baseline.  

Why, then, did the authors see fit to include data from the USDA's 1965-66 Household Food Consumption Survey?  A quick perusal of this document and it becomes obvious that the reporting is not nearly as detailed.  For example fats are not itemized, and the term saturated doesn't even appear in the document.   Furthermore, this data was obtained by food recall of the homemaker for all members of the household.  Meanwhile NHANES is filled out by the individual (over age 12).    In addition, this 1965 study from the USDA is described in the paper as a "predecessor study" to NHANES, which is not exactly the case:  Key Concepts About the History of Dietary Data Collection.  NHANES was started by another department and existed concurrently with the USDA surveys until efforts were combined in 2002.  Given as there were such nominal differences between fat intakes between the NHANES years, the decision to include this outlier, without any effort to reconcile, is more than a little bit suspect.  To be fair, I do not see an equivalent to Table 93 in the 1965-6 in the 1977-8 Survey with which to compare with NHANES. 

So the question becomes:  Why was the 1965-6 used at all?  

The authors make no case for its inclusion despite the inconsistency with all of the other data.  The inability or lack of any attempt to reconcile the USDA and NHANES data is even MORE reason why the 1965 "outlier" should never have been introduced at all.   To make matters worse, the main "novel" feature of this paper is supposedly the "resampling of NHANES" to adjust the numbers for possible skew due to oversampling and shifting demographics.  They must remind the readers that they could not do these same adjustments on the 1965 survey data ... because they didn't have the same detail available.  

The 1960's studies on fat (see Harcombe above) pretty routinely used aroun 40% fat as "normal" intake.  The McGovern committee recommended -- in 1978 -- reducing fat from around 40% to 30%.  The 1971-1974 NHANES had fat intake at 36.9% for men and 36.1% for women (both age adjusted, which didn't differ from crude average for men and by 0.1% from 36.0% for women).  From the data available -- if it is taken at face value -- Americans started around 3% below where estimates of the day had us ... we didn't have that far to go.

Here's the crunched down table for the most comprehensive single source table  I could find.  It doesn't go through the 2011 start year included on the plots in this study.

Here are the results of "statistical re-sampling and adding an unrelated data point" by Hite & Economists.

As you can see, the resampling seems to have worked to inflate the NHANES fat percentages by about a percentage point for the 1970s (1971,76) and mixed it up a bit for the additional coinciding years.  The result is no big difference.  The biggest effect seems to have been achieved by including the non-NHANES data from 1965.    So much for re-sampling mumbo jumbo.

Finding 1:  Americans Adhered to Dietary Guidelines

This is a joke, right?    No honest person could look at the data and say that Americans adhered to the guidelines.   And their re-analysis?  How much time did the economists spend on this "statistical stuff"?   It didn't alter anything in NHANES significantly.  The Guidelines didn't include the 30% specific until 1990.  So even if we focus on just fat percentages, this virtually flatlined around 33-34% since 1988.    The increase in carbohydrates without respect to total intake was never part of the Guidelines.  

Here's what happened with absolute macronutrient intakes.  One of the authors used their chart generator to create it!  Ignore the 1965 data, it is irrelevant.  You see the change in the height of the bars while the gray and dark blue remain relatively constant?  That was not what the Guidelines ever counseled.  But let's focus on the fat intake.  I even stretched the vertical axis some to accentuate things on the image below (further adapted to create this meme).

Americans complied.  Puh - leeze!

Finding 2:  Americans Ate More Calories Because They Ate More Carbohydrates:

This is not news. No resampling or even tossing in an extraneous data point was needed -- although it makes it look "significant". The authors wrote that the "reduction in the proportion of fat ... arose largely from an increase in the daily intake of carbohydrate".  This much is readily apparent in a two year old publication in the American Journal of Clinical Nutrition :  Trends in energy intake among adults in the United States: findings from NHANES.

Note that the protein and fat virtually overlap in terms of grams, and are virtually constant over time.  The 2013 plot accentuates how the increased intake is one of increase in carbohydrate intake on a gram basis. 

   

As to the percentages, I highly doubt that resampling had much effect on this plot, which was nothing new.  The plot below is the complement of the one above, from the Trends paper.  That is but one of many but includes more recent data than some.

Lastly, Nina Teicholz, author of The Big Fat Surprise, is acknowledged for her "support and editorial contributions".    Apparently nobody remembered that in Teicholz's own reference on macronutrient intake for the book, they stated:

The decrease in the percentage of kcals from fat during 1971--1991 is attributed to an increase in total kcals consumed; absolute fat intake in grams increased.

Finding 3: Close to 90% Correlation of Diet and Obesity

This finding is the most curious of all, especially since the article is touted as being some sort of statistical analysis in the title.  Up to this point, the only statistics they've discussed that are anything new would be their "re-sampled" NHANES values.  But they didn't present any calculations or summary, and comparing to existing data, it is easy to see that any changes are inconsequential to the interpretation of NHANES data.  There is nothing at all new about their analysis otherwise.    

So then comes finding 3:

There is a high correlation (close to 90%) between the change in dietary pattern and the rise of obesity over the last 40 years.

After discussing a superfluous analysis of NHANES to determine a change in the distribution of obesity, there is this plot of BMI for men and women.  Where did the 1965 data come from?  It is not sourced, and it is impossible that it came from NHANES.   

Eventually we learn the following:

There is a strong relationship between the increase in carbohydrate share of total intake and obesity. This is reflected by the high correlation at the individual level between caloric share from carbohydrate and adult BMI: 85.3% for men and 91.2% for women.

So the "dietary pattern" is carbohydrate share of total intake, but this is misleading.  Changes in carbohydrate share are entirely due to eating more carbohydrates without cutting fat, and thus more calories.   It would have been helpful to show a scatter plot for this and report either correlation coefficients, r, or the r-squared values.  Alternately, how about yet one more composite plot?   In either case, there may be another rookie statistical mistake on display here.  It seems impossible to me that the correlation coefficients could be 0.9236 and 0.9597 for men and women respectively to translate r-squared values to these percentages.  (See here).    Even correlation coefficients of 0.853 and 0.912 seem high, but per my link:

Many students who are new to the concept of correlation coefficients make the mistake of thinking that a correlation coefficient is a percentage. They tend to think that when r = .90 it means that 90% of the changes in one variable are accounted for or related to the other variable. Even worse, some think that this means that any predictions you make will be 90% accurate. This is not correct! A correlation coefficient is a “ratio” not a percent.

Until I have time to pull numbers from their plots to do the correlation to determine the correlation coefficients, I cannot be sure if this is just shoddy work or a common student mistake in statistics.  In any case, even if the correlation bears out, this is lazy presentation as there is no discussion of possible other factors, (hello calories!!), nevermind neglecting to caution regarding correlation and causation.    Ancel Keys would never do such a thing (and he didn't).

What we have here is an example of advocacy in disguise.  More hyper duper bovine fecalemia.  

---

It's really quite disheartening that the peer-review process has abandoned their standards and allowed this to happen more and more frequently.  Can the Titanic be turned around before it hits the Kerrygold iceberg?