Manheim Steamed-over Paleo: The Shai'ning of Mellberg

The controversy should serve as a warning about meta-analyses, Willett adds. Such studies compile the data from many individual studies to get a clearer result.
"It looks like a sweeping summary of all the data, so it gets a lot of attention," Willett says. "But these days meta-analyses are often done by people who are not familiar with a field, who don't have the primary data or don't make the effort to get it." And while drug trials are often very similar in design, making it easy to combine their results, nutritional studies vary widely in the way they are set up.
"Often the strengths and weaknesses of individual studies get lost" 
"It's dangerous."

Walter Willett MD
Speaking about Chowdhury,
(HT: Colby Vorland of

Yes.  This.  

With apologies to Eric Manheimer, lead author on That Paleo Meta-Analysis, I just couldn't help myself playing name games with my blog post title here.  This meta analysis serves nothing but to re-heat a trial that landed with a thud.  For ALL the statistical mumbo jumbo, none of the Cochrane acronyms mean a thing if the original data is compromised.  This is why I posed the rhetorical question of when the time is right for a systematic review and meta-analysis.  The time is not there for the "paleolithic nutrition pattern", whatever the heck that means, and this is evidenced by the lack of data to analyze in general, even before its quality is judged.

One of the study authors, Zbys Fedorowicz (aka Guest) posted the following here:
“If the research question is important and there are no reliable data available with which to address it, then the world needs to know that. There are several potentially useful consequences of doing this. Firstly, a reader may draw attention to a trial, which has been overlooked. Secondly, a patient may realise that their preferences should dominate in any decisions about choice of treatment, and thirdly, researchers and research funding organisations will see that there is an important gap in reliable evidence and take steps to plug it. Cochrane reviews can be - and have been - helpful in all of these ways”. 
Sir Iain Chalmers, Cochrane Collaboration Founder 
Hmmm.   Well, leaving aside whether the research question of recommending a "paleo diet" ... erm ... "paleo nutrition pattern" is an important one, we only need the literature at hand to demonstrate that "there are no [exceedingly little] reliable data available with which to address it".  And so "the world needs to [now] know[s] that.   Rather than ...
The available data warrant additional evaluations of the health benefits of Paleolithic nutrition.
... as Manheimer concluded, the available data warrant additional evaluations prior to conducting any sort of formal statistical review.  Just to be clear, the Paleo Meta was NOT a Cochrane review, rather three of the four authors are Cochrane reviewers -- a subtle distinction obscured by Fedorowicz's repeated dropping of the Cochrane name and acronyms here.

I'm going to address some of the issues with meta analyses in general, and why this current study exemplifies most of those issues, in a future post.  However what single long term paleo study, Mellberg, is being transformed into the paleo equivalent of what the Shai study has become for Atkins / LCHF / keto / etc.  By virtue of inclusion in a meta-analysis entitled  Paleolithic nutrition for metabolic syndrome: systematic review and meta-analysis, the community will have a link to an article in a prominent journal that transforms Mellberg from ho hum nothingness to shining example of long term effectiveness in treatment of metabolic syndrome.  


I have a new version of the summary table in the works.  It's "worse" than the original in terms of study comparisons and data used in the meta.

Before revisiting the nitty gritty of Mellberg, here's one example of how this meta-analysis (hereinafter MA) is being billed -- an "auto-tweet" from Dan Pardi of Dan's Plan.

But the changes were NOT significant for two of the five outcomes, in such a -- pardon me -- LAME meta-analysis.  The auto-tweet will perpetuate this myth that this MA showed some meaningful wholesale effect, and further inflate the already irresponsible "Shai'ning" of the individual study results, Mellberg in particular.

For these 5 characteristics of MetSyn, here is a table of the results and statistical significance from the original Mellberg paper.  Note that I've also included the percentage weight this study had in the MA.  

Note:  HDL and FBG were not significant factors in the meta-analysis
It is important to note that these subjects were obese, but otherwise in generally good metabolic health.  There is no mention of metabolic syndrome anywhere in the original publication. Women with a history of CVD, diabetes or abnormal fasting blood glucose were excluded from participating in the study.  Only three subjects were on ACE-inhibitors for "mild hypertension", and only mean waist circumference was in the MetSyn range.   It's possible there were a few, perhaps even several, women in this study who would qualify as having MetSyn, but there is no mention of this in the primary report, no accounting for the numbers of individuals or their distribution between study groups.

In terms of Mellberg's impact on the MA results, I've highlighted its almost 43% weighting for Systolic BP.  You will note that neither the paleo nor the control groups had a statistically significant change in systolic BP, and there was no statistical significance to the difference between groups.    Note also that the variation is shown as standard error (SE) instead of standard deviation (SD).  For this measure, the SDs were on the order of the magnitude of the mean change.  These are reported as such in the full MA results that I've included as a small inset at right.  As always you can click to view and resize.   

Before revisiting problems with Mellberg itself, the two biggest issues with its inclusion in this MA are as follows:

  • Subjects did not have Metabolic Syndrome
  • Despite factoring heavily into the pooled effect in the MA, there were no statistically significant changes in Systolic BP.

The MA now, by "reverse engineering", implies an effect in Mellberg that was not observed.  But let's not forget:

The Mellberg Dietary Interventions Revisited

There are some things that are rather odd about Mellberg.  From the introduction:
Short-term studies have suggested beneficial effects of a Palaeolithic-type diet (PD) on body weight and metabolic balance. We now report long-term effects in obese postmenopausal women of a PD on anthropometric measurements and metabolic balance, in comparison with a diet according to the Nordic Nutrition Recommendations (NNR).
The primary outcome for this study was fat mass (measured by DXA), so this was a weight loss study.   Why were both diets administered ad libitum?  I get that the PD is generally considered to "work" by spontaneously reduced intake due to the higher satiety factor of the foods consumed (protein and fiber).   But the Nordic Nutrition Recommendations has a fairly lengthy chapter on Energy, including the following:
Reference values for energy requirements are based on assumptions regarding weight stability, normal (healthy) weight, and energy balance. However, these assumptions are not always valid. For instance, a negative energy balance is needed for the treatment of obesity. If energy intake is 2.1 MJ/d below the requirement for energy balance, the estimated weight reduction during the first month is approximately 500 g/week. This rate of weight loss is often recommended although a larger negative energy balance (up to 4.2 MJ/d) leading to a weight loss of 1000 g/week still seems to be compatible with a healthy weight reduction (8, 90). The long-term estimation (several months to years) of weight loss due to a fixed reduction in energy intake is much more complicated (91). The reason for this is that energy expenditure decreases with weight loss. Hence, with increasing weight reduction the energy deficit decreases (same intake but less expenditure). Therefore, the 500 g/week weight loss for each 2.1 MJ (500 kcal) reduction in energy intake cannot be used for anything other than predicting initial weight reduction
It is quite odd, then, that the NNR group did not have a target caloric intake geared towards establishing a target deficit.  Surprisingly, both groups self-report similar caloric reductions and intake at 6 months.  

This is just the baseline and 6 month dietary intake table.  This data came from 4-day logs where amounts were *estimated* using pictures and measuring devices.  Apparently this was assessed monthly, but it was only reported at 6 months.  IF these data are to be believed, both groups were in similar caloric deficit.  Other reasons to suspect that they are not aside, a peek at the 2 year data (see original post) has PD reporting slightly lower caloric intake and NNR slightly higher (while still in deficit) compared with 6 months.  This corresponds to slight weight and fat regain in PD and further but small weight and fat loss in NNR.  This brings the accuracy of reporting into considerable question. 

Note NNR ranges:  Carb 55-60%, Fat 25-30%  
Table uses averages
I discussed this study at some length HERE.   Another partial table from that post is included here as well, this time the dietary treatments and actual intake.  The NNR group barely altered their dietary pattern.  While the PD group altered theirs significantly, they failed to meet targets.   

One last thing (direct quotes from study text):
  • Nitrogen excretion in urine (NU) was used as a biomarker for protein intake, with three 24-h urine samples collected at baseline and after 6 and 24 months.
  • Nitrogen excretion did not differ between groups.
  • Adherence to protein intake was poor in the PD group.
OK, I lied.  One more thing.  The NNR intervention, other than macro percentages, includes only the following additional specifications in the study:
  • ... with emphasis on low fat dairy products and high fibre products.
It would seem that adherence to consumption of high fiber foods was poor as absolute fiber intake was essentially the same as baseline (PD actually declined slightly).  Even though intake was lower, an emphasis on high fiber foods should have resulted in some increase in fiber intake.   

One last afterthought here, I promise.  I mentioned this in previous posts on this study.  If you are not setting out at least a caloric guideline, it is impossible to recommend that people consume macronutrient percentages.   Obviously NNR subjects could choose low fat foods, etc., but without a calorie target, resulting in a gram target, "aiming" for percents is futile.  I think this factored into the bad adherence here.

Mellberg did NOT test a Paleolithic Diet vs. Nordic Nutrition Recommendations.

THIS is the main issue with the Paleo Meta.  None of these issues is discussed in the MA.  They would have been (or should been) had an appropriate general review of the literature been done instead.   While the authors run about claiming that recommended diets underperformed compared with paleo, they are not being truthful, because the subjects -- at least in this, the largest of only four studies -- did NOT follow the recommended diet (or PD).

Thanks to being a cornerstone of "the first Paleo Meta", Mellberg is well on its way to becoming the Shai of Paleo.  Below is from just one of many posts I've written on that Israeli diet trial in mostly men.  

While Shai is held up as an ultimate test of calories vs. carbs by the likes of Gary Taubes and the be all and end all best of the best quality studies on high fat diets by Nina Teicholz, it too suffers fatally from "treatment fail".    As can be seen, the LC diet is nothing like #LCHF "in current practice" (much like the PDs in clinical trials bear little resemblance to anything you'll see served at PaleoFX).  There's also considerable discrepancy between the 1 year intake data reported in the original NEJM article, and reported in another publication as assessed by different means in a subset of the participants.    Furthermore, the LC and LF groups were regaining after around 6 months and had plateaued out by the 1 year mark.   In other words:  This intake data is crap.  Therefore the study outcomes are essentially meaningless.

The Cochrane Collaboration and some of the authors of this current MA seem concerned with reducing wastage in medical research.  I hope they'll join me in wasting no further time and resources analyzing irreparably compromised data from free-living, low-compliance RCTs.  Furthermore, perhaps they'll join me in Calling for the End of Diet Comparison RCTs.

Coming soon (hopefully), I'll discuss meta-analyses in general using Manheimer as an example of both a (horribly) bad example of such a study, and how issues with the individual studies such as those discussed in this post can render many doubley problematic.


Blogger said…
Using EasyHits4U you can earn free advertising credits by visiting other website-ads from a member base of over 1,200,000 accounts. Earn advertising credits fast with a 1:1 exchange ratio.