That Paleo Meta Analysis

Screenshot of Meta-analysis lead author
Eric Manheimer's article on Cordain's
ThePaleoDiet commercial website.

Is it that time already? For systematic review and meta-analysis? Apparently someone thinks so and managed to get some time at or off of work to do one on the paleo diet. Seriously? What even is the paleo diet? I have asked that question many times here myself. If a premier expert in paleolithic nutrition cannot provide an answer, then who can?

I've also written on this many times. I believe that THIS POST is a great place to start as it contains links to the various studies and blog posts I've done on those studies. I'll repeat some links here in a bit.

These were the clinical trials (not all randomized-controlled) to date as of January 2014. I've included the purported composition of "paleo diets" as well.

Direct links to blog posts: Frassetto, Lindeberg & Jönsson, Ryberg, Osterdahl

There were two additional studies published in 2014. One of these was a two week trial (Boers, et.al.) that I did not even bother to blog on. As far as I'm concerned, you can give me all the excuses in the world about funding, etc., but at this stage in the game, if you can't at least do a month (seriously, for a diet trial, 3 months should be some kind of law), don't waste the money. Shorter time frames are only valid for mechanistic investigations, not general dietary influences on long-term health predictors. The clinical paleo diets are hardly that extreme after all: No bacon, butter and butternut squash bolognese. Ethics should not be an issue.

The other new study on the block was the sole long-term trial, two years in postmenopausal women. This is presumably one of these promised EIGHT studies, LOL. Robb's hot air aside, the singular long study was rather more a bust and disappointment than anything, though it did provide some data with which to discuss the topic of appropriate caloric intake for your average woman.

But back to paleo, Robb was so excited by this study that he invited his critics perform inappropriate acts on a married man. Yeah, just letting newer readers in on the psyche of one of my favorite men of paleo ;-)

Still waiting on the paper on paleo and AI. So now we have ....

Paleolithic nutrition for metabolic syndrome: systematic review and meta-analysis

I'm going to do something a bit out of the ordinary here. I'm going to write this post without reading the full text of this paper. Why? Because I want to have a discussion on what warrants a systematic review and/or a meta-analysis. When it comes to "paleolithic nutrition" we're not even close to any meaningful body of work such as to warrant either.

Adel Moussa of Suppversity has written favorably of this analysis. I respectfully disagree, and it has nothing to do with the diet per se. Thanks to this article the 4 RCTS (could they have at least listed these in the Abstract?) are identified.

Here is the registration in PROSPERO. Interestingly, they mention other studies that were excluded because of lack of control, etc. but don't mention Frassetto. (EDIT: My mistake, it was mentioned, doesn't change matters anyway as it was not a controlled trial and lasted only 10 days). That study wouldn't have made the cut either. At the very least, the author's employers gave them the time and support to do this analysis. The first thing one might note, just from the study titles, is that two of them are pilot studies. But forget all statistical mumbo jumbo. Do the four studies that made the cut, pass the common sense smell test to be grouped together? I constructed the following table.

Let's run down the table roughly in order here.

Timeframes: You're mixing a 2-week study up with two 3-month studies, and the earliest 6 month assessment from a 2-year study. This doesn't pass muster with me. Changes in metabolic parameters that may appear at 2 weeks may be gone by 3 or 6 months, conversely effects seen at longer time points will likely not be significant at 2 weeks. There really ought to be a scientific law somewhere that cuts this meta-analysis off right here.
Numbers: As to the total number of subjects on the paleo diet, 14 + 13 + 18 + 35 = 80. One study comprises 44% of the subjects.

It begins to get worse when we're talking about the subject demographics. This is the purpose of these meta-studies after all, presumably to identify consistency across enough studies to warrant general recommendations.

Gender: One study was in all men, one in all women, one in mostly men, one in mostly women. All in all, 29 men and 51 women.
Age: The lowest average age for all studies is around 53, the rest are 60 and above. This is probably one of the few characteristics where these four studies are acceptably lumped together.
Health Status: One study is in subjects with existing ischemic heart disease and either IGT or diabetes. A second study is in relatively long standing diabetics while the third study only required two characteristics of MetSyn. The largest study was in a fairly specific demographic of otherwise healthy postmenopausal women. We're not even talking apples and oranges here, we're more in pineapples and coconuts terroir.
Medications? See the table, we've got the full gamut here. It is important to note that when you have small numbers, and multiple medications, it is impossible to do any meaningful post hoc analysis to control for these variables.

OK can we please stop now? The results of dietary trials are horribly confounded by medication use, period. When you are comparing small groups, and medications are continued throughout a study, then changes in diet will affect some medicated individuals more than others. Most of the time we're talking the occasional statistical significance that barely rises to the level of clinical significance. A single "responder" in the group can skew all of the results. A more extreme dietary intervention, especially short term, is likely to influence medications. It's too interrelated to be able to attribute changes to diet alone. At best, studies in medicated populations relate to those populations being managed for chronic disease.

Nina Teicholz bashes the USDA Guidelines in The Big Fat Surprise on this whole point in particular. That being that in her view, they are recommending the diet for middle aged men with heart disease for all. Well, even if these studies coalesce to support some diet, at best it is for middle aged people.

OK ... Onward to the diets themselves! Let's talk calories first.

The Lindeberg and Jönsson studies were both done in mostly men (24 men and 3 women consumed paleo between these two studies). The calorie differences between the groups are enough to account for any of the differences in results. In Lindeberg, 1344 calories is exceedingly low for men, and roughly 450 calories less than controls. In Jönsson, 1581 is still quite low for majority male subjects, and roughly 300 calories less than controls.

I am always amused at those who consider anything under 2000 calories to be a starvation diet, while referring to the second paper (referencing the same clinical trial) by Lindeberg's group regarding satiety per calorie. This is really not a generalizable study on just about every level. It was conducted in a very narrowly defined population and involved calories that are far too low for most men and a sizable proportion of women. Couple that with significantly higher intake on the control diet, and again, we can toss this study in the "higher protein causes spontaneous intake reduction" pile. Interesting, but nothing paleo-baybeee about that.

Boers seems equally matched on calories and was intended to be weight stable. Paleo lost a bit of weight (about 3 lbs vs. control), but are we really going to "go there" with nominal *weight* losses over 2 weeks?

So that leaves Mellberg (here again is a link to my primary blog post on this). I understand that they used the 6 month data so as to create a "short term study" from this long term study, but in a way, you have to ignore the fact that any significant improvements in paleo vs. control, became non-significant by 2 years. But since we're talking calories for now,

Calories and deficits were ostensibly matched in Mellberg.

How about adherence? Here is where I'm going to dwell on Mellberg for a moment, and this will drift into macronutrients. Participants kept logs to report intake at the different time points. Compliance in this study was awful.

Note the middle column specifies the target macro ratios. While paleos upped protein, they basically made it halfway to target. They hit the carb target and overshot the fat. Meanwhile the control group was to reduce protein percentage slightly and they actually went in the wrong direction. They fell way short of reductions in fat, but perhaps most significantly, they slightly reduced carbs as a percentage rather than increasing them significantly. As the meta-analysis goes, the actual intake ratios are consistent with other "recommended diet" controls, but this was not a fair shake test of the Nordic Nutrition Recommendations.

Macros

These are the diets sorted in descending number of calories consumed. Are there trends? Sure. Can you definitively pick out a paleo ratio vs. a "recommended" ratio? No. Let's take a look at that PROSPERO filing. It's rather extensive as to what databases would be searched, studies included, etc. How about the control diets?

We will not require that the diets compared in the trials have the same caloric intake because if one of the diets results in a lower or higher reported caloric intake, it may be because that diet improves metabolic parameters or hormones, which may in turn regulate satiety. For example, if there were a trial that randomized participants to ad libitum consumption of either the paleo diet or the Mediterranean diet, and the paleo diet group ended up consuming fewer calories, we would not exclude that trial from our review because the reduced caloric consumption in the paleo group may be due to the positive effects of the paleo diet on regulating levels of leptin, the satiety hormone (Jonsson 2010 Nutrition and Metabolism). Additionally, more direct effects of macronutrients on satiety that limit calorie intake may be involved, which would improve metabolism in their own right.

Ummm ... I'm going to leave this out there except to say that there's an awful lot of editorializing going on there and belies a "pre-familiarity" with the studies they hoped to, or intended to, include in their analysis. While there are clear trends in paleo vs. controls, when we're talking four studies, the lack of consistency is yet another strike against this statistical hyper-data-massage.

The fourth paleo diet could be swapped with the second control diet for two groups of more similar dietary composition.

Did I Mention that Lead Author Manheimer Wrote An Article for Cordain?

Kind of a no no in my book. Some excerpts, and I'll leave Manheimer's links to content on Cordain's site intact. Clearly this is not a dispassionate person here.

The Paleo diet is the hottest diet in the world. It was ranked as the top diet related term searched in Google for 2013 and 2014. Celebrities are on it. Politicians are on it. It’s on the news. It’s what people are talking about.

People have been talking about Atkins, and South Beach was uber popular too for a while. Paleo is fading so fast, folks are ditching the label left and right. Cordain, I suppose to his intransigent credit, remains one of the few pure paleo advocates out there.

It’s also controversial. Paleo advocates selecting foods based on evolutionary theories and principles. This doesn’t make sense to everyone. Some have argued that it’s impossible to know what our ancestors from the Paleolithic era ate.

Yes, well, there is lots of argument. Let's begin with the idea that paleo advocates base their diet on evolutionary theories and principles. This is a hot steaming pile of bovine feces. If it were true, they wouldn't ignore evidence in support of considerable pre-agricultural consumption of grains, legumes and potatoes -- lectins, phytates and saponins be damned!

Others have argued that there is no clinical evidence to support Paleo.

Loren Cordain, with the aid of Robb Wolf to repeat it to a larger, hipper audience, has been summarizing all the clinical trials on the paleo diet every time the US News & World Report comes out. (I again re-link you to my 2014 response to the most recent version of this game). Everyone knows there is SOME limited clinical evidence, none of which validates calling such a dietary prescription as some evolutionary prescription.

A systematic review and meta-analysis of RCTs can generate robust evidence, particularly when the trials are well-conducted and the outcomes are hard, objective outcomes. Although there have only been four RCTs of the Paleo Diet, they were all well-conducted.

I'd challenge that assessment. More importantly, a bunch of studies can all be meticulously conducted and unless there's some reasonable similarity between the studies, a meta-analysis is useless.

The outcomes we examined are the components of the metabolic syndrome, such as blood pressure, glucose, cholesterol, waist circumference, and triglycerides. These are the major predictors of chronic disease risk, particularly cardiovascular disease and diabetes. Our pooled analysis showed that the Paleo diet resulted in greater improvements in these outcomes compared to a standard guideline-based diet. The pooled results were not always statistically significant, but they favored the Paleo diet in all cases.

Emphasis mine. I'm willing to bet a lot more weren't stat sig than were, and where statistically significant, likely meaningless in clinical context. You're pooling six month outcomes in generally healthy postmenopausal women, on a around 1650 calories of P/C/F=23/29/44 with three month outcomes in 65 year old men with ischemic heart disease and diabetes or impaired glucose tolerance. on roughly 1350 calories of P/F/C=28/40/27. Sorry, that doesn't fly.

our meta-analyses compared Paleo to diets similar to DASH and Paleo came out on top.

This borders dangerously on a lie.

Are dietary guidelines around the world recommending the wrong diet? It’s too soon to tell, but our meta-analyses warrant further evaluation of the health benefits of Paleolithic nutrition.

Loren Cordain recently wrote about milestones of research into the Paleo diet. No doubt when Loren updates his summary, he’ll mention our systematic review.

Ahem.

Some Modest Suggestions:

Define the paleo diet. If it's Cordain's current version, you cannot use any of the available clinical trials. This is because despite being bullied into going back on his previous steadfast support for rapeseed/canola oil, it is used in "paleo diet" trials.
Acknowledge that the "wildly popular" paleo diet is NOT what has been studied in clinical settings.

This will never happen. But if it were to actually occur, one would then need to test this diet against other real-whole-foods-based diets. Oh ... like a true implementation of the dietary guidelines, or how about a control diet where 65% carbohydrate from whole grains, legumes, fruit and tubers is actually achieved. It is not honest to continue to do otherwise.

This meta-analysis or systemic review is way, wAY, WAY premature. The fact that it's being touted on Cordain's site by the primary author doesn't sit well.

Search This Blog

The Carb-Sane Asylum