SysteMeta-Antics in the Paleo Systematic Review & Meta-Analysis

This was intended to be a targeted post on one specific issue pertaining to the recent AJCN study and the actions of some of the study authors involved.   Not sure I succeeded ;-)

When I wrote my first post on the topic, I had not read the full text of the study.  Instead I compiled a table of characteristics for the four individual studies involved, and concluded that no meta-analysis was appropriate for such disparate studies.  That initial "analysis" holds despite the table containing a few errors.

After being called "unscientific" by author Esther van Zuuren on Twitter, I obtained a copy of the full text and read it.  Additional information contained in the full text makes matters worse, not better, on virtually every point I made in my original post.  So I am working on a letter to the AJCN and in doing so am working on an expanded and more accurate version of the summary table.  During this process I sought to check values published in the original studies against those used in the Manheimer analysis.  The original focus was on the statistical significance and variation.  But along the way I found myself unable to reconcile some values and the reason is because they didn't all come from the original publications.  This important piece of information is not mentioned anywhere in the full text of Manheimer.  As I said.  Worse, not better.

What follows are the mentions in the Supplementary Materials in the full text of the article:
  • 2Supplemental Tables 1–3 are available from the “Supplemental data” link in the online posting of the article and from the same link in the online table of contents at
  • Full details of the Paleolithic nutritional patterns and control diets used in the 4 RCTs are included in Supplemental Table 1.
  • Detailed explanations for our judgments for each risk-of-bias domain are included as Supplemental Table 2.
  • There are 2 ongoing RCTs, .... (Supplemental Table 3).
Here is the sole mention in the text-proper of contact with study authors
  • We contacted the principal investigators of all of the trials with specific questions related to the design and outcomes of their trials.
[see note at end of post]

This is insufficient, IMO, to inform the reader that the meta-analysis included unpublished data obtained from at least one investigator.   Again, I didn't really see any reason to look at the Supplementary Material based on references to it in the full text.  Then, I came across this Twitter exchange with two of the authors (@Ezzoef = Esther van Zuuren and @zbysfedo = Zbys Fedorowicz) that also tagged the lead author Eric Manheimer.   

I wondered if some of the confusion could be cleared up by the supplementary materials, and I believe the Great Deleter somewhat misspoke here.  There are several "devils" in the details, but these only detract further from the meta-analysis on the whole, and bring the agendas and indeed the integrities of these reviewers into question.  This is my opinion anyway, your mileage may vary.  The details far exceed what one might expect to find *hidden in plain view* in a bulky supplement based on indications in the full text.

Before going on, here are the links to the materials I refer to here: Full text Manheimer , Supplementary Materials .  You simply need to click the links and request access as is my preferred means of sharing. 

Metabolic Syndrome Redefined?

This post concerns a meta-analysis with the following title:  Paleolithic nutrition for metabolic syndrome: systematic review and meta-analysis

Metabolic Syndrome, or MetSyn for short, is a specific designation, and these specifics are outlined in both the PROSPERO registration
The metabolic syndrome is a cluster of risk factors ... The cluster, as defined by the National Cholesterol Education Program’s Adult Treatment Panel III (2002), includes plasma glucose, triglycerides and HDL cholesterol concentrations in fasting condition, blood pressure and waist circumference. If at least 3 of these parameters are in the pathological range, an individual is diagnosed with the metabolic syndrome.
And the full text:
The metabolic syndrome is a cluster of 5 risk factors, including waist circumference, blood pressure, and serum concentrations of glucose, triglycerides, and HDL cholesterol in the fasting condition, which often occur in concert and predispose people to type 2 diabetes and cardiovascular disease. If ≥3 of these factors are in the pathological range, an individual is diagnosed with the metabolic syndrome
It is generally understood that "sicker" individuals, e.g. those who have progressed to frank diabetes, be identified as such, not by their lesser cluster of symptoms.  And yet according to PROSPERO:
Participants/ population

Inclusion criteria: men and women; one or more of the components of the metabolic syndrome as defined by the National Cholesterol Education Program (Adult Treatment Panel III).
Exclusion criteria: any chronic (progressive) disease requiring hospital care; eating disorder; special other diet.
and the full text:
We included RCTs that compared Paleolithic nutrition vs. any other dietary intervention in participants with one or more of the 5 components of the metabolic syndrome (see Outcomes) but without a chronic (progressive) disease (11).
[Note green = inclusion, red = exclusion.]   Manheimer have "dumbed down" MetSyn by saying they will include studies on subjects possessing only one or more of the five factors.  In other words, as was the case of their largest study (Mellberg), they are including subjects who would not be recognized as having MetSyn.  This is perhaps acceptable on a single study basis, but for a meta-analysis?  More to the topic of this post, there's a little fuzzy wording there in the sicker folks.  Diabetes is considered a chronic (progressive) disease, but doesn't necessarily require hospital care.    IHD is clearly such a disease and several subjects were recently hospitalized for this disease.  And yet, the Lindeberg study was included in Manheimer.   

It gets worse.

Creating Significance from the Insignificant and Incomplete

For the 5 components of MetSyn I have summarized the outcomes for difference between groups in the published full texts of the original studies.   It seems rather the more absurd that we're even talking about a MA given these "holes" but there you have it.   The asterix on Jonnson is due to the fact that the MA only used the "first leg" data (apparently obtained from Jonnson) and thus values reported for paleo vs. control are different than those published for the full crossover study.  I may or may not address this at a future date.  For now I would point you to the supplemental materials and the lack of ANY data presented indicating statistical significance of the 7 "paleo first" vs. 6  "control first" subjects in Jonnson.

Lindeberg Revisited:

It deserves including some text content from the original study here, because this illustrates the danger of meta-analyses in obscuring the details of individual studies by reducing matters to outcomes and numbers.
The study was a 12 week controlled dietary intervention trial in 29 (out of 38 eligible) male IHD [ischemic heart disease] patients with waist circumference >94 cm and increased blood glucose or known diabetes, recruited from the Coronary Care Unit at Lund University Hospital, Sweden. We included patients with any of the following conditions: an ongoing acute coronary syndrome, a history of myocardial infarction diagnosed by creatine kinase MB isozyme or troponin elevation, percutaneous coronary intervention or coronary artery bypass surgery or angiographically diagnosed coronary stenosis ≥30%. ...
These are not well men. CLEARLY this study does not meet the criteria as stated by Manheimer Not only are they not being treated by diet alone, there are two very important stipulations made up front regarding pharmaceutical interventions:
[Medications excluded were] hypoglycaemic agents, warfarin or oral steroids. Other drugs were not restricted and treatment with statins and beta blockers was usually initiated and/or changed during the trial.
Here are the baseline characteristics, note that almost none of the men were on statins at outset, and many were started on the dietary treatment within days of an acute coronary "event":

ORIGINAL DATA:  I have spliced together the results tables for the 5 Components of MetSyn at baseline and 6 weeks (time point used in MA):
No ... that's not an error.  I didn't accidentally delete important sections of the tables. This is the reality of Lindeberg as relates to components of MetSyn.  They were dealing with men that were considerably further down the metabolically deranged path so as to *require* immediate pharmaceutical intervention.  Thus, only the differences in waist circumference and fasting blood glucose were reported.  Note that statistical significance for 0-6 week change in glucose between groups was not reported.   As to triglycerides and HDL:
"Serum lipids changed to a similar extent in both groups, due to initiation of statin treatment in most patients ..."
And blood pressure:
" ... there was no decrease in blood pressure"
Followed by "data not shown".

Here is where the Supplementary Materials become interesting.  There is no indication in the text of the MA that data used in their analysis included previously unpublished data provided by the study authors.  This should at least be included in footnotes of the data tables provided.  

SUPPLEMENTARY DATA:  The information I'm going to discuss now begins on page 18 or 44 in the Supplementary Materials.   It becomes apparent that there were two email exchanges with Prof. Lindeberg, one in January of 2015, and one in June 2015 regarding change from baseline data that was missing in the published study.  Indeed the only measure for which complete data (mean change from baseline with standard deviation) was available in the original publication was waist circumference.  So let's fill in the blanks:

1.  Triglycerides:  
  • Email response from Lindeberg Jan. 2, 2015:   "Unfortunately, serum lipids were not analysed. (The majority of patients were statin-naive and all of them were prescribed a statin.)"
  • Email response from Lindeberg June 27, 2015:   "With regard to serum lipids, I still believe they are difficult to analyse with regard to initiated statin treatment, but here are the changes from baseline to 6 weeks (we don't have HDL):..." 
  • Provided Mean change from baseline (SD):  for Paleo = -0.36 (0.66) , and  Control = -0.45 (0.69) mg/dL
  • No p-value on difference between groups is provided, but reported as "similar" in original text.
2.  HDL:   "we don't have HDL"

3.  Systolic Blood Pressure:  
  • Email response from Lindeberg June 27, 2015:  "A problem with blood pressure is that beta blockers were routinely prescribed to those with a myocardial infarction, irrespective of their blood pressure.  Another problem is that blood pressure typically decreases during an acute coronary event (baseline values were 131/78±16/10 mm).  Unfortunately, we have not analysed this in detail, so I don't know if there was imbalance between the diet groups in these aspects."
  • Provided Mean change from baseline (SD):  for Paleo = -0.04 (8.95) , and  Control =  +0.96 (8.88) mmHg.
  • No p-value on difference between groups is provided, but default back to original text where "no decrease" is reported and difference between groups is thus non-significant.

4.  Fasting Blood Sugar:  

  • Added missing Mean change from baseline (SD):   for Paleo = -1.60 (1.99) , and  Control = -1.30 (2.35) mg/dL
  • It is noted that the SD for above are "reverse calculated" using p=0.01 for Paleo and 0.05 for Control
  • Even if SDs are over estimated, the changes in means are closer at 6 weeks than at 12 weeks thus p-value for difference between them is almost certainly higher than the 0.2 at 12 weeks.  Not Significant.
I have highlighted in red, the two admonitions by Staffan Lindeberg himself as regards the interpretation of data from this study.  You cannot attribute changes in lipids and blood pressure to a dietary intervention when any changes are more likely to be due to specific medications.

Medication & Health Status -- Not Diet

    The bottom line here is that if the usual systematic review culling process were used to identify trials to include in a study entitled:  Paleolithic nutrition for metabolic syndrome, Lindeberg would not even make the cut.  
    • Subjects all had ischemic heart disease, almost half of whom had "acute coronary syndrome" = heart attack or angioplasty within one week of starting the study.  (Reminder:  6 week progress data was used for meta-analysis) 
    • Of the 29 subjects, 8 had diabetes assessed by fasting capillary glucose levels and 19 had diabetes assessed by OGTT.  Diabetes is generally considered to be a more advanced level of disease progression.
    • Of the 5 components of MetSyn:
      • One was not measured at all:  HDL
      • Two were measured but not reported in the original study due to concerns that the results were complicated by recent initiation and/or changes in medications that directly alter those components: Triglycerides and Systolic BP.
      • One was reported incompletely at the 6 week time point, but almost certainly not statistically significant anyway when supplementary data was used:  ∆FPG
    • Of the 5 components of MetSyn, only ONE was different between groups -- waist circumference.  
    Over the express written concerns of the principal investigator, Manheimer included Lindeberg study data for 2 of the 5 components of MetSyn in their meta analysis, thereby attributing these results to the dietary intervention.  

    This was done:
    • Without any mention of the potential role medications may have played on the results of any of the individual studies and thus the meta-analysis overall, and
    • Without any mention of Lindeberg's specific concerns about the role of medications (and health status) in the results of his study.
    The result is now that previously unpublished data, that was not statistically significant anyway, has contributed to a meta-analysis concluding that the paleo diet offers superior improvement for MetSyn characteristics compared with recommended diets.  Meanwhile, information about medications that was reported in the original publication was omitted from the meta-analysis entirely -- an omission that cannot be passed off as an innocent oversight given as passages concerning these issues in the original text were confirmed by the principal investigator, as outlined in supplementary materials.

    [Study] Conclusions: The Paleolithic diet resulted in greater short-term improvements on metabolic syndrome components than did guideline-based control diets.

    This qualifies as scientific malfeasance in my opinion.

    *  Here is the only reference to obtaining data from study authors.  It comes in the form of a general acknowledgment at the end of the study full text:  "We thank Inge Boers, Tommy Jonsson, Staffan Lindeberg, and Caroline Mellberg, who are corresponding authors of included RCTs, for confirming and providing data related to their RCTs."

    We thank Loren Cordain for providing references and resources.

    Things that make you go hmmmm....