The $12M NuSI/Ludwig Study: Part I: Critique of the Study Design
SUMMARY
The results of the $12 Million Dollar NuSI-sponsored study, headed up by Dr. David S. Ludwig, are finally out.
Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial
While the good doctor is making the rounds touting them as evidence in support of the Carb-Insulin Hypothesis (TWICHOO in these parts), a review of the raw data made available to the public casts grave doubts on his victory lap. This study built upon the "promising" results of the 2012 study:
Effects of Dietary Composition on Energy Expenditure During Weight-Loss Maintenance
This post focuses on comparisons of Study Design between these two studies, some improvements, and the ultimate failure that renders the primary outcome data suspect, if not outright useless. (And I don't say that lightly!)
Improvements:
- Study size
- Length of time on one test diet (20 weeks) vs. 4 weeks crossover on each diet w/o washout
- Protein held constant between test diets
- Macro extremes the same -- e.g. LCHF = 20% carb/60% fat , HCLF = 60% carb / 20% fat
- Post weight loss (PWL) assessment of TEE
- Intake adjusted to maintain weight
- Drastically altered run-in phase
- Shorter in total: 16 weeks vs. 20 weeks
- No monitoring period before gathering baseline data.
- No baseline intake assessment
- Shorter weight loss phase: 10 weeks vs. 12 weeks
- More varied weight loss and no minimum loss established to be included in the test phase. In previous study, all subjects had to achieve at least 10% loss, averaged 13.5%. Current study, losses ranged from 5.6% to 16.0%
- Only 2 weeks for weight stabilization vs. 4 weeks.
- Added ad libitum snacks for those who needed to increase caloric intake in maintenance but could not tolerate the larger meals necessary.
- Assessing post weight loss DLW-TEE during the same two weeks immediately following weight loss that are designated for stabilizing reduced weight.
- Changing protocol to anchor TEE changes to this flawed PWL measure of TEE instead of baseline per original protocol.
Hello blog! My dear readers! The drought might be over here at the Asylum. I hope anyway :-)
I will likely be back at some point to edit this out and/or flesh in some links to background, etc., but I don't wish to waste time on "book keeping" at this time.
A recent study published in BMJ, is making the rounds with copious cheerleading by lead investigator Dr. David S. Ludwig. This study, dubbed Framingham State Food Study or FS², was the largest of the original three studies funded through (then) Gary Taubes' and Peter Attia's Nutrition Science Initiative, NuSI.
The Study:
Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial
Make note there are two supplements and a link to the data that I'll provide here: https://osf.io/rvbuy/ . I downloaded that data and have been conducting various analyses that I will be sharing in future posts.
This study was touted as that "good science" the nutrition world has been missing for decades, and one that would add meaningful data to what causes obesity. It is modeled on a study by Ludwig's research group published in 2012: Effects of Dietary Composition on Energy Expenditure During Weight-Loss Maintenance
While Ludwig and co-lead investigator Cara Ebbeling PhD made some significant changes to the protocol for the "expanded repeat" of the 2012 study, I want to kick my analysis of the latest study off with a critique of the study design.
Several other scientists and journalists can't help but fall over themselves in praise of this study. "Amazing!" ... "Tour de force of nutrition science" ... "a unique study because of its size and rigor" ... I'm going to call it like I see it: NONSENSE.
Can the scientists and journalists please stop slobbering over this long enough to recognize that at the very least, Ebbeling et.al. 2012 was a test run that gave these researchers much insight into how to improve their methodology. There's absolutely nothing ground breaking or innovative in adapting a crossover design to a group comparison study. NOTHING. There's nothing "really complex" about the design. For all its "meticulous" nature in prepping foods and weighing to within 0.1 or 0.5 grams (depending on portion size) ... other parts of the design are anything but meticulous in ensuring some critically important things regarding the primary outcome: Total Energy Expenditure by DLW.
Let's cut to the chase with this graphic comparing the intervention time frames:
I aligned these as best as I could, the 2012 study on top, the current study on the bottom. Let us compare and contrast (you will need to refer to supplements and original articles for details).
Important Note: The DLW-TEE is a "free living" test that is conducted over a period of 14 days = 2 weeks.
Subjects
2012:: 21 subjects completed, 8 F = 38%, Start ~105 kg, End ~90 kg.
2018:: 109* subjects completed, 76 F = 70%, Start ~90 kg, End ~80 kg.
*I accessed available data and include only those for whom complete energy assessments were available, and deemed weight compliant (within +/- 2kg start of test phase start weight): 41 LC, 37 MC and 31 HC.
Run-In Phase
Total Length:
2012: 20 weeks 2018: 14 weeks
Pre-Weight Loss - Baseline
2012: Monitored for weight stability of "usual diet" etc. for four weeks. Gathered data for TEE by doubly labeled water method (DLW), intake, and REE during the last two weeks of monitoring. These were baseline values used for comparison.
2018: Two weeks DLW-TEE and REE, essentially "off the street", NO assessment or reporting of baseline intake.
Weight Loss Phase: Diet 45% Carb / 30% fat / 25% Protein
2012: Used Mifflin-St. Jeor equation to estimate REE, multiplied by individual activity factors based on a questionnaire to estimate baseline energy requirements. Subjects were restricted to 60% of energy requirements for 12 weeks to target a ~12% weight loss.
"When necessary, we made adjustments in energy intake to achieve weight loss and stabilization during the run-in phase. We allowed the duration of the run-in phase to vary among participants, to account for individual differences in the rate of weight loss"
Weight loss: Minimum of 10% weight loss for all subjects. Average loss of 13.5% initial body weight. Average starting weight ~105 kg, average post weight loss weight ~90 kg.
2018: Used Mifflin-St Jeor and a fixed activity factor of 1.5 to estimate energy requirements. Subjects restricted to 60% energy requirements for fixed 10 week period to target ~12% weight loss**. There is some indication of altering intake for those not losing weight, but no mention of individualized time or targeting a minimum percentage.
Weight loss: Ranges from 5.6% to 16%, mean weight loss ~10% ... this is almost a 3-fold difference in the "baseline" for the test period. Absolute weight loss from average ~90 kg to post ~80 kg.
Reduced Weight Stabilization: (diet same as weight loss)
Both Studies: "Energy needs for the weight stabilization period at the end of the run-in phase were estimated based on recent rate of weight loss during the energy-restricted diet: energy intake during weight loss (kcal/day) + [rate of weight loss (kg/day) × 7700 kcal/kg]"
2012: Four weeks total. Intake ramped up by one-quarter of the additional calories for four days. No DLW-TEE or REE data collected during this phase.
2018: Two weeks with no indication of ramping up, during which DLW-TEE and REE were assessed.
Critique of Run-In Phase:
I believe it is critical to stress that the entire reason for a run-in phase in any study is to ensure as much consistency as possible for factors that might influence the results of the test phase. In this case, since both studies were looking at effect on TEE in weight-loss maintenance for different diet compositions, the following factors were important:
- Composition of weight loss diet
- Rate and duration of weight loss
- Amount of weight loss
- Stabilization of reduced weight
As such, except perhaps for some (presumably minor) differences in the rate and duration of weight loss, the run-in for the 2012 study is pretty solid. There was also a period of monitoring before gathering pre-study data used as a baseline anchor. It is fair to say that not only did the researchers fail to improve on this study design, the significantly shortened run-in phase of the current 2018 study represents a giant step in the wrong direction, and is problematic to say the least. I would go so far as to suggest that it is so problematic we might well be dealing with GIGO: Garbage In - Garbage Out.
To itemize:
To itemize:
- Pre-weight loss baseline data, including the two-week DLW-TEE were obtained for subjects "off the street" with no period of monitoring for weight stability.
- Weight loss goal of 12% was not changed (that is the figure in all of the literature and being cited in articles) but the weight loss portion was reduced from 12 to 10 weeks.
- There was no minimum weight loss target, so while the average loss was predictably about 10%, the range from 5.6% to 16.0% represents an almost 3-fold difference.
- The weight stabilization period was cut in half to only two weeks.
Leading up to this study, there was considerable focus on metabolic adaptation to weight loss. As such, the combination of the last two bullet points represent "fatal flaws" in the Study Methods. They should have:
- "Standardized" metabolic adaptation by targeting a fixed percent weight loss (or narrow range), or at the very least a minimum! Then adjusted intake to achieve in similar time frames.
- Extended, not shortened, the stabilization period to ensure weight stability and minimize short term adaptations (shown to reverse in at least some studies).
All of this said, probably worst of all is that the issues with the accuracy of DLW-TEE when weight is not stable are well known. As such:
The 14-day test, that became the "anchor" for changes in energy expenditure, should NEVER have been done during the same 14 days used to stabilize weight immediately following weight loss.
Test Phase
Total Length:
2012: 12 weeks , 4 weeks on each of 3 diets , random crossover , no washout
2018: 20 weeks, 3 groups, 1 on each test diet , midpoint assessment
Test Diets
2012:
- Very Low Carb: 10% Carb / 60% Fat / 30% Protein
- Low Glycemic Index: 40% Carb / 40% Fat / 20% Protein
- Low Fat: 60% Carb / 20% Fat / 20% Protein
Caloric level for all three diets established as intake from the weight stabilization phase of run-in, regardless of weight fluctuations.
2018:
- Low Carb: 20% Carb / 60% Fat / 20% Protein
- Moderate Carb: 40% Carb / 40% Fat / 20% Protein
- Low Fat: 60% Carb / 20% Fat / 20% Protein
Caloric level started at weight stabilization phase of run-in and adjusted periodically to ensure weight stability to within ±2 kg of reduced stable weight.
"We provided extra ad libitum snacks, reflecting the macronutrient composition of respective Test diets, to participants who continued to lose weight and had difficulty consuming large meals."
Energy Assessments
2012: DLW-TEE and REE for last two weeks of each 4-week test diet phase. Intake fixed, reported to average 2626 kcal/day.
Comparisons made to pre-weight loss baseline measures.
2018: DLW-TEE, REE and intake assessed Weeks 9&10 for midpoint, and Weeks 19&20 for endpoint.
Comparisons made to post-weight loss, two week weight stabilizing measures.
Critique of Test Phase:
Let me begin with some praise for the researchers in correcting some issues from the 2012 study, most prominently the compositions of the test diets. In 2012, protein -- with its known increased thermic effect -- varied among the diets, in the current study, protein was fixed at 20% across all test diets. Thus they were directly testing carb vs. fat including the extremes that directly flipped contributions of each from 60% for the high macro, 20% for the low macro, and the middle split them 40%-40%. Well done!
My major issue with the diets in the current study is the ad libitum snacks they provided to subjects who were losing weight because they couldn't eat larger meals. WHY did these have to be ad libitum??? It seems to me that they could have made some macro-friendly granola bars, muffins and/or shakes in pre-measured 100 cal increments where subjects were instructed to consume one a day, two, etc. until weight stabilized. For all the meticulous accounting of macros and calories in the dietary delivery system in this study, why blow it with this??
Clearly a 20 week test phase on a consistent diet is a huge improvement of the prior 4 weeks of vastly varied diets.
My major methodology issue is with a carryover of a problem from Run-In. The study design changed several times, and most importantly instead of using the "off the street" baseline TEE as anchor, they changed it to the "weight stabilizing" TEE. It's repetitive, but to use this highly problematic measure as baseline anchor for the test phase is a fatal flaw.
It is my understanding that (my favorite "young biophysicist") Kevin Hall PhD will be diving deep into this aspect of the results of the study. I'll be sharing that as it becomes available. For myself, I'm focusing more on the discrepancies (often rather large, and in opposing direction) between the reported intake and DLW-TEE -- values that SHOULD be relatively similar in a study such as this. I've embedded the individual energy plots below (sorry for the resolution) that I shared as a teaser on Twitter. You can make out the trends in the REE, TEE and Intake. I think it is most telling that REE essentially flat-lines across the board, while TEE and Intake track well rather more rarely than frequently. I will put a lot of numbers to this in a future post (or more). Stay tuned!
It is my understanding that (my favorite "young biophysicist") Kevin Hall PhD will be diving deep into this aspect of the results of the study. I'll be sharing that as it becomes available. For myself, I'm focusing more on the discrepancies (often rather large, and in opposing direction) between the reported intake and DLW-TEE -- values that SHOULD be relatively similar in a study such as this. I've embedded the individual energy plots below (sorry for the resolution) that I shared as a teaser on Twitter. You can make out the trends in the REE, TEE and Intake. I think it is most telling that REE essentially flat-lines across the board, while TEE and Intake track well rather more rarely than frequently. I will put a lot of numbers to this in a future post (or more). Stay tuned!
 |
Plots of individual energy values for all weight compliant subjects with complete data. 1. PWL (post weight loss/Week ), 2. MID (10 weeks), and 3. END (20 weeks) Blue = REE , Green = TEE , Pink = INTAKE Low carb, 41 subjects, Yellow Background / Top Moderate carb, 37 subjects, Blue Background / Middle High carb, 31 subjects, Green Background / Bottom |
On Deck:
I'm unsure if I'll address the gross inconsistencies between intake and TEE (that the above thumbnail plots show) first, or address some general interpretations of the results first. Both are in progress, I'll link here when published.
https://carbsanity.blogspot.com/2018/12/the-12m-nusi-ludwig-study-part-ii-12.html
Comments
Post a Comment
Comment Moderation is ON ... I will NOT be routinely reviewing or publishing comments at this time..