A Friday JAMA-lama Ding Dong!
Well ... I wasn't really going to weigh in further past using that recent JAMA article (EDIT: full text no longer free at JAMA, see link below) as an example of where statistics can lead us astray. But ... as is probably expected, this study has kicked up some dust in the community. I'm sure I'll miss a few, but here's the weighing in so far:
That's just the blog posts, not the tweets, FB postings or other social media buzz ... and I'm happy to edit in more, just drop the hint in comments. And here's the full text of the study: Effects of Dietary Composition on Energy Expenditure During Weight-Loss Maintenance
The more I look at this study, the more angry I get about the piss poor state of affairs that is study design and implementation these days. While I'm keenly aware that it is impossible to control and account for everything in such studies, that's no excuse for not even paying lip service to those that should be paramount to the integrity of your data. There's a saying in statistics that no amount of statistical manipulation can fix bad data. More succintly stated: Garbage In = Garbage Out. This study, after closer examination to make sure my first impressions were accurate, is GIGO, and GIGO cannot be taken seriously in any discussion of CICO.
Why do I make such a bold GIGO claim? Well, because the data presented has some glaring holes. Missing information on no-brainer obvious stuff. Why? And not only that, there is no mention of measures taken to procure the data, e.g. did they even bother to collect it? While there are certain aspects of any free-living study that can't ever be controlled completely, there are ways to minimize the influence of these factors on outcomes, and well-designed studies generally outline the specifics of measures they took to minimize this.
Not so, Ebbeling, et.al. Here's what they did, and if I'm in error in anything I state here it is only because it is not more clearly laid out in their peer reviewed account. They took what ended up being 21 obese individuals, determined weight stable usual intake for 4 weeks, then put them on a reducing diet. The reducing diet was by % P/C/F 25/45/30. FWIW, this would qualify as "low fat" and "high protein" compared to the SAD, and perhaps higher in protein but lower in carb than your standard CRD. After 12 weeks on this diet, subjects lost an average of 14.3 kg and were then weight stabilized for 4 weeks on this diet. It would appear that these subjects essentially lost 13.6% body weight and remained obese in maintenance with intake at around 2600 cal/day. This was determined in a 4 week post-diet stabilization period. REE and TDEE were assessed during free living periods at the end of the initial 4 week (pre-diet) weight stable phase and during the last 1 or 2 weeks of each 4 week "maintenance" test diet.
- MISSING: What was the REE/TDEE of these subjects in the last two weeks of the weight-reduced stabilization period??
In a cross-over design, they were then assigned to one of three isocaloric diets -- LF, LGI, and LC -- presumably calibrated to maintain current weight. The Methods would indicate that "controlled feeding" here amounted to providing participants with menus to consume from.
- MISSING: Food logs, dietary recall, etc. ... Basically ANY sort of indication as to compliance with either of the three regimes. Worse yet ...
- MISSING: Any indication that the researchers made any effort to assess or insure compliance. Therefore ...
- MISSING: Actual intake composition data. Recent studies have shown just how important this is in making comparisons!
Though we get a litany of data points for various metabolic parameters at those time points where they were reported,
- MISSING: Body weight at each time point.
- EDIT: FOUND -- Apologies, would have been nice to have this in a table somewhere, but average weight was reported for each time period and they remained weight stable, on average (~91-91.5 kg). Still, this compounds the mystery of why there wasn't weight loss on VLC with that increased expenditure? Also still not getting why they dont report standard deviation vs. CI (confidence interval). I'll have to address the CI sometime, odd ... just odd ... to present that for results (SD is an exact calculated value for a sample, CI's are estimates projected for a population based on samples and assumptions of probabilities, etc.).
This is important, because in most weight stability studies, weight is monitored closely and intake adjusted accordingly. And we should be suspicious here because ...
- MISSING: The usual dropouts over a 3 month period for 1 month each spent on radically different diets. (The final n=21 resulted from only 1 dropout during this portion of the study, although the "rotten apples may have been weeded out in the weight loss portion). Still ...
- MISSING: Evidence of maintenance!
- EDIT: Found, as stated above, that mean weight remained relatively constant as assessed at the end of each 4 week period. There does not appear to be any discussion of monitoring stability.
So we have no clue how closely any of these participants adhered to the prescribed diets or whether they maintained, lost or even gained weight during that leg of the study. GARBAGE IN.
Let's assume, however, that all 21, were perfect subjects, dutifully following menus and weighing daily and self-adjusting intake to remain weight stable for three months. That is a huh-YOOGE assumption, but let's make it anyway. You know what else is missing? Standard Deviation of the Mean values reported for REE and TEE (I reserve the right to focus my argument on these outcomes that are the major buzz generator). Almost all reports of mean data include a +/- of SD, SE or give that value in parentheses. Not so here.
- MISSING: Transparent reporting of statistical error/variability.
What is given in the table is the range of values, which can be highly misleading due to outliers, and elsewhere we get the 95% CI ranges, all Mean [95% CI] kcal/day -- LF: –205 [–265 to –144] ; LGI: –166 [–227 to –106] ; VLC: −138 [–198 to –77] . From this, and the blue squares on the plots, it would appear that the margin of error was ~60 kcal/day. For the 95% CI, margin of error, E = 1.96*SD/sqrt(n) ... working backwards then SD = E*sqrt(n)/1.96 = 60*sqrt(20)/1.96 = 137 kcal/day. This would only get worse if statistical laws were obeyed for minimum sample size and appropriate distribution used, etc. Harummph. In other words, I don't know what kind of statistical magical massage parlor they put this data through, but they need to clean under the baseboards more often cuz it's starting to smell.
I'm going to pull a Taubes here. Let's ignore the "Mediterranean style" diet and just put LC vs LF up head to head. Our null hypothesis would state that changes in REE were equal (e.g. the means were the same), and if the calculated probability of this being true was very low (usually <5%, but sometimes other benchmarks are used), we would reject this null hypothesis, and this would support the contention that the two means were different. And ha! I don't care what kind of mumbojumbo over all P = 0.03 they are reporting, they deal with the mean values in drawing their conclusions. Thus their stats need to deal with the mean. Putting the LF & LC data through this online stats, the REE's are not statistically different to P < 0.1, though they are for P < 0.2 -- or better odds than a roll of the die?!
GIGO! GIGO!
I invite checking of my calcs/assumptions/methods here, but this was the quickest way to check this and it looks bad. Please, tell me it ain't so, and I'll take it all back!
The authors Comment:
The results of our study challenge the notion that a calorie is a calorie from a metabolic perspective. During isocaloric feeding following weight loss, REE was 67 kcal/d higher with the very low-carbohydrate diet compared with the low-fat diet. TEE differed by approximately 300 kcal/d between these 2 diets, an effect corresponding with the amount of energy typically expended in 1 hour of moderate-intensity physical activity.
But:
- MISSING: Statistical support for this misleading assertion.
At best you could hang your hat on a weak trending. And even if the difference in the means was statistically significant, putting the value of 67 cal/day on it for REE (and 300 for TEE) is not supported by that test alone. To be fair, absolutes and percents comparing means are used in reporting every day, but at least they pass the stat.sig. bar. When individual comparisons are taken into account -- with erratic swings in both directions, particularly vs. LGI, it's difficult to support a trending based on carb/fat content at all.
And lastly,
- MISSING: Magical weight loss on isocaloric VLC despite this supposed "metabolic advantage" because clearly these subjects were still markedly overweight or even obese. They would have had to lose weight ...
I'm filing this one in Science Krispies. It belongs there :-(
MORE EDIT:
First, this should not happen. The supplemental materials should be referenced within the main text when describing methods, especially when they provide critical information for the reader to assess the significance of the study. From HERE:
MORE EDIT:
First, this should not happen. The supplemental materials should be referenced within the main text when describing methods, especially when they provide critical information for the reader to assess the significance of the study. From HERE:
We tracked body weight and monitored adherence at daily (Monday through Friday) visits for lunch or dinner. When necessary, we made adjustments in energy intake to achieve weight loss and stabilization during the run-in phase. We allowed the duration of the run-in phase to vary among participants, to account for individual differences in the rate of weight loss. The energy intake required for weight stabilization at the end of the run-in phase was established as the energy intake for the entire test phase, with no further adjustments regardless of any weight fluctuation with the test diets. Participants completed a diary during daily visits to document deviations from study protocols, such as not eating all study foods or consuming any foods or beverages not consistent with the assigned diet. When participants had challenges in complying with protocols, we initiated behavioral counseling. We neither prescribed nor restricted aerobic physical activity but asked participants not to take part in a weight training program.
Since this was to ascertain the effectiveness of a maintenance regime, why not report the counseling frequency? Since compliance was assessed, how about reporting the actual intake or some measure of the degree of this compliance? This furthers the mystery about the TEE and weight, because if caloric level was held constant, the low carbers should have lost weight. Eating 1 meal a day at the facility M-F leaves a lot of "free living". Not saying that's bad, but some mention of self-reported compliance with provided diets would have been nice.
Are you hungry this morning? How do you feel right now? WHY BOTHER!!!
Hunger and Well-being. Prior to breakfast during each hospital admission conducted as part of the test phase, we asked participants to rate level of hunger using a 10-cm visual analog scale, anchored with the descriptors “not at all hungry” and “extremely hungry.” To assess well-being, we asked participants, “How do you feel right now?” Responses were obtained using a visual analog scale, anchored with the descriptors “really terrible” and “really great.” A rating of 10 would represent the highest level of hunger and the best sense of well being on these two scales, respectively.
Are you hungry this morning? How do you feel right now? WHY BOTHER!!!
Comments
If they are free living, compliance could still be an issue, but if this is actually true, might explain the lack of food diaries.
We know they were free living for half of each test diet phase, and half of the initial evaluation.
Since they collected deviation information, they should have commented on whether any of it appeared to be significant or subjects had difficulty adhering to any diet.
If a higher REE/TEE doesn't produce weight loss on an isocaloric intake, that sounds like a dagger in the heart of metabolic advantage >:)
Haven't really addressed the big thing, which you mentioned in your blog -- duration. If there's a metabolic boost to "switching it up", 4 weeks is probably not long enough to settle down to normal. Too many true metabolic ward studies -- including the maintenance leg of Gray & Kipnis -- showing otherwise to read much more into this.
Thus, your claim of "GIGO!" is seriously misplaced. To all appearances (to those of us who understand randomized trials) this is a VERY well-conducted trial.
Anecdote, although a reverse case: when I stopped leptin in may of 2011, I weighed almost 10 pounds less than I do now. I have gained a lot of weight, but all my clothes still fit. While I have obviously gained some body fat, I definitely gained a lot of lean mass too, as evidenced by stronger larger muscles. OMG fatass you gained 10 pounds, but yea, I still wear all my old clothing (it is slightly tigher), it seems much of the weight I gained was lean tissue. I doubt I gained more than 5 pounds of fat given the fact there is not even 1 piece of clothing I cannot wear (there is one pair of jeans which is almost too tight but I can squeeze into them; it should be noted these jeans were tight even last year!)
If at my size (very thin) I can gain pretty much nearly 10 pounds and have a lot of it be shunted into lean tissue, I am pretty sure a bunch of obese people eating VLC diets can lose body fat/retain or gain lean tissue, and weigh exactly the same as another group of fat people who are altering their endocrine dynamic to preserve fat tissue and waste lean from that "insufficient substrate availability in the late postprandial period". Eventually over a longer period of time we would observe weight loss in the VLC group (when lean tissue retention becomes maximal, and fat tissue atrophy leads to loss of scale weight) or alternatively an attenuation of the energy out advantage and no loss of mass.
The higher thyroid and leptin in the LF group suggests fat tissue growth, as a relatively higher T3 and leptin is an expected result of insulin activity and larger fat tissue.
Diet (RQ) = cho% : fat %
----------------------------------
PWLB ( .901) = 67.5 : 32.5
LFD ( .905) = 69 : 31
LGI ( .861) = 54.3 : 45.7
VLC ( .826) = 42 : 58
Using the above numbers subtract the actual fat burnt percentage from the fat supplied in the diet ( all the while assuming weight stability and dietary compliance )
Diet (supplied fat - RQ determined burnt fat) = Result
--------------------------------------
PWLB (30 - 32.5) = 2.5% Dietary Fat Deficit
LFD (20 - 31) = 11% Dietary Fat Deficit
LGI (40 - 45.7) = 5.7% Dietary Fat Deficit
VLC (60 - 58) = 2% Dietary Fat Surplus
So only the VLC diet is burning less fat than actually consumed in the diet?
Sort of makes you wonder where the metabolic advantage is taking you :-)
Does the production of lean body mass, sans water, require a greater energy expenditure than fat mass? If so, do we know how much?
1. It might have been better to have only LF and VLC diets for 6 weeks each. Less confusion, less cost.
2. The VLC could have just as easily be labeled a High Protein diet. 150g/day is not much different than that of a lot of bodybuilders. (This relates to Wooo's comments on body composition. And yes I know that body composition change via eucaloric diet composition differences is controversial, but...)
3. How can physical activity be the same and TEE difference from REE be so large? What generates this additional expenditure?
4. Where did the additional TEE or REE go? Body temperature? They didn't think to take it? Futile cycles? Which cycle? Etc.
5. And finally, the big one. To a layman, the scatter plots indicate a number of things. First, as Nigel says, we are all different, and prescribing a maintenance diet based on these results would be ludicrous. Perhaps which diet to try first, maybe. Second, how can the differences be so great and the blue box (CI?) be so small? I'm guessing it relates to getting the same mean if you repeated the test or something, but why not state the obvious - the results are all over the map, and all the statistical mumbo jumbo you throw at it can't change that. The words confounder and measurement error quickly come to mind.
In the REE data, the error in the CI is almost the maximum differential between the means (60 vs 67). P = 0.03????????
Maybe little pharma funded this >:)
If you're going to do a 3-way, and attempt to draw inferences for one variable (carb quality/quantity), at the very least control what you can -- that being prescribed protein, fatty acid profile and heck, even fiber.
That said, I do believe that the most people will probably achieve acceptable (if not optimal) results somewhere in the middle. Unless they are able to control their environment and limit socializing it's going to be tough. And I think it's dangerous to forge friendships based on eating style. Cuz "come out" when you change course and then what?
All three of us - me, my husband and our son are prone to have allergies. Our family has more serious challenge than a diet which may affect relationships with other people, especially Russian friends - we had to stop social drinking because it immediately makes our allergies worse. My son who goes to a college experiences the most social discomfort, despite that his friends are glad to have him as a constant designated driver. I am the only one LCarber in my family, and as I commented before, my diet made my allergies less of an issue. I now not limited any longer to 45 min present time in my best friend house because she has 3 cats, I can have a glass of vine or two again without wheezing from asthma afterwards, I can eat salmon again. My son stopped eating gluten, it made his eczema much better, but still drinking produces a huge eczema flare. He even considered LCarbing in order to be able to drink at the parties, it would be the most bizarre reason for a diet. He likes mashed potatoes and buckwheat crepes too much for that. He is on vocation at home now, and I have a jar with a mix for the crepes in my fridge all the time.
I've read a lot of Alan Watts, etc., but let's not forget that the Buddha's goal was to eliminate ALL desire. And how that relates to anything escapes me right now, but erasing is hard on my tablet!:-)
I see Evelyn's new post addresses some, but not all, of my 5 points. Anyone have any ideas about say Issue 4?
Post a Comment
Comment Moderation is ON ... I will NOT be routinely reviewing or publishing comments at this time..