Some comments on the Bazzano "LC" vs "LF" Study

So as the weighing in continues regarding:  Effects of Low-Carbohydrate and Low-Fat Diets (full text) I thought I'd add a little bit to the mix.  

First, I do believe the low carb advocates hailing this as any sort of endorsement for their high saturated fat, high animal fat, eat a ton of food, ketogenic or paleo diets need to all watch this entire video of Lydia Bazzano discussing the diet:



Then look at the reported intakes from the study:


To the LC and anti-CICO advocates:  This is not "your" diet no matter who it is "you" are out there touting this study as evidence supporting your diet.  

That aside, I was going to do one of my analyses on this but Kevin Klatt over at Nutrevolve beat me to the punch, and he raised pretty much most if not all of the concerns I was going to raise.  There's not much more to add.    I also began working on a post on clinical trials in general that included some of the issues I have with this study -- and other studies -- but that was getting long and scattered and who knows when I'll get around to crunching it down to be publish-worthy.   

For this post I want to focus solely on the methods and data presentation for the body weight and composition in this study.  


Body Weight and Intent to Treat


Body weight was the primary outcome assessed in this study.  While retention rates were reasonably good for a trial of this nature -- around 80% -- that's still one in five participants who did not complete the trial.  Here is the flow chart following randomization of the 148 participants.  

So, regardless of if they discontinued the study or not, data for all 148 subjects was used in the data analysis.  In other words, they did an "intent to treat" analysis, but they did not state as much unambiguously in the text of the study.   But what's that footnote there?  Really??  I've never seen this before!  They didn't have body weights for 11 subjects at baseline?  How can that be?  This footnote seems to imply that those subjects were included in the analysis anyway.  But let's back up the bus -- why??  
Eleven participants (5 in the low-fat group and 6 in the low-carbohydrate group) declined to have their body weight measured at randomization and were not included in the analysis of our primary outcome.
Excuse me?  I would think that one very basic requirement for qualifying for a diet/weight loss trial would be agreeing to be weighed.  C'mon.  How did these folks make it into this trial at all?   Next time make it clear that in being evaluated at all they are consenting to all diagnostics involved in the study.  Sheesh!!

Thankfully these subjects were not included in the analysis, so LF: n = 67 , LC: n = 69.  Is there any data on how many of these 11 dropped out of the study at any timepoint or in any group?  Nope.  Then we get this:
We used t tests or chi-square tests to compare baseline characteristics between the groups. Dietary composition data were expressed as means (SDs) and compared using t tests. We used a random-effects linear model that was fitted to continuous outcomes (primary and secondary). Each random-effects model consisted of a random intercept and a random slope to adjust for the within-participant correlation among the observed longitudinal data. To examine the change in each study end point, we included an indicator variable in the model for time (3, 6, and 12 months), diet group, an interaction term for diet group by time, and baseline level of the corresponding end point. ....  The random-effects model allows the assumption of data missing at random (MAR). We performed sensitivity analyses to assess the robustness of our conclusions and  departures from the MAR assumption. We used Markovchain Monte Carlo techniques to impute missing values, including additional covariates (age, sex, race, marital status, education, and employment status), in the model to make the MAR assumption more plausible (18).
All of which means that in this analysis, missing values for the dropouts were predicted based on modeling rather than "carrying forward" values either from baseline or last known value.  The two methods are described here.  Now this is just me talking here, but when it comes to diet trials, those that drop out due to issues with the diet were probably less likely to adhere to protocol were they to have stayed in for some reason.  The calculated values are more likely to be in line with the others that stayed in.  While neither method is going to be perfect, it seems to me that at the very least, some sort of comparison to just the completers should be completed and presented.  

Which raises the question of why not just present the completers as the "gold standard" and save the intent-to-treat stuff for the supplementals?   This is a problem I have with all of this and a lot of this was developed for clinical trials of pharmaceutical treatments or medical procedures.  If I have a condition and there are two competing drugs available to treat it, I don't want my doctor relying on some singular ITT parameter to make his recommendation for my first course of action.  If Drug A has more potential side effects so only 40% tolerate it well, but 80% of those respond fully, I think that matters compared to Drug B where 80% can tolerate it, but only 40% of those respond to treatment.   In ITT, these two drugs are equivalent but I wouldn't want my doctor just flipping a coin, I'd like him to help me decide which might be my best first approach given my individual makeup.  I've said this many times before with diet trials -- I want to know what the long term implications are for those who actually follow a diet.  Similarly in the comparisons, when folks are considering which diet to try, it helps if the expectations are based on results attained by others who followed the diet rather than how 10, 100 or even 1000 people assigned to eat that way fared.  We don't need YET another study to demonstrate that long term adherence is the single best predictor, or that personal preferences will factor heavily into that.  

So back to the study at hand, 11 people were not included in the analysis for weight ... but they were included for other outcomes?  Strange.  One must presume they were also not included in the body composition analysis as well?  Speaking of ...


The Strange Body Composition Results:


One of the problems with many diet trials is that they just measure weight loss and don't assess the composition of the weight in terms of fat loss vs. lean loss.  We are told that:
We measured body composition using whole-body bioelectrical impedance analysis (RJL Systems) while the participant was in a supine position.
Wikipedia gives a pretty good rundown of BIA.  Usually "test day" procedures are outlined far more thoroughly than was presented in this study, and given the drawbacks of BIA, there should have been some uniform process such as: subjects were instructed to fast overnight, drink or not drink fluids, abstain from activity, all get measured at the same time of day, etc.  

Next, we are given the following descriptive stats for body weight and composition at baseline.   I included weight because you can see you have the asterisk on weight but not on body composition which would imply that there were eleven participants who declined to step on a scale but submitted to the BIA?   But far worse is that you have percentages expressed to the whole percent when later changes are to the tenths of a percentage (of change, but that's a whole lotta rounding for small values).   It seems highly unlikely that two groups of 73 and 75 participants would have the exact same mean fat mass percentage.  Still, that could happen.  But that the distribution of the 70-plus individual values is so nearly identical such that the standard deviation of the values is also exactly the same between groups?  That seems danged near impossible to me and highly suspect.  
Weight loss from baseline values was greater in the low-carbohydrate group than in the low-fat group at 3, 6, and 12 months (Table 3). The reduction in body weight was significantly greater in the low-carbohydrate group (mean difference in change at 12 months, 3.5 kg [95% CI, 5.6 to 1.4 kg]; P 0.002). Compared with participants on the low-fat diet, those on the low-carbohydrate diet had significantly greater proportional reductions in fat mass (mean difference in change at 12 months, 1.5% [CI, 2.6% to 0.4%]; P 0.011) and significantly greater proportional increases in lean mass (mean difference in change at 12 months, 1.7% [CI, 0.6% to 2.8%]; P 0.003).
Here is the table of changes.  Columns are left to right:  LF, LC and LC-LF difference.


Can you think of any more obscuring manner in which to present these results?  In the baseline table, they refer to it as fat mass and lean mass but report percents.  So when they say "reduction in fat mass" are they referring to an actual reduction in the mass or the % of fat mass -- because these are two different things.  One can presume they mean % fat mass, but there should be no ambiguity here.  I've constructed a series of scenarios to demonstrate.  In this first series, we begin with a 100 kg person who is 40% fat and 60% lean, thus begins with a starting fat mass of 40 kg and lean mass of 60 kg.  This person loses 5 kg of total mass in six scenarios ranging from all fat to all lean.


As you can see, up through losing 2 kg of lean mass loss, this person "gains" percentage lean mass, breaks even at 3 kg lean mass loss, and only after that begins to "lose" lean mass when evaluated as a percentage.   The tipping point of this would depend on the original composition and as you can see, occurs when the weight loss composition matches the initial composition.  Let's compare this to the same scenarios with the only difference being that the person starts out at 90 kg.


As you can see, when the 100 kg person lost 5 kg fat, this translated to a change in fat mass percent from 40 to 36.8 for 3.2% point loss in fat mass and a corresponding 3.2% "gain" in lean mass (despite no change in absolute lean mass).   This fat loss and lean "gain" is increased to 3.5% for the same 5 kg fat loss in the 90 kg person.     With a 3 kg fat: 2 kg lean split, the fat loss/lean gain is 1.1% for the 100 kg, 1.2% for the 90 kg person.

However in all but the first scenario in each series, the hypothetical individual LOST lean mass in absolute terms.

But if the losses reported were differences in percents, then the % fat and % lean should always be equal and opposite as in the above scenarios regardless of starting mass or distribution of fat/lean of the loss (and as would make sense so that they add to 100%).  They are not, so I am inclined to believe that the changes reported as "proportional" are the percentage change of the initial component percentage.  For fat:

[(fat%final - fat%initial) / fat%initial]*100

This is even more complicated!  Whether they are averaging either an absolute change in fat mass percentage or a percentage change in fat mass percentage, these values are hopelessly variable compared to the absolute mass loss for each individual as they will be impacted by three factors:  original mass, original composition and composition of the weight change.   Here's where those 12% men in the mix can throw things off as well as one would expect them to weigh more and have lower body fat at a given weight.  How many were among the 11 who didn't participate in being weighed?  How was the distribution of men between groups after attrition and how did this impact imputed values?   I played with some scenarios on my spread sheet, and it is possible that for the same kg mass, fat and lean losses, it is not all that difficult to have one person "losing" fat and "gaining" lean while the other is just the opposite, then taking that as a proportion of the percents and the difference between them?  Tired brain here ...

So I constructed the table below.  Note that the values in blue are the data reported directly in the paper.  The top row of values in each table is the baseline data.  The values above the % Fat and % Lean columns represent the percent change in fat or lean percentage as reported.  Thus the percentages at the time points were calculated as follows:

% Xtimepoint = % Xinitial + (% change in %X as decimal)*(% Xinitial)

I took these out to two decimal places.  Also on the right in small font is the sum of these two percentages.  (As you can see there is some rounding error introduced.)  I then used these values and the total mass at each time point to calculate the average absolute fat mass and lean mass.  Lastly I took the differences in all to show absolute changes in total, fat and lean mass, and in the faded numbers below those I calculated the proportion of the total represented by fat and lean.  (Again, rounding is what it is here).

So, contrary to the impression made by the reporting of differentials in the average percentage change in the percentage of fat or lean -- that is both a mouthful and a seriously confusing way to present the data -- the results were as follows:
  • For both groups at all time points, there was an average net LOSS of total mass, fat mass and lean mass compared with baseline.  
  • The low fat group regained 0.8 kg from 3 to 12 months while the low carb group regained half that or 0.4 kg from 3 to 12 months.
  • The low fat group regained 31% of the weight lost at 3 months by 12 months, while the low carb group regained only 8% for the same time points.
  • At three months, the low carb group lost roughly twice the total mass, fat mass and lean mass as did the low fat group.  
  • The low fat group lost roughly 1.3 kg of lean mass at three months and then gained back mostly fat mass through 12 months.  This impacted body composition by percentage in such a way as to make it appear that more lean mass was lost but this is not true.  It also makes it appear that there is a net gain of fat mass from baseline and this is also not true.  (see below)
  • The low carb group lost roughly 2.5 kg of lean mass at three months and then also gained back mostly fat mass through 12 months.  The combination of greater losses and smaller regains obscures this and makes it appear that there was a net gain of lean mass when this is not true.  It also makes it appear as if fat loss continued (albeit slight) when it did not.  (see above)
  • The maximum absolute fat loss compared to baseline for both groups was at 3 months and was 1.15 kg for low fat vs. 2.68 kg for low carb, a difference of 1.53 kg.  


Some calorie math:


As Kevin so thoroughly documented in his blog post, the probability that the self-reported intake is accurate is very small.  Let's assume for the sake of argument they are accurate and the calories reported at each time point were constant for the preceding months.  There was a daily calorie intake difference (vs baseline) reported between LF and LC at 3, 6 and 12 months of 124, 121 and 43 calories respectively.  Using 30.4 days in a month, the total calorie differential would be:
3*30.4*124 + 3*30.4*121 + 6*30.4*43 = 30,187 calories.
30,187 cal divided by 3500 cal/lb = 8.62 lbs  (divided by 2.2 = 3.9 kg)
This is more than enough to account for the weight and fat loss differences regardless of the accuracy of the BIA.  Of course these values do not reach a level of statistical significance but since the low carbers consistently reported lower intake than the low fat group, it is not an unreasonable possible explanation for differences in fat and weight loss observed.  It certainly beats magic!


In Conclusion: 


The combination of potential issues with the method of body composition analysis and the inherent issues with 24-hour recall in free living studies renders this study useless in answering any questions regarding the utility of macronutrient restriction.  This is irrespective of the fact that the LC diet would be easier to implement (gram goal vs. percentage of calories of an unspecified target goal).  It also doesn't appear that protein differences had much of an effect with the magnitudes of losses and gains observed.  The fat to lean loss ratio appears to favor the higher protein low carb diet (which even at 3 months was around 100 grams of carbohydrate so not ketogenic or VLC), and it is possible that some lean losses were regained.  Although the absolute protein consumption was not hugely different (another obscured factor) it may be that maintaining protein to absolute baseline levels (85-90 gram area) was more important than the magnitude of the protein intake reduction (at most 20 g - I'm out of time this is from memory).

It's time for some Clinical Trial Clarity!!!

Comments

Greta Carbo said…
Wow! Good post. This is going to keep me up all night figuring it all out.
Catweazle said…
Who paid for this kind of useless studies? Sorry, but i can't believe that people can do 1500kcal over 12 month without a fully blown methabolic and hormonal shutdown. And on the other hand, 1500kcal would be the average breakfast intake of Mr. Keto-La-Vida himself.
StellaBarbone said…
Actually, 1500 kCal is just a bit less than I need for maintenance of my weight. I'm older and thinner than these women, but had they been truly eating 1500-1600 kCal per day, they would have lost a considerable amount of weight. By eyeball, they should have lost something in the 30-50 lbs range. Since they were clearly not eating what they said they were eating, it doesn't make much sense to go analyzing the made up numbers in any detail....
carbsane said…
Yeah, I feel this way more and more any time I take a deeper look into one of these studies. Almost to a of the conclusions should state: The method we used to assess intake has been shown to be inaccurate in many studies, and the results of this study seem to have demonstrated this once again. No more studies are needed until this problem has been addressed.
carbsane said…
I'm not sure about that. While I'm sure there was under-reporting involved, the average intake for American women in 1970 was around 1550 cal/day. Even if the average is 2000, a conservative estimate for standard deviation is 200 cal/day and thus 95% of women need 1600-2400 cal/day.


These (mostly) women are reporting a roughly 5-700 cal/day deficit. Even if that were accurate this would hardly cause "full blown metabolic and hormonal shutdown". While you can surely mess with your system with severe restriction, the whole broken metabolism thing is vastly overstated.
carbsane said…
Thanks Greta! I'm still trying to respond to your comment about Gretchen's paper. Gotta go find that :-) If I forget (things are INSANE here) please remind me as it raised some interesting thoughts!
EJ said…
1500 calories per day is very doable if the food is nutritionally dense. 1500 calories is about 10 cans of water-packed sardines, or about a dozen containers of plain Greek yogurt, or about 20 large eggs, or more than 20 cups of raspberries, or about 8 packages of frozen peas, or about 20 bunches of spinach, or about 50 medium-sized zucchinis, or an entire loaf of Ezekiel Bread, or some combination of these foods.
Emmie said…
Not only are the low-carvers all over this study, but on one board (LCF) there's a 'nutty K' group--and all of them are now doing fat fasts and egg fasts because they weren't losing any weight with nutty k! Of course, Jimmy is always right, so they're still committed to nutty K!
NEPAer said…
Whatever became of your book I saw mentioned on some other site?
Kitty said…
Wasn't whole Atkins pitch that one could eat all the high-fat foods one loves (e.g. steak, cheese, rich sauces, etc.) and lose weight? Now it's down to fat fasts (dear God) and butter and egg diets?


I remember an Amazon review that Moore did of Joel Fuhrman's Eat to Live diet book wherein he said, "If this is how I eat to live, then kill me now!" At least he knows what people think of his diet. What kind of person would be thrilled to hear that they can eat all the butter and eggs they want and still lose weight? What's so sad is gaining weight on such a grim diet.
Greta Carbo said…
Principle funding source was the National Institutes of Health. The study does seem rather useless though, I agree.


The groups reported approximately a 500 calorie deficit but lost weight consistent with a 100 calorie deficit. Hmmm, seems about right to me. :)


Other than that, I find this study to be entertaining in the various reactions around the internet and instructive to delve into as it is examined here in the Carb Sane Academy or in Kevin Klatt's excellent blog post and precious few other sites that look a little deeper than the surface.
Emmie said…
These nutty K folks are all Jimmy Moore fans, so they actually believe that they can eat all the fat they want and it 'doesn't count.' So since they're not losing, they are off on fat fasts and egg fasts in an attempt to 'prove' that nutty K works. Some of them have even resorted to "It's health that's important, not weight loss," as though they 'know' that nutty K is improving their health. I find it amazing how people are wed to dietary ideologies regardless of outcome.
MacSmiley said…
In calculating anticipated weight loss, what people fail to take into account is the need for fewer calories as one's body mass decreases. That's one reason why the math is not so simple as that.
fatsane said…
You should all be aware they've now conducted 24 low carb vs low fat studies now - in most cases the outcome is similar to this one.

We actually won't know whether low carb is more metabolically efficient for weight loss until the end of the rigorously controlled study going on right now at the Boston Children's Hospital that is being conducted by David Ludwig and Cara Ebbeling.

Wired actually wrote a nice piece about it:

"In January of this year, the first subject checked into the metabolic ward at the National Institutes of Health in Bethesda, Maryland, to participate in one of the most rigorous dietary studies ever devised. For eight weeks, he was forbidden to leave. He spent two days of each week inside tiny airtight rooms known as metabolic chambers, where scientists determined precisely how many calories he was burning by measuring changes in oxygen and carbon dioxide in the air. He received meals through vacuum-sealed portholes so that the researchers' breath wouldn't interfere with their measurements. The food itself had been chemically analyzed to ensure an exact number of carbohydrate, protein, and fat calories.

The two-day stays in the chambers were only a small part of the testing, which was also being carried out on subjects at three other institutions around the US. Twice a month, the subjects were required to lie down for dual-energy x-ray absorpti­ometry scans, an accurate way to measure body fat. They offered up their veins again and again so that scientists could measure their lipids and hormone levels. They provided samples of their stools so the researchers could record the different colonies of bacteria residing in their guts.

And yet for all the poking, prodding, measuring, and testing, the most remarkable thing about the $5 million undertaking may be that it's designed to answer a question you'd think we'd have answered long ago: Do we get fat because we overeat or because of the types of food we eat?"

http://www.wired.com/2014/08/what-makes-us-fat/
fatsane said…
Why is fat loss without low calories "magic" - when people are overfed - some gain no fat - is this magic? No, the body is more than capable of not storing calories as fat whether through energy expenditure or simple excretion. Some people can't gain weight no matter how much they eat - type 1 diabetics for instance - no matter how much they eat - they die emaciated if left untreated with insulin. Why is the obverse of this, that some people can lose weight eating larger than expected amounts "magic".
2lbs of Starch said…
Assuming undigested calories are the same in both groups, then you're saying the low-carb group raised their basal metabolic rate? Would a few (believable) percentage points make a difference?
David Pete said…
Please explain why this never happens in a metabolic ward, with normal healthy people
carbsane said…
Why do I feel like I'm having a conversation with someone cribbing comments from some blog circa 2010??


If you have a fat malabsorption problem, I'd get that checked out. Otherwise, no human has ever verifiably failed to gain weight eating more or lose weight eating fewer calories.


I'm also going to suggest you read a physiology text as to the role of insulin and why T1's died of the disease before insulin was discovered. If you think it's about fat tissue only you are simply ignorant. Educate yourself.
carbsane said…
Nice puff piece.


Interesting how they are going to use a metabolic chamber to measure calories. I thought we weren't bomb calorimeters so calories don't matter :p
carbsane said…
There's an update in the link at the side bar but even that is now woefully outdated. I am still working on the updated and expanded book. There has been a lot going on "in real life" as they say with a very ill relative. Book writing requires both undistracted chunks of time and ability to focus that have been hard to come by. My blogging has fallen off a bit as well, but it is a much different process so I'm still at it.
carbsane said…
In addition to what MacSmiley said,


Online calculators are good for estimating, but as I've discussed here multiple times, individual variability makes them pretty useless for a large number of people.


One of the problems most people face is they really have no clue what they are consuming to maintain whatever weight they are at. I've blogged on a number of studies where some manner of baseline self report was then fed to subjects for a few weeks and weight loss is seen. Any study in which a standard "low fat" CRD is tested ought to involve clear determination of TDEE at baseline adn NOT through 24 hour recall. At the least, it should be from a couple of weeks of logging where foods are weighed and measured.


Free living studies where all foods are prepared and/or provided for the participants get better results. Programs like Jenny, Nutrisystem and even Medifast and SlimFast are successful when people adhere because they don't have to do any "work". It's of course not perfect,
Jane Karlsson said…
That Bazzano video is odd. Her study suggests she knows nothing about whole vs refined carbs, but in the interview she knows everything you'd expect. At least everything you'd expect since David Katz came along. Perhaps she designed the study pre-Katz, and did the interview in an attempt to tell the world she does know the difference between white rice and brown rice.
MacSmiley said…
In metabolic ward studies, they've seen instances where formerly obese people will burn calories at a rate of several hundred calories per day less than people of the same weight who were never obese.

True that.

But when a study like Bazanno is published, if the weight loss is several hundred calories per day different than the food diaries would suggest, then the immediate assumption is that the subjects lied to their food diary.

That would not be the case if in other metabolic ward studies, obese subjects who lost weight in the ward were then let out began to gain weight again on the same alleged number of calories they lost weight on before. Then returned to the ward, lost weight again.

I know I'M prone to eating several extra spoonfuls of beans or lentils while Im serving my usual serving sizes. Frozen mango is particularly tempting out of the bag as well.

I think there was one case where a woman in a study was failing to report about 8 tablespoons of peanut butter she was licking up between meals. Her husband finally fessed up on her behalf. Perhaps Evelyn knows which studies those were.
But . . . Bull****proof Coffee is sheer magic!
Trae Palmer said…
Pfft, yeah, the resultant high-viscosity coffee is just gross. I think people like to imagine that it tastes like some of those flavored roasts, but it's not even close...even if one adds 'fixins' like cinnamon, Sisson-/Fallon-approved cream or whatever. There's also the little fact that flavored-roast coffees contain negligible calories, whereas Bulletproof coffee at 2 tbsp min of Kerrygold and 2 tbsp of coconut or MCT oil (the minimum added fat amounts Asprey seems to recommend) works out to around (100 * 2) + (128 * 2) = 456 kcal per 2 cups, if one uses the MCT oil he sells and recommends.

But, of course, the insulin faeries just take over (maybe at night when the Underpants Gnomes do their work, as well) and the regular coffee I'm buying from the store is probably full of 'toxins' that the powers that be are all in collusion to keep hidden from me, and Bulletproof coffee, albeit having twice the caloric density of Coca Cola, would actually cause me to lose weight and give me more energy and so on and so on -_^

Dave apparently likes taking the Kerrygold up to 6 to 8 tbsps when he
makes it for himself...I bet that's a drink that everyone would
just love to chug down in the morning...gosh these ancestral health
diets sure sound so much more appetizing than those bland-old, low-fat
breakfast options (eyeroll).
Absolutely . . . Who'd want peasant food when there's a handy-dandy alternative of salted-battery acid--err, I mean special ketogenic coffee.
Trae Palmer said…
ROFL...I'm gonna have to use the 'salted battery acid' thing.