Some comments on the Bazzano "LC" vs "LF" Study

So as the weighing in continues regarding:  Effects of Low-Carbohydrate and Low-Fat Diets (full text) I thought I'd add a little bit to the mix.  

First, I do believe the low carb advocates hailing this as any sort of endorsement for their high saturated fat, high animal fat, eat a ton of food, ketogenic or paleo diets need to all watch this entire video of Lydia Bazzano discussing the diet:

Then look at the reported intakes from the study:

To the LC and anti-CICO advocates:  This is not "your" diet no matter who it is "you" are out there touting this study as evidence supporting your diet.  

That aside, I was going to do one of my analyses on this but Kevin Klatt over at Nutrevolve beat me to the punch, and he raised pretty much most if not all of the concerns I was going to raise.  There's not much more to add.    I also began working on a post on clinical trials in general that included some of the issues I have with this study -- and other studies -- but that was getting long and scattered and who knows when I'll get around to crunching it down to be publish-worthy.   

For this post I want to focus solely on the methods and data presentation for the body weight and composition in this study.  

Body Weight and Intent to Treat

Body weight was the primary outcome assessed in this study.  While retention rates were reasonably good for a trial of this nature -- around 80% -- that's still one in five participants who did not complete the trial.  Here is the flow chart following randomization of the 148 participants.  

So, regardless of if they discontinued the study or not, data for all 148 subjects was used in the data analysis.  In other words, they did an "intent to treat" analysis, but they did not state as much unambiguously in the text of the study.   But what's that footnote there?  Really??  I've never seen this before!  They didn't have body weights for 11 subjects at baseline?  How can that be?  This footnote seems to imply that those subjects were included in the analysis anyway.  But let's back up the bus -- why??  

Eleven participants (5 in the low-fat group and 6 in the low-carbohydrate group) declined to have their body weight measured at randomization and were not included in the analysis of our primary outcome.

Excuse me?  I would think that one very basic requirement for qualifying for a diet/weight loss trial would be agreeing to be weighed.  C'mon.  How did these folks make it into this trial at all?   Next time make it clear that in being evaluated at all they are consenting to all diagnostics involved in the study.  Sheesh!!

Thankfully these subjects were not included in the analysis, so LF: n = 67 , LC: n = 69.  Is there any data on how many of these 11 dropped out of the study at any timepoint or in any group?  Nope.  Then we get this:

We used t tests or chi-square tests to compare baseline characteristics between the groups. Dietary composition data were expressed as means (SDs) and compared using t tests. We used a random-effects linear model that was fitted to continuous outcomes (primary and secondary). Each random-effects model consisted of a random intercept and a random slope to adjust for the within-participant correlation among the observed longitudinal data. To examine the change in each study end point, we included an indicator variable in the model for time (3, 6, and 12 months), diet group, an interaction term for diet group by time, and baseline level of the corresponding end point. ....  The random-effects model allows the assumption of data missing at random (MAR). We performed sensitivity analyses to assess the robustness of our conclusions and  departures from the MAR assumption. We used Markovchain Monte Carlo techniques to impute missing values, including additional covariates (age, sex, race, marital status, education, and employment status), in the model to make the MAR assumption more plausible (18).

All of which means that in this analysis, missing values for the dropouts were predicted based on modeling rather than "carrying forward" values either from baseline or last known value.  The two methods are described here.  Now this is just me talking here, but when it comes to diet trials, those that drop out due to issues with the diet were probably less likely to adhere to protocol were they to have stayed in for some reason.  The calculated values are more likely to be in line with the others that stayed in.  While neither method is going to be perfect, it seems to me that at the very least, some sort of comparison to just the completers should be completed and presented.  

Which raises the question of why not just present the completers as the "gold standard" and save the intent-to-treat stuff for the supplementals?   This is a problem I have with all of this and a lot of this was developed for clinical trials of pharmaceutical treatments or medical procedures.  If I have a condition and there are two competing drugs available to treat it, I don't want my doctor relying on some singular ITT parameter to make his recommendation for my first course of action.  If Drug A has more potential side effects so only 40% tolerate it well, but 80% of those respond fully, I think that matters compared to Drug B where 80% can tolerate it, but only 40% of those respond to treatment.   In ITT, these two drugs are equivalent but I wouldn't want my doctor just flipping a coin, I'd like him to help me decide which might be my best first approach given my individual makeup.  I've said this many times before with diet trials -- I want to know what the long term implications are for those who actually follow a diet.  Similarly in the comparisons, when folks are considering which diet to try, it helps if the expectations are based on results attained by others who followed the diet rather than how 10, 100 or even 1000 people assigned to eat that way fared.  We don't need YET another study to demonstrate that long term adherence is the single best predictor, or that personal preferences will factor heavily into that.  

So back to the study at hand, 11 people were not included in the analysis for weight ... but they were included for other outcomes?  Strange.  One must presume they were also not included in the body composition analysis as well?  Speaking of ...

The Strange Body Composition Results:

One of the problems with many diet trials is that they just measure weight loss and don't assess the composition of the weight in terms of fat loss vs. lean loss.  We are told that:

We measured body composition using whole-body bioelectrical impedance analysis (RJL Systems) while the participant was in a supine position.

Wikipedia gives a pretty good rundown of BIA.  Usually "test day" procedures are outlined far more thoroughly than was presented in this study, and given the drawbacks of BIA, there should have been some uniform process such as: subjects were instructed to fast overnight, drink or not drink fluids, abstain from activity, all get measured at the same time of day, etc.  

Next, we are given the following descriptive stats for body weight and composition at baseline.   I included weight because you can see you have the asterisk on weight but not on body composition which would imply that there were eleven participants who declined to step on a scale but submitted to the BIA?   But far worse is that you have percentages expressed to the whole percent when later changes are to the tenths of a percentage (of change, but that's a whole lotta rounding for small values).   It seems highly unlikely that two groups of 73 and 75 participants would have the exact same mean fat mass percentage.  Still, that could happen.  But that the distribution of the 70-plus individual values is so nearly identical such that the standard deviation of the values is also exactly the same between groups?  That seems danged near impossible to me and highly suspect.  

Weight loss from baseline values was greater in the low-carbohydrate group than in the low-fat group at 3, 6, and 12 months (Table 3). The reduction in body weight was significantly greater in the low-carbohydrate group (mean difference in change at 12 months, 3.5 kg [95% CI, 5.6 to 1.4 kg]; P 0.002). Compared with participants on the low-fat diet, those on the low-carbohydrate diet had significantly greater proportional reductions in fat mass (mean difference in change at 12 months, 1.5% [CI, 2.6% to 0.4%]; P 0.011) and significantly greater proportional increases in lean mass (mean difference in change at 12 months, 1.7% [CI, 0.6% to 2.8%]; P 0.003).

Here is the table of changes.  Columns are left to right:  LF, LC and LC-LF difference.

Can you think of any more obscuring manner in which to present these results?  In the baseline table, they refer to it as fat mass and lean mass but report percents.  So when they say "reduction in fat mass" are they referring to an actual reduction in the mass or the % of fat mass -- because these are two different things.  One can presume they mean % fat mass, but there should be no ambiguity here.  I've constructed a series of scenarios to demonstrate.  In this first series, we begin with a 100 kg person who is 40% fat and 60% lean, thus begins with a starting fat mass of 40 kg and lean mass of 60 kg.  This person loses 5 kg of total mass in six scenarios ranging from all fat to all lean.

As you can see, up through losing 2 kg of lean mass loss, this person "gains" percentage lean mass, breaks even at 3 kg lean mass loss, and only after that begins to "lose" lean mass when evaluated as a percentage.   The tipping point of this would depend on the original composition and as you can see, occurs when the weight loss composition matches the initial composition.  Let's compare this to the same scenarios with the only difference being that the person starts out at 90 kg.

As you can see, when the 100 kg person lost 5 kg fat, this translated to a change in fat mass percent from 40 to 36.8 for 3.2% point loss in fat mass and a corresponding 3.2% "gain" in lean mass (despite no change in absolute lean mass).   This fat loss and lean "gain" is increased to 3.5% for the same 5 kg fat loss in the 90 kg person.     With a 3 kg fat: 2 kg lean split, the fat loss/lean gain is 1.1% for the 100 kg, 1.2% for the 90 kg person.

However in all but the first scenario in each series, the hypothetical individual LOST lean mass in absolute terms.

But if the losses reported were differences in percents, then the % fat and % lean should always be equal and opposite as in the above scenarios regardless of starting mass or distribution of fat/lean of the loss (and as would make sense so that they add to 100%).  They are not, so I am inclined to believe that the changes reported as "proportional" are the percentage change of the initial component percentage.  For fat:

[(fat%final - fat%initial) / fat%initial]*100

This is even more complicated!  Whether they are averaging either an absolute change in fat mass percentage or a percentage change in fat mass percentage, these values are hopelessly variable compared to the absolute mass loss for each individual as they will be impacted by three factors:  original mass, original composition and composition of the weight change.   Here's where those 12% men in the mix can throw things off as well as one would expect them to weigh more and have lower body fat at a given weight.  How many were among the 11 who didn't participate in being weighed?  How was the distribution of men between groups after attrition and how did this impact imputed values?   I played with some scenarios on my spread sheet, and it is possible that for the same kg mass, fat and lean losses, it is not all that difficult to have one person "losing" fat and "gaining" lean while the other is just the opposite, then taking that as a proportion of the percents and the difference between them?  Tired brain here ...

So I constructed the table below.  Note that the values in blue are the data reported directly in the paper.  The top row of values in each table is the baseline data.  The values above the % Fat and % Lean columns represent the percent change in fat or lean percentage as reported.  Thus the percentages at the time points were calculated as follows:

% Xtimepoint = % Xinitial + (% change in %X as decimal)*(% Xinitial)

I took these out to two decimal places.  Also on the right in small font is the sum of these two percentages.  (As you can see there is some rounding error introduced.)  I then used these values and the total mass at each time point to calculate the average absolute fat mass and lean mass.  Lastly I took the differences in all to show absolute changes in total, fat and lean mass, and in the faded numbers below those I calculated the proportion of the total represented by fat and lean.  (Again, rounding is what it is here).

So, contrary to the impression made by the reporting of differentials in the average percentage change in the percentage of fat or lean -- that is both a mouthful and a seriously confusing way to present the data -- the results were as follows:

• For both groups at all time points, there was an average net LOSS of total mass, fat mass and lean mass compared with baseline.  
• The low fat group regained 0.8 kg from 3 to 12 months while the low carb group regained half that or 0.4 kg from 3 to 12 months.
• The low fat group regained 31% of the weight lost at 3 months by 12 months, while the low carb group regained only 8% for the same time points.
• At three months, the low carb group lost roughly twice the total mass, fat mass and lean mass as did the low fat group.  
• The low fat group lost roughly 1.3 kg of lean mass at three months and then gained back mostly fat mass through 12 months.  This impacted body composition by percentage in such a way as to make it appear that more lean mass was lost but this is not true.  It also makes it appear that there is a net gain of fat mass from baseline and this is also not true.  (see below)
  

• The low carb group lost roughly 2.5 kg of lean mass at three months and then also gained back mostly fat mass through 12 months.  The combination of greater losses and smaller regains obscures this and makes it appear that there was a net gain of lean mass when this is not true.  It also makes it appear as if fat loss continued (albeit slight) when it did not.  (see above)
• The maximum absolute fat loss compared to baseline for both groups was at 3 months and was 1.15 kg for low fat vs. 2.68 kg for low carb, a difference of 1.53 kg.  

Some calorie math:

As Kevin so thoroughly documented in his blog post, the probability that the self-reported intake is accurate is very small.  Let's assume for the sake of argument they are accurate and the calories reported at each time point were constant for the preceding months.  There was a daily calorie intake difference (vs baseline) reported between LF and LC at 3, 6 and 12 months of 124, 121 and 43 calories respectively.  Using 30.4 days in a month, the total calorie differential would be:

3*30.4*124 + 3*30.4*121 + 6*30.4*43 = 30,187 calories.
30,187 cal divided by 3500 cal/lb = 8.62 lbs  (divided by 2.2 = 3.9 kg)

This is more than enough to account for the weight and fat loss differences regardless of the accuracy of the BIA.  Of course these values do not reach a level of statistical significance but since the low carbers consistently reported lower intake than the low fat group, it is not an unreasonable possible explanation for differences in fat and weight loss observed.  It certainly beats magic!

In Conclusion: 

The combination of potential issues with the method of body composition analysis and the inherent issues with 24-hour recall in free living studies renders this study useless in answering any questions regarding the utility of macronutrient restriction.  This is irrespective of the fact that the LC diet would be easier to implement (gram goal vs. percentage of calories of an unspecified target goal).  It also doesn't appear that protein differences had much of an effect with the magnitudes of losses and gains observed.  The fat to lean loss ratio appears to favor the higher protein low carb diet (which even at 3 months was around 100 grams of carbohydrate so not ketogenic or VLC), and it is possible that some lean losses were regained.  Although the absolute protein consumption was not hugely different (another obscured factor) it may be that maintaining protein to absolute baseline levels (85-90 gram area) was more important than the magnitude of the protein intake reduction (at most 20 g - I'm out of time this is from memory).

It's time for some Clinical Trial Clarity!!!