On Cherry Picking & Debunking

A repeated response - in the comments here and over at Weighty Matters - to my recent post, Calories, fat or carbohydrates? Why diets work (when they do) ,  included two criticisms.

  1. That I am guilty of the same cherry picking that I accuse Taubes of, and
  2. That my conclusions are somehow misguided, or whatever,  because they are not what the original authors concluded or were studying.  
OK.  So to the first point, what is cherry picking?  Well, there are two types:
a.  Including only references that support one's theories and ignoring those that refute it, or
b.  Selecting phrases or data from a single source while ignoring other phrases or data that contradict one's thesis.
I hereby award Taubes an honorary PhD (as in piled higher and deeper) in Cherry Picking from the distance learning institute at the Asylum.  He has demonstrated an extraordinary aptitude in cherry picking of all varieties.  If the triple thesis half decade of work culminating in GCBC weren't enough, he solidified his worthiness for this degree with the thesis he published in the peer review journal of the ISLCF*.  

Example of (a) here:  Insulin Resistance ~ Taubes v. McGarry  While that post focused on Taubes misrepresenting McGarry's work, one can see how rather than pursuing McGarry's work, he chose to highlight the older suppositions of Neel.   Taubes' references in GCBC are copious, but his research and referencing in the text are FAR from comprehensive or exhaustive.  There's a difference.  

But the examples of (b) are far easier to cite and list where Dr. Taubes is concerned.  Firstly we have the Shai study.  A more clear demonstration of cherry picking would be difficult to find.  He ignores one third of the data in the study!  If that data were consistent with his thesis, this might be acceptable.  Pitting LC vs. LF is a pretty fair head-to-head given the low fat craze.  But the MDTN group data was inconsistent with his thesis.  They restricted carbs the least and lost more weight than the LF group and ever so slightly (statistically the same) less weight than the LC group.  You can't get around the blatancy of this!

Some more examples of (b):  

GCBC Reference Check ~ Part I of ? ~ Metabolic Adaptability & Energy Balance
In this one, Taubes cites a the "dieting is difficult" quote from a text he now admits he never read.  But at the time one might assume he at least read the chapter of the text from which he *picked* this quote.  Agree or not with Frayn's analysis in this chapter, it does not support Taubes' theories.
Then there's  Taubes failure to include contradictory evidence from elsewhere in Frayn's book as regards Insulin Resistance and de novo lipogenesis to name a few.  I would add that Taubes' discussion of IR is fraught with examples of (a) as regards his utter failure to pursue the works of one of his cited scientists.
GCBC Reference Check ~ Part III of ? ~ Is glycerol phosphate rate-limiting?

This one is the most egregious of all.  Not only did he cherry pick, but he referenced the entire discussion and flat out misrepresents what the text said.  Can't get much worse than this one.
Lastly, although this list could go on and on and on, we have GCBC Reference Check ~ Part IV of ? ~ Kipnis.  This is perhaps the most pertinent example of all, because Grey & Kipnis was the study I discussed in the  Calories, fat or carbohydrates? Why diets work (when they do) post.  At the time I was not aware that Taubes was familiar with this work.  But he was, and then some!  He also interviewed Kipnis.   At least he presented Kipnis' views forthrightly but he did not do much of the sort of "deep thinking" I'm accused of avoiding as regards this GCBC reference.  

So basically I'm accused of cherry picking from this same study because:
a.  I didn't address that high or low carb did alter fasting insulin levels, and
b.  I drew a different conclusion from the data than did the authors.  Their's being: "composition of the diet [carb content] may represent a significant causative factor for the elevated plasma insulin levels observed characteristically in marked obesity", and "Thus the hyperinsulinemia characteristic of obesity may be the result of dietary factors rather than exclusively a consequence of the insulin antagonism associated with obesity" 
Let's address these one at a time.  In Calories, fat or carbohydrates? Why diets work (when they do), I was looking at the evidence in support of the assertion that it is calories that determine weight loss irrespective carbohydrate intake.  In that regard I focused solely on calorie and carb consumption and the resulting changes, if any, in body weight in G&K.  This, folks, is not cherry picking.  Cherry picking would have been failure to include some data in this study demonstrating otherwise.  But I discussed ALL of the results pertinent to just this topic.  Similarly with Shai, I discuss all of the data and offer a reasonable (almost assured) explanation for all of them:  e.g. that self report dietary records are to be taken with a grain of salt.  (Which coincidentally I'm in agreement with Taubes on.)

On to point 2.  What of the dust-up over drawing different conclusions than the authors?  First, I would state that I'm not the only one who has ever looked at the data from studies to look at different variables.  I mean do the names Denise Minger or Ned Kock not ring a bell for most of my readers?   Aren't bloggers routinely having a field day "debunking" scientific studies by concluding just the opposite of those willful failure, Big ______ (fill in the blank) funded, American post-WWII era, lacking the sense of a well educated child, researchers all the time?  (Some successfully, some not so much.)  Forgetting the blogosphere, peer review journals are filled with "letters to the editors" and responses to published studies challenging the conclusions of the primary researchers.  

So if you want to challenge my conclusions from G&K, do so on the basis of whether or not the data support them or not, and not that G&K were looking at other things.  In this regard:
  • G&K demonstrated absolutely no correlation between fasting insulin and weight change.  Therefore fasting insulin is not a factor in determining weight loss.
  • G&K demonstrated that controlling for calories and varying carbohydrate content of the diet resulted in weight maintenance or similar weight losses.  Calories at ad lib levels maintained weight  and calorie reduction resulted in weight loss irrespective of carb/fat composition of the diets.  Therefore, calorie deficit, not carbohydrate content is the reason diets work when they do.

I have a post in the hopper addressing what G&K were looking at.  But I'm under no obligation to address that -- it's not a focus of mine because I do not subscribe to the insulin-centric theories on obesity.  However I would like to address why I think carbohydrate content of the diet can contribute to chronic hyperinsulinemia in the ALREADY OBESE & possibly IR.  

Still, I invite any and all of my cherry picker accusers to present me with a study, or data, that supports the insulin hypothesis.  I'm happy to discuss them in detail.  You won't get a "oh that's just in rats" or other such dismissive, and failure to address the data on merit from me.  For that kind of interesting exercise in deep thought you can go to the professionals.  ;-)

* International Society of Low Carb Fanatics

Apparently my attempts at humor and sarcasm are lost on the carbohydrate deprived.  Or maybe too much saturated fat causes osteopetrosis of the funny bone.


Sanjeev said…
another case where one needs to look deeper

(same link as the above clickable)
Alan said…

Just curious. In the dietary blogosphere. Which of the many, do you feel are worthwhile?
Anonymous said…
That was an interesting exchange in the comments section of the Weighty Matters article! FWIW, I think you (CS) came out ahead in the exchanges with Low Carb Liberal in the civility department.

One thing that was not mentioned was the boredom factor in diets (which also contributes to the difficulty of adopting a particular diet, long-term). If you restrict foods, you find yourself in a narrow range of 'allowable' foods and they eventually lose their appeal. You eat less of them, and if you are sticking to that narrow range of choices, it amounts to cutting calories.

William Bennet, M.D., focused on this years ago (published 1983) in his book, 'The Dieter's Dilemma.' Diets that restrict become monotonous. Stick to such a diet and you eat less.
CarbSane said…
It seems LCL preferred to debate only to a point, realized he/she was losing and then pick his/her toys up and go home. Only LCL returned to put words in my mouth as to points of agreement. Oh well ;-)

For some reason low carbers do not seem to like it when boredom or simply limited food choices are mentioned as possible contributing factors for why LC diets work. It sure does play at least a small role. Heck, look what happens to Jimmy Moore when he tries his limited foods plans. He's by far not the only one.

Paleo eliminates dairy - a huge source of easy calories if someone consumes a lot of cream and cheese. Same goes for anyone who just goes dairy free and loses weight.

The people who lose weight when they go wheat free? Same thing.

IMO, carbs are great vehicles for increased fat calories. I maintain that it would be difficult to get fat on a high carb low fat diet. Folks don't get obese eating dry toast, plain white rice, boiled potatoes, etc. They get fat eating buttered bread, fried rice, butter/cheese/sour cream with or fried potatoes, etc.

The only reason I personally care about the "why" is to prevent regaining with the least conscious effort. For me (and studies sure do seem to support this), it's the satiating power of protein.
CarbSane said…
Posting in parts for Todd Becker:

Hi CarbSane,

I would like to resume our debate about insulin and fat regulation. I see that you've written several posts with your interpretation of the Grey & Kipnis article (which you kindly credit me for bringing to your attention). However, I respectfully believe that you have misinterpreted G&K.

You state: "G&K demonstrated absolutely no correlation between fasting insulin and weight change. Therefore fasting insulin is not a factor in determining weight loss." I believe this overstates the facts.

1. The G&K study demonstrates that reducing carbohydrates will lower fasting insulin levels quickly. But sustained fat loss (not just "weight loss") may require not just instantaneously "lower" insulin levels, but achieving very low insulin basal levels, probably below about 8 uU/ml. As G&K note (p. 828), "Basal plasma insulin levels in normal-weight nondiabetic subjects in our laboratory range from 4 to 15 uU per milliliter (95 percent confidence limits). The obese subjects in this study exhibited significantly elevated basal plasma insulin values (22 to 32 uU per millilter)." Yet G&K's dietary interventions were very short term (only 3 weeks) and the insulin levels dropped from 38 +/-9 to 19 +/-3 uU/ml on the isocaloric "low CHO" diet (which was not all that low - at 25% CHO, i.e. more than 500 grams carbs per day. A favorable relative reduction, but 19 uU/ml insulin is likely too high for reliably achieving sustained fat loss.

**Note: although Todd's comments that he emailed to me did not contain HTML, he ran up against the HTML character limitations. I've no clue why :(
CarbSane said…
Part II from Todd Becker:

2. Your claim that there is "no correlation" between insulin levels and weight looks only at the instantaneous correlation. But fat loss may be a delayed response, as the Woodhouse paper I linked to you indicates. Figures 1 and 2 in G&K suggest that weight either dropped or plateau'd after a slight lag in the few cases where insulin levels approached 10 uU/ml. That the weight loss continued in some cases even after insulin levels begin to increase suggests that weight regain may be a delayed metabolic response.

3. The G&K protocol used "fasting insulin" as a surrogate for basal insulin. These are not the same thing. Fasting insulin was measured twice weekly at 7-8 a.m. after a 12 hour overnight "fast". But fat loss may depend on how much of the day insulin levels remain very low. Truly low basal insulin levels may take many weeks to establish.

4. Weight loss is an imperfect measure of fat loss. Woodhouse subjected non-obese but sedentary men to weight training, finding that "The early fall in serum insulin concentration was significantly correlated with the concomitant decrease in body fat, the increase in lean body weight and the age of the subjects. Body weight and reported dietary intake on the other hand, did not change significantly over this period." So the absence of weight loss may result from a combination of fat loss and increased lean mass. Thus, the subjects with lower insulin levels in the G&K study may have lost fat even if they didn’t lose weight.

I don’t deny that reducing total calories helps promote weight loss. But even this may also be correlated with an overall reduction in insulin levels. The isocaloric diet data in G&K's Figure 1 shows the insulin "spikes" often exceed 30-50 uU/ml, sometimes even higher than 70 uU/ml, whereas in the hypocaloric diets of Figure 2, even the insulin "spikes" were generally below 40 uU/ml, and never above 50 uU/ml.

In short, one can’t make a blanket statement that G&K demonstrates "absolutely no correlation" between insulin levels and fat loss. That study has several limitations: the subjects were obese, had elevated insulin levels, and the interventions may have too short for real body recomposition to occur. My own basal insulin levels took months of low carb, IF and HIIT exercise to drop from 15 uU/ml to 4 uU'ml - and the weight loss did not come immediately, but took time.

So we need to be careful when making generalizations about "correlations". Sometimes we must go beyond instantaneous relative changes to consider absolute thresholds, time delays, and other complexities.



** I'm a little backlogged at the moment, but will address these. In the meantime, I invite y'all to consider and respond!
CarbSane said…
@Alan: I hesitate to give a list because I read quite a variety of blogs - many I disagree with quite strongly, others I agree with almost entirely. I think they are all worth reading for various reasons - if for no other reason but to prevent trapping yourself within an internet echo chamber. WAAAAAY too much of that going on IMO.
Sanjeev said…
This comment has been removed by the author.
Sanjeev said…
this was in reference to exercised subjects: 4. ... the
TB > subjects with lower insulin levels in the G&K study
TB > may have lost fat even if they didn’t lose weight.

I don't think in G&K they started exercising when they had lower insulin and stopped when the insulin went up.

TB > 4. ... subjects with lower insulin levels in the G&K
TB > study may have lost fat even if they didn’t lose
TB > weight.

put this together with this:

TB > 1. ... G&K's dietary interventions were very short
TB > term (only 3 weeks)

(POINT A)and you get: fat loss and muscle growth in response to insulin reduction is quite quick

but hold on ... except that there should have been no response, as in:

TB > 3. ... fat loss may depend on how much of the day
TB > insulin levels remain very low. Truly low basal
TB > insulin levels may take many weeks to establish.

but returning to (POINT A),

TB > 2. ... weight loss continued in some cases even after
TB > insulin levels begin to increase suggests that weight
TB > regain may be a delayed metabolic response

So response to insulin is immediate when it suits the argument, and delayed when it suits the argument. You still have the cake you finished earlier.

TB > 1. ... sustained fat loss (not just "weight loss")
TB > may require not just instantaneously "lower" insulin
TB > levels, but achieving very low insulin basal levels,
TB > probably below about ... but 19 uU/ml insulin is
TB > likely too high for reliably achieving sustained fat
TB > loss.

from "may require ... very low insulin" to the confident sounding "likely too high". With no proof or substantiation for the given numbers, and no justification given for the change in language.

They measured fasting insulin, they measured weight loss. The 2 were unrelated. Writing "fasting insulin is not a factor in determining weight loss." sounds like a fair interpretation.

TB > Therefore fasting insulin is not a factor in
TB > determining weight loss." I believe this overstates
TB > the facts.

after this, until the end I read nothing showing the facts are overstated by that writing. The writing was not " ... determining FAT loss", so "they didn't measure fat loss" is a valid criticism of the study, but does not support

" this overstates the facts."

If one wanted to see a different study one might write "I want to see larger studies that measure actual fat loss" ... but that would be a different study, and would have a different fair synopsis.
CarbSane said…
You picked up on something I'm going to blog on separately here Sanjeev! It goes back to the original inspiration of the Insulin Wars series and James' posts. I think you picked up on the same having cake and eating it too thing that struck me :)
Anonymous said…
"I maintain that it would be difficult to get fat on a high carb low fat diet."

FWIW, the skinniest I've ever been (I won't say the healthiest) was in my late 20s when I spent about a year or so on a very low fat, high carb diet. I did not count calories, only kept track of keeping fat to about a tablespoon a day.
CarbSane said…
Hey Steph, I'm sure not advocating becoming a vegetarian or vegan, but I'm also not seeing massive numbers of overweight, let alone obese, folks who eat that way. I'm sure it's possible if someone is an ovo-lacto veg and eats a crapload of cheese, eggs and buttered/fatty bean casseroles, but that's not how most veggies I know eat. My cousin is an ovo-lacto and got a little pudgy following the birth of her child. She lost 30 lbs in a year by cutting dairy from her diet. The only animal protein she ate was the occasional egg (singular). I couldn't eat like that, but that's not the point. ;)

I used to do this 7 day diet thing in my 20's. Day 1 all fruit (no banana)
Day 2 all veg (non-starchy) + 1 baked potato with 1T butter
Day 3 all fruit & veg from days 1&2
Day 4 bananas & skim milk (I would do cottage cheese)
Days 5-6 non-starchy veg + meat/chicken/fish
Day 7 brown rice and fruit juice

Hmmm - 2 days of low carb out of 7.
Day 3
Anonymous said…
Oh, the infamous Cabbage Soup Diet! Snif. I'm all nostalgic now...
CarbSane said…
For Todd (Part I)

Carb Sane,

So far I’ve seen nothing from you that looks at differences between people with absolute basal insulin levels below about 10 uU/ml vs. those witih higher insulin levels (generally obese or insulin resistant individuals). To answer your points specifically:

1. The first plot you reference is not from Grey & Kipnis, but from a different study which tracks the 24 hour insulin profile for 3 overweight women on an identical 50% carb diet. So I don’t understand the relevance to the interpretation of Grey & Kipnis or low carbohydrate, insulin-lowering diets. In the plot your reference, insulin levels never drop below about 10-15. The diet consumed by these women is high enough in calories and carbs and frequent enough to keep insulin levels elevated for all but a very short time. My point is that the blue colored “fat loss zone” in James’ figure likely reflects an absolute insulin threshold, not a mere relative reduction in insulin levels. And the insulin levels do not remain low for very long. Especially noteworthy are the elevated insulin levels after dinner and through most of the night. So I don’t deny these 3 plots are similar – but I'm not sure I see how they are relevant to the question we are debating.

2. The CHO proportions in the LC and HC diets you reference from G&K apply only to the “hypocaloric” 1500 calorie diets, not the higher “isocaloric” diets, so I assume you are focusing on Figure 2. In all cases within that Figure, the basal insulin levels are relatively lower in the LC phases than the HC phases, but they never drop any lower than 18 uU/ml according to the statement on p. 829 (although from the Figure 2 graphic they appear to get as low as perhaps 10-12 uU/ml). So again, insulin levels remain above my proposed “efficient fat loss” threshold of 5-10 uU/ml. This is supported by G&K’s statement on p. 830: “Although significant changes were induced in basal plasma insulin levels by CHO restriction on both isocaloric and hypocaloric diets, the levels reached are still greater than those seen in normal-weight persons on ad lib diets.” Further, G&K acknowledge that reaching “normal” insulin levels would take more than 3 weeks: “It seems unlikely that substantial involution of beta-cell mass would occur in the dietary periods of three to four weeks used in this study…” In this context, they reference other studies where much longer periods of time were needed to reverse insulin resistance and bring absolute basal insulin levels into the normal range.

3. Now I’m sure you will object: “But Todd, how do you explain the steady weight loss of 0.75 to 2.0 kg/week in Figure 2, regardless of percent CHO in the diet, even while fasting insulin levels are above your proposed 10 uU/ml threshold? Shouldn’t you take this data for what it is, rather than starting from your own conclusion?” I will concede that you have a good point here. I cannot readily explain the weight loss (presumably fat loss) shown in Figure 2, in terms of the "fasting" insulin levels shown.
CarbSane said…
(Part II for Todd)

4. However, fasting insulin levels give only a momentary snapshot of the full temporal dimension of what I am calling "basal insulin". To understand fat accumulation and mobilization, we must look at not how low insulin levels get, but length of time insulin remains low enough to disinhibit fat loss. In James' plot this represented by the integrated area within the blue zones. What we know from Figure 2 of G&K is only what insulin levels were when measured upon waking in the morning (one or two mornings each week around 7 or 8 a.m.) In normal insulin-sensitive individuals, insulin levels plummet quickly overnight and usually give a low reading in the morning. In insulin resistant individuals, this function is impaired, and insulin levels come down more slowly; high fasting insulin in the morning may be reflective of insulin resistance. Insulin levels may drift lower for longer periods of time if the subjects are dieting -- eating small, low calorie meals. We don't know from G&K what the meal frequency or timing was, or how insulin levels varied through the 24-hour day.

What's important to consider is that one can eat a diet high in carbohydrate as a percentage, but if the size of the meal (calories) is low and the meals are spaced out sufficiently, insulin levels can still remain in the fat mobilizing zone long enough for weight loss. If the meals are small enough, insulin spiking or briefly high morning insulin readings become less and less relevant. In effect, smaller meals lead to more "intermittent fasting" time. Lyle McDonald wrote a nice piece that explains this in more detail:

...when you lower caloric intake, over a 24 hour period, the body will end up spending relatively more time in the post-absorptive (remember: body burning stored nutrients) than post-prandial (remember: body storing ingested nutrients) phase. This is simply a consequence of having less nutrients coming in relative to what’s being burned. On a diet, meals are smaller (or activity is higher, or both) so any given meal will only maintain an anabolic state for so long (and that time period will be shorter than if the person were eating more) before the body shifts back to burning stored nutrients. So even in the face of dietary carbohydrate intake, the body still will tap into stored fat; hence fat loss. I’d note that theoretically this might mean that eating less frequently would improve fat loss, since the body would spend more time between meals in the post-absorptive stage.
CarbSane said…
(Part III for Todd)

McDonald also cites some evidence to the effect that occasional brief insulin spikes may paradoxically induce basal insulin levels to come down to even lower levels for longer periods, than trying to maintain a low insulin level by avoiding carbs. Metabolism is often more complex than we at first suppose.

5. In short, I will agree with you that weight loss is evidently occurring with the low calorie diets of Figure 2, regardless of meal macronutrient composition, and despite moderately elevated "fasting" insulin levels. But, as McDonald's argument indicates, an instantaneous insulin level may be a poor indicator of the amount of time the body is in a catabolic state during the day. Smaller meals -- yes, lower calories per se -- may lead to weight loss, particularly if well spaced, allowing sustained lower insulin levels. To go back to the diurnal insulin plots of the 3 overweight women you cited, the insulin levels remain elevated for all but the shortest amounts of time.

Thus, the "low calorie" and "low insulin" explanations for weight loss are not necessarily in conflict, if one considers the entire diurnal insulin cycle, with an emphasis on how long and sustained the period of low insulin is. Low calorie diets, even if high in percent carbohydrates can be "low insulin" diets, if the meals are small enough and well spaced. The weight loss seen in Figure 2 of G&K is consistent with extended periods of low insulin levels -- even given elevated insulin levels in the morning and a meals that contain higher percentages of carbohydrate.

Sanjeev said…
> McDonald also cites some evidence to the effect
> that occasional brief insulin spikes may
> paradoxically induce basal insulin levels to
> come down to even lower levels for longer
> periods

Where do see anything about that being a long term effect?

I may have missed it but I see nothing there to indicate the effect lasts longer than the next time glucose goes up, forcing the usual insulin response; when discussing that (his words) "weird little study" Lyle does not write "permanently crashing" or "long term crashing/reducing/clamping/suppressing of insulin".
Sanjeev said…
> Thus, the "low calorie" and "low insulin"
> explanations for weight loss are not
> necessarily in conflict

They definitely, absolutely conflict: low carb and the corollary, low insulin is INSUFFICIENT for fat loss - indeed many are reporting (Lex Rooker, myself, Jimmy Moore ... unfortunately anecdotally) that in the absence of an insulinogenic diet one may still put on fat mass. The ASP studies support this: insulin NOT NEEDED to put on fat mass.

Further, a fatty meal suppresses HSL too. Insulin not needed to suppress fat mass loss.

An insulin-suppressing diet is UNNECESSARY to burn fat, as most of the Tour de France competitors scarfing down 6,000 calories of high fat and high sugar food during the day can attest - the most insulinogenic foods do not prevent them from burning off all the fat they can stuff down their gullets.

Same for ultra-distance cyclists (many did not buy into the low fat hysteria of the runners) - the Race Across America guys, for example, many of whom ate fat and starch, fat and starch, fat and starch. The most insulinogenic diet. And at the race's end they invariably have less fat mass than at the start.

One of the ASP studies discussed here showed this too: fat being released from fat mass in the face of rising insulin, and ASP later putting fat into the fat mass.

> I’ve seen nothing from you that looks at
> differences between people with absolute basal
> insulin levels below about 10 uU/ml vs. those
> witih higher insulin levels

If there's no compellingly good reason to believe the theory, and some clues that it's complete bunk, why will anyone spend who knows how much money to do this? And if they did it, you'll move the goal posts to another arbitrary number and demand proof again. It be so nice if everyone would jump through my hoops to satisfy my arbitrary musings where I put out meaningless, arbitrary, unjustified numbers.
Sanjeev said…
> They definitely, absolutely conflict:

No, they don't always conflict.

But if one is not needed and the other always is, why dwell on, obsess over, spend so much time and energy on the one that's not needed?
CarbSane said…
I agree with everything Sanjeev said in these last two comments. I think one of the critics of insulin theory said it best when they pointed out the pharmaceutical angle. If this is all there was to it, drugs do exist to lower insulin levels (or a partial beta-cell kill-off would seem far less invasive than GPB). No doubt they would be widely in use by now if insulin were the critical lynchpin.