A Matter of Control

In advance of my next installment of Nina Tei¢holz, Shai'ster, wherein I will discuss her claims that the Shai/DIRECT study was "rigorously controlled", I thought I'd give this post a bump.

Original publish date Dec. 10, 2011


It seems to me that many people misinterpret -- at least in their minds -- the meaning of this word when it is used in the scientific context.   While most of those no doubt understand the concept, after hearing the term enough, it just seems it comes to mean something else to them after a while.  I submit as evidence, statements made by two popular bloggers.  

First up, J. Stanton at gnolls.org with How “Heart-Healthy Whole Grains” Make Us Fat

Wait!!  Did you click that link already?  I forgot to caution you that the post you are about to read contains science, so you might want to proceed with caution.  < / sarcasm > .  Anyway, the study he discusses in that post is:  High Glycemic Index Foods, Overeating, and Obesity.  Stanton prefaces his discussion with the following:
Yes, I admit to a degree of hyperbole—but this study is so well instrumented and controlled, and its results so informative, that I believe it’s important for everyone to read it.
On an inpatient (24 hour) basis, the study involved feeding 12 obese teenage boys meals of varying GI's for breakfast and lunch, and then allowing them to request food and eat ad libitum any time after the lunch meal. It seems to me that by "well controlled", Stanton was referring to the control over food intake. Because, as it turns out, this study was not very well controlled from a scientific standpoint. 

Let's step back, for a moment, and discuss control.  I'll use a potential weight loss drug as an example.  I can conduct an experiment on its efficacy by going so far as to monitor all subjects taking the drug at rigidly enforced intervals while residing in a metabolic chamber for a month ... and yet such a study could not yet be called controlled.  Why?  Because control in a scientific sense, at the very least, compares the "treatment" to some standard, e.g. "no treatment".   The purpose of which is to separate out other factors that might be responsible for the result.  In this regard, not all controls are equal.  I could call my study controlled if I merely monitored an equal number of free-living people for the same period of time.  That would compare the weight loss impact of my intense intervention to doing nothing.  Controlled.  Of course nobody would consider this a well controlled study.  Why?  Because well controlled studies go the extra mile to control for potential confounders.

A confounder or confounding variable, is some variable (or factor) not being studied that could potentially influence the outcome.  Controlling for confounders involves keeping those variables as constant between groups as possible.  Using my example, we have several potential confounders including, but not limited to:  (a) placebo effect,  (b) awareness of being monitored, (c) access to food, (d) restricted activity, (e) isolation, (f) exposure to daylight.  Stuff like that.  And so, if I'm to study my weight loss drug in the metabolic ward scenario, the proper control would be to submit a similar group of subjects to the exact same overall procedure and altering just the treatment -- e.g. they get a placebo pill instead of the drug.  

So back to Stanton's study.  Is it well instrumented?  I would tend to agree.  Is it well controlled?  NO.  Why?  Well they altered not only the GI value of the study meals, but the glycemic load, the protein content, and the food type (whole/solid v. processed/liquid).  The meals are shown below:

The low GI meal contains almost twice the protein as the medium and high GI meals.  This is significant enough that the results are meaningless to me (except to further demonstrate the satiating power of protein).  And I don't know about you guys, but I don't tend to chew oatmeal all that much when I eat it (certainly not the instant variety).  At the very least they should have made a (disgusting?) smoothie/porridge of the low GI meal.  I would also point out that the double asterisk on the milk in the high GI diet refers to the fact that the milk was treated with lactase -- IOW, it was artificially higher in simple sugars.  I'm no big fan of instant oatmeal, but the mid-to-high GI comparison is a bit unfair to the poor instant oat as it is handicapped further by being paired with the treated milk.  

Bottom line, this study fails the smell test for control.  Any researcher looking at satiety and eating behavior should be aware of the well documented satiating quality of protein.  That they were looking at satiety and failed to control for protein, is a scientifically fatal flaw in the study design.  The use of differing food types (solid vs. essentially liquid), and an unnatural food (lactase treated milk, not consumed by most) further compound this error, and I submit these researchers should have at least taken this into consideration.  Speaking of oatmeal though, did proceeds from WheatBelly fund this study?  ;-)  It should be noted that this study is relied heavily upon by its principal researcher -- Dr. David Ludwig -- and numerous others to support the concept of glycemic index in weight management.  It's a shame the shoddy study design is not challenged every time it is held up as some sort of proof.

Next up, we have "Diet Doctor" Andreas Eenfeldt touting yet another wonderful study showing the superiority of low carb diets.    

In response to this study presented at a meeting, Eenfeldt gushes:
A new study of the highest quality (RCT) shows that low carb gives more weight loss than a Mediterranean diet as well.
So, based on a very sketchy press release of a yet-to-be published study, Andreas has concluded that the study was of the "highest quality". An RCT stands for a Randomized Controlled Trial.  The definition from  medterms.com of an RCT is:
A study in which people are allocated at random (by chance alone) to receive one of several clinical interventions. One of these interventions is the standard of comparison or control. The control may be a standard practice, a placebo ("sugar pill"), or no intervention at all.
OK, so as long as I randomly assigned subjects to my weight loss drug study discussed earlier, it would qualify.  That is, I take half of them and confine them not just to a metabolic ward, but a metabolic chamber, and administer my drug, and the other half receives no intervention at all -- their "standard" behavior.  Would you think such a study was of the "highest quality"?  I hope not!  In the study in question, the intervention was different diets.  The description in the press release is confusing, but it sounds like they used a conventional Mediterranean-style CRD every day as the standard.  The LC study groups followed that diet for 5 days/week and for two days a week they either ate a restricted LC diet or an ad libitum LC diet.  They did this for four months.   Sounds like an interesting design if I'm reading correctly and they are essentially controlling for intake (to some degree) between the LC groups, and controlling for carb intake (2 days per week)  vs. steady macro intake.  Nice.  I'm going to be blogging on this one soon.  Tis amazing how many studies low carbers put up to boast over the superiority of ad libitum high fat low carbing that show otherwise.  But back to the point.

Just because this study appears to be well controlled for comparing the diets, and is randomized, does not necessarily garner it the label "of the highest quality".  Remember that study where they used predicted weight loss to compare diets?  Or how about those studies that include subjects who drop out in the final analysis?   Or the studies where they don't verify compliance even by self-report?  Or how about if the Mediterranean diet was not of the proper composition, or the LC diet not truly LC?  (BTW, it was not VLCHF, or did Eenfeldt miss the part about lean meats?).  We have no information on drop out rates, compliance, verification, etc.etc.etc.  (Indeed we don't even have a published article to assess, but Eenfeldt is already convinced because he thinks this study shows LC superior to all other diets.)  In other words, "controlled" does not make a study of higher quality by default.  Randomized doesn't necessarily either (also perhaps a topic for another day), but in a nutshell, there are instances when randomization does not produce a better study.

So ... on that matter of control.

Remember what it means.  Don't mistake control over implementation/compliance with controlling for variables.  And don't mistake controlling for variables with control over implementation/compliance.  Lastly, the best studies are controlled squared:  well controlled for as many possible confounders as one can think of, and as well controlled in implementation/compliance as practical.


Amy said…
As article readers we lack information to fully understand science. I saw this report on a study that low carbing 2 days a week caused better weight loss for women than women eating a 1500 calorie a day diet. But, I don't have much information on the study other than that. I have no idea how much protein each group was eating and I sample sized of 88 is pretty weak.
Sue said…
Amy, I think it was over 4 months and the weight loss was piddly for all though LC arm more weight loss. Also the 2 days of LC was at 650 calories per day. So could have just being lower calories causing weight loss.
CarbSane said…
Thanks Amy, that's the same study but we get a little more info from the article.
Anonymous said…
'The effects of intermittent or continuous energy restriction on weight loss and metabolic disease risk markers: a randomized trial in young overweight women.' preceded that (epub, Oct. 2010)

'Energy restriction and the prevention of breast cancer.'
looks to me like it could be the study mentioned in the article.

Unfortunately, it's not free to read the study's text, so I only see the abstract.

What's interesting to me is that neither of these two studies looks like it is about low-carb diets. Reading the abstracts, the objective seems to be on the value of intermittent dieting (energy restriction) and the second uses 'intermittent energy restriction' and 'intermittent fasting.' The effort to diet relates to the goal of losing weight as that ties in to breast cancer prevention.

If you read the abstract of the first, earlier study, it is only with digging into the pdf that you find the 'very low calorie diet' was very VERY low in calories (75% restriction of baseline requirements, while the constant Mediterranean diet had just 25% restriction of baseline requirements). The very VERY low calorie diet was also low carb - and it was a two-day stint out of a week during which the participants consumed only their weight maintenance caloric requirements.

So could they have lost weight because it was calorie reduction? Gee, ya think?!

The low-carb VERY low calorie group reported that they ate fewer calories. With just two days of really hunkering down, they reported greater reductions of caloric intake and they were 'on a diet,' albeit a more conventional diet, the rest of the week.

The low-carb group (remember, it was actually the VERY very low calorie group) used the appetite-satiating high-protein advantage, but the dieters only had to deal with that for two days, and ate more the other calorie-restricting days of the week, but still calorie-restricted.

Almost intermittent fasting, but not quite, if you are going by the convetional meaning of 'fasting.'

Interesting to me was the fact that 9% of the people eligible for the study declined to participate because they didn't think they could handle 6 months of this. I mean, come on - it's a diet! They're no fools. They knew dieting is dieting.

Also interesting: the constant dieters, not the low-carb group, did not report problems with hunger and had no adverse effects, such as headaches, fatigue, etc. The low-carb group did. Also, they had much more trouble fitting that diet into their daily routine. Yet, they only had to do it for two days!

Unknown said…
Comparison of Weight-Loss Diets with Different Compositions of Fat, Protein, and Carbohydrates

We randomly assigned 811 overweight adults to one of four diets; the targeted percentages of energy derived from fat, protein, and carbohydrates in the four diets were 20, 15, and 65%; 20, 25, and 55%; 40, 15, and 45%; and 40, 25, and 35%. The diets consisted of similar foods and met guidelines for cardiovascular health. The participants were offered group and individual instructional sessions for 2 years. The primary outcome was the change in body weight after 2 years in two-by-two factorial comparisons of low fat versus high fat and average protein versus high protein and in the comparison of highest and lowest carbohydrate content.

At 6 months, participants assigned to each diet had lost an average of 6 kg, which represented 7% of their initial weight; they began to regain weight after 12 months. By 2 years, weight loss remained similar in those who were assigned to a diet with 15% protein and those assigned to a diet with 25% protein (3.0 and 3.6 kg, respectively); in those assigned to a diet with 20% fat and those assigned to a diet with 40% fat (3.3 kg for both groups); and in those assigned to a diet with 65% carbohydrates and those assigned to a diet with 35% carbohydrates (2.9 and 3.4 kg, respectively) (P>0.20 for all comparisons). Among the 80% of participants who completed the trial, the average weight loss was 4 kg; 14 to 15% of the participants had a reduction of at least 10% of their initial body weight. Satiety, hunger, satisfaction with the diet, and attendance at group sessions were similar for all diets; attendance was strongly associated with weight loss (0.2 kg per session attended). The diets improved lipid-related risk factors and fasting insulin levels.

Reduced-calorie diets result in clinically meaningful weight loss regardless of which macronutrients they emphasize.

All the diets reduced risk factors for cardiovascular disease and diabetes at 6 months and 2 years (Table 2). At 2 years, the two low-fat diets and the highest-carbohydrate diet decreased low-density lipoprotein cholesterol levels more than did the high-fat diets or the lowest-carbohydrate diet (low-fat vs. high-fat, 5% vs. 1% [P = 0.001]; highest-carbohydrate vs. lowest-carbohydrate, 6% vs. 1% [P = 0.01]). The lowest-carbohydrate diet increased HDL cholesterol levels more than the highest-carbohydrate diet (9% vs. 6%, P = 0.02). All the diets decreased triglyceride levels similarly, by 12 to 17%. All the diets except the one with the highest carbohydrate content decreased fasting serum insulin levels by 6 to 12%; the decrease was larger with the high-protein diet than with the average-protein diet (10% vs. 4%, P = 0.07). Blood pressure decreased from baseline by 1 to 2 mm Hg, with no significant differences among the groups (P>0.59 for all comparisons). These changes in risk factors in the intention-to-treat population were about 30 to 40% smaller than the changes seen among participants who provided data at 2 years, reflecting the effect of the imputation of missing values (Table 1 in the Supplementary Appendix). The metabolic syndrome35 was present in 32% of the participants at baseline, and the percentage was lower at 2 years, ranging from 19 to 22% in the four diet groups (P = 0.81 for the four-way comparison).
Unknown said…
The full study


Experimental diets
Both diets involved a 25% energy restriction from estimated baseline energy requirements using reported METs x estimated basal metabolic rate (28).

The CER group were prescribed a daily 25% restriction based on a Mediterranean type diet (30% fat, 15% monounsaturated, 7% saturated fat, 7% polyunsaturated fatty acids, 45% low glycaemic load carbohydrate, and 25% protein) (29). The IER group were asked to undertake a VLCD (75% restriction) on 2 consecutive days and to consume estimated requirements for weight maintenance for the remaining 5 days according to the nutrient composition above. The VLCD provided 2060 to 2266 kJ of energy and 50 g protein/day and comprised 1.136 litres (2 pints) of semi skimmed milk, 4 portions of vegetables (~80 g/portion), 1 portion of fruit, a salty low calorie drink and a multivitamin and mineral supplement. Participants were advised to maintain their current activity levels throughout the trial, and did not receive specific exercise counselling. Energy prescriptions were reviewed throughout the trial to account for changes in weight and exercise levels to maintain a 25% restriction below estimated requirements for weight maintenance.

Diets were not provided to participants but were self selected using detailed individualised food portion lists, meal plans and recipes. To maximise compliance patients received fortnightly motivational phone calls and monthly clinic appointments where weight and anthropometrics were measured and reported back to patients. All subjects were encouraged to use cognitive behavioural techniques such as self monitoring, obtaining peer/family support and stimulus control to maintain diets

Breast cancer risk markers

Both groups experienced large reductions in serum leptin, decreases in the ratio of leptin: adiponectin, no changes in serum levels of testosterone, androstenedione and prolactin. The CER group had a greater reduction in DHEAS compared to IER (mean difference [95% CI] CER vs. IER −6 [−14 to 1] %, P=0.08) however both groups experienced comparable increases in SHBG and a decrease in FAI (Table 5). Menstrual cycle data was available for 44 IER (83%) and 47 CER (87%). During the 6 month study period the mean (±SD) length of menstrual cycle was significantly longer in the IER group compared to the CER group (29.7 [±3.8] vs. 27.4 [±2.7] days, P=0.002).

Cardiovascular risk markers

Both diets led to comparable reductions in total and LDL cholesterol, triglycerides, systolic and diastolic BP. Neither group experienced changes in HDL levels

T said…
Aren't you the guy who was recently seen badmouthing Evelyn on other sites? What are you, son, mental?
CarbSane said…
I don't get it either. I post a science post and Charles posts links to paleodrama in the comments section. Then he trashes me for too many drama posts. Then he posts more here as if nothing happened.

There are a number of muck rakers out there who post different things/ways in different places. Charles is the most confusing of them all because he seems to have a genuine interest in the science
Jane said…
I remember that post of Stanton's. I thought, those obese boys look to me like they have reactive hypoglycemia in response to carbs but not to protein. Their beta cells would have normal sensitivity to protein but not to glucose. This is apparently what happens when they start to malfunction.
MacSmiley said…
Thank you for the clarification, Evelyn. Context is everything. The word does not mean what Teicholz thinks its means. A controlled study does not automatically mean a good study.