Nina Tei¢holz, Shai'ster ~ Part III: Well Implemented Clinical Trials

Shyster [ shīs-tər ]:  A person, especially a lawyer, who uses unscrupulous, fraudulent, or deceptive methods in business.

Shai'ster [ shī-stər ]:  A person, especially a science journalist, who is unscrupulous, fraudulent or deceptive in their representation of the Shai clinical trial.

---

Part I:  The Diets of Shai et.al. and Extrapolating from Weight Loss Studies

Part II: Applicability of RCTs to the General Population

Related posts:  A Matter of Control, Control in Clinical Trials

DISCLAIMER:  What follows is in no way intended to be a review or analysis of the findings of the LA Veterans Study or the role of saturated vs. polyunsaturated fat in heart disease.  The purpose of this post is solely to discuss the quality and nature of the implementation of two research studies, as described by Teicholz in her book and interviews vs. how they really were.

---

A running theme in The Big Fat Surprise, and in practically every one of the various and several interviews with author Nina Teicholz, is this: 

1. Studies upon which the Diet-Heart hypothesis was advanced = Riddled with Methodological Problems 
2. Studies funded by the Atkins Foundation in the past decade = Gold Standard Well Controlled Paragons of RCTs 

The "poster study" for #1 is the LA Veterans Study: A Controlled Clinical Trial of a Diet High in Unsaturated Fat in Preventing Complications of Atherosclerosis. Dayton, et.al. 1969.

The "poster study" for #2 is the Israeli DIRECT Study: Weight Loss with a Low-Carbohydrate, Mediterranean, or Low-Fat Diet.  Shai, et.al. 2008.

Let's start out with a compilation of quotes about these two studies made by Teicholz in a podcast she did with Sam Feltham (link to full interview)

I have another montage of her repeating much the same for these two studies in several interviews, but just don't have the time to do it justice and make for the proper attributions, etc. I will address the smoking issue in Part IV, but for now point out that she likely mentions this first so as to rile up her audience. You can hear audible gasps and/or see the rolling eyes/sneers of her interviewers in various clips -- after hearing how stupid the researchers were not to control for smoking (the story is FAR more complicated than that as you'll soon see) -- the audience is primed to believe anything negative about the study. Then she hits you with the nonsense about diet compliance.

The Anti-Sat Fat Research Was Very Very Bad!!

In Chapter 4, The Flawed Science of Saturated versus Polyunsaturated Fats,  Teicholz begins the chapter by explaining a bit about how nutrition research is conducted:

"Clinical nutrition trials are controlled experiments,"  {1295}

Well, that's not too bad, but she's already getting herself in trouble with that control word.  Many nutrition trials, especially pilot studies, and a lot of metabolic ward studies, are uncontrolled experiments.   For example, the Frassetto paleo study was uncontrolled.  They just put 9 people on their diet and measured various biomarkers before and after, we don't know if simply having all those tests run wasn't responsible for changes observed.   Another example might be the somewhat infamous (around here anyway) Grey & Kipnis study.   Nit picking?  

In the best (most “well-controlled”) trials, researchers prepare or provide food to study participants to control exactly what they eat. Sometimes subjects are invited to dine in a special cafeteria, or sometimes the researchers will go so far as to deliver meals to their subjects ’ homes— although these kinds of measures can be quite expensive. In less well-controlled trials, subjects are simply counseled about what to eat and perhaps given a diet book to take home.  {1297}

Sorry Nina, but the C in RCT has nothing to do with how the diets are administered to subjects.  She goes on to describe the use of a control group and randomizing ... oh so ironic given the subject of the upcoming Part IV.   The chapter then discusses four studies the AHA apparently uses as talking points for their various recommendations, one of which is Dayton/LAV.   I do believe I have found an actual error in the book!  She describes this as a six year study when it lasted eight.   

In her rough description of the study, we're told that nearly 850 men living in a VA home were broken into two groups where one group had oils replacing saturated fats and the control group ate "regular foods".  As a result cholesterol dropped in the experimental group, and that group also experienced fewer heart related deaths, although total death rates were the same.  Teicholz makes note of the fact that almost twice as many died of cancer vs. controls, which Dayton addressed in a later article.   But the scientists were elated at the heart disease results and that's all they really cared about.  Then the "debunking" begins:

A number of European scientists were more skeptical, and the editors of Britain’s oldest and most prestigious medical journal, The Lancet, wrote a withering critique.  They cited such problems as the rate of heavy smoking being twice as high among the controls as it was in the experimental group^I and that people on the special diet ate only about half their food in the hospital (nothing was known about the food they ate outside).  Moreover, as even Dayton admitted, only half the men in the experimental group stayed on the diet successfully during the six years of the study. The results were also skewed because there was a tendency for men who got well to leave the VA center and be lost to the trial. Dayton defended his study in a letter to The Lancet, standing squarely by his conclusion that a “prudent diet” could lower heart disease risk. And “LA Veterans” has since frequently been cited as evidence for that point, even as the original controversy surrounding the trial has been forgotten.   {1360}

OK ... before I go on, I have a bone to pick with this "science journalist".  The above paragraph would have earned me an F if submitted as a high school term paper.  The referencing is ridiculous.   

Footnote I:  Dayton wrote a reply in The Lancet, in which he analyzes the smoking data and, based on a number of assumptions, asserts that it had “no net effect whatsoever” on the outcome of the trial (Dayton and Pearce 1970).    {1786}

Would it have been too difficult to give the proper reference for all of these papers within the chapter?  The further dig at Dayton in the end-of-chapter footnote still leaves the reader having to search the unnumbered notes at the end of the book.  It's almost as if Teicholz doesn't want people to look at her references!  Sigh.

Teicholz has grossly mischaracterized both the nature of this study and the critique of it.  I'd hardly call it scathing in that it merely lists some of the limitations in terms of how it can be interpreted.  Again, leaving the smoking issue for another day, Teicholz gives the reader the impression that Dayton/LAV was some shoddy, haphazardly executed clinical trial not even worthy of being called one!  In her run-up she stresses that to her mind, a "well controlled" study is one in which great care is taken in administering the diets.

... people on the special diet ate only about half their food in the hospital (nothing was known about the food they ate outside).  Moreover, as even Dayton admitted, only half the men in the experimental group stayed on the diet successfully during the six [there's that error again!] years of the study.

There are some serious issues with how this reads.  Though it wouldn't be correct, it is structured in such a way as to imply that only a quarter of the men followed the proper diet (½*½) or only a quarter of the food was likely "special".  So at least now I see where she gets this "they had no idea what the men were eating half the time" mantra -- which, by the way, is exactly the game-of-telephone style characterizing she claims to decry!  Lancet wrote:

... Diet adherence was also poor: 56% of the total possible meals for the control group and 49% for those on the experimental diet. These figures are exaggerated by counting the absences from the Center of some men who later returned ...

... Secondly, more than 50% of the total food eaten by the men in the diet group was not the special diet, and little is known of the composition of the other food eaten. ...

I think it was irresponsible of the Lancet editors to write what they did.  They presume facts not in evidence so to speak, and I'll come back to this point later.

Implementation for Compliance in the LA Veterans Study

I wish I had the time to put the montage of audio/video clips together here, as it is astounding the way she repeats the second snippet from Lancet as if it were an adequate portrayal of how Dayton/LAV was implemented.   Fraught with methodological errors!!   How can she say this with straight face and cool frank demeanor?  Ignorance is not an option here, because she has been at this for 9-10 years, although she does still seem grossly ignorant of the terminology.  Still, one gets the impression that she didn't read the actual study, only the Lancet critique, and that would be unacceptable!  I know the PDF of the original publication is 63 pages long in Dayton, but ... ten years is plenty of time to read what it is she's telling her audience about.

This study, as conducted, was truly of impeccable quality, I don't think it is even feasible to match nowadays.  They go into considerable detail regarding how they decided to do the study.  This was a large -- n > 400 per group -- double blinded, randomized controlled trial that lasted for eight years.  It was also well implemented.   The paper goes into considerable detail regarding food preparation and service, etc.

Faced with these considerations and working in an institution which included a 2,600-bed veterans' domicile, we began in 1957 to consider the desirability of carrying out a field trial in this Center. Certain advantages of this institution commended it for this purpose. The length of stay among domiciled men was relatively long. The situation was favorable for serving controlled meals in which the characteristics and composition could be monitored.  It would also be possible to monitor adherence of the subjects to the diet. It appeared too that we would be able to randomize a study population into control and experimental groups and apply to this nutritional problem the double-blind technique which had been so effectively employed in drug trials.

The chief disadvantages were that the potential participants were relatively old -and therefore perhaps beyond the postulated point of preventability- and that normal institutional turnover, even in this chronic-care facility, would result in a disturbingly high loss of subjects for follow-up. However, the opportunity for carrying out a trial under relatively rigorous control seemed so attractive as to override these disadvantages.

The paper goes on to discuss in rather great detail considerations of the diet and how it would be implemented.  Specifically that "most of the subjects were not highly motivated toward self-deprivation" thus necessitating a highly palatable diet, and one that did not restrict fat.  However since this was not a weight loss or otherwise restrictive regime, there was "the incidental advantage that if it were ultimately found to be of value, it might be adopted by the general public more readily than would a diet requiring gastronomic sacrifice."

Most persons in the Domicile and all subjects in this study ate in a single large facility with one kitchen and two dining rooms. The food was served ad libitum, cafeteria style, with the two groups of subjects separated physically from each other and from men not on the study.  To minimize errors in food preparation and service, cooking and serving containers were color-coded.

Since the participants had access to the community as well as to a canteen, total adherence to the study diet was the exception.  Because rigid adherence could not be enforced, the level of adherence was monitored. Each man in the Home, including those not on the study, was issued a meal ticket of distinctive color which was punched at each meal. The punched ticket served as a record of meal attendance. This system also assured that each man remained on his assigned diet. All participants in the study were personally known to the dietitians and food service workers, who were thus further able to minimize deviations from prescribed diet within the dining room.

So total adherence was the exception.  How does Teicholz go from that, and rigid adherence could not be enforced to portraying this as if these men were strolling in at their leisure to eat their meals and nobody had a clue half the time what they were eating?    She got that from the Lancet editors, who misrepresented what was said in the paper and added their own suppositions to the mix.   Here's what the original study had to say:

Adherence data here and elsewhere in the report are given in terms of percentage of maximal possible number of meals taken in the dining room. When this is computed only for periods of residence in the Domicile, and only up to the time of withdrawal for those persons who withdrew, adherence was 80% for the control subjects and 78% for the experimental subjects. When adherence was computed for each individual from assignment to the study diet up to the date of death or termination of the study, whichever came first, disregarding the subject's location or status in the interval, mean adherence levels were 56% for the control subjects and 49% for the experimental group.

click to enlarge

There was considerable attrition in this study, but also several subjects who were discharged and re-admitted to the facility.   While this limitation was acknowledged going in,  losing subjects due to them being discharged is hardly the fault of the researchers.  Nor can it be "blamed" for non-adherence!

The red-bolded statement, combined with the blue bolded definition of how adherence was calculated, would indicate that the subjects that were "active participants" in the trial ate around 80% of the meals that were served at the facility.    Those figures closer to 50% come from including all the meals "missed" by subjects who were no longer residing at the facility!  It is a legitimate bone to pick as to whether these should have been included in the analysis, etc., but this doesn't imply that the implementation of the dietary protocol was lax.

Let me ask each of you this -- did you eat 21 meals last week?  What?  You slept in and skipped breakfast on Sunday, worked through lunch on Tuesday, ate a lot at lunch Wednesday so you skipped dinner that day, didn't feel well on Friday and only had a small dinner?  You are not adhering to a normal diet!   As freshman we all had mandatory meal plans in college.  The dining hall was open for breakfast, lunch and dinner every single day.   The guys I used to hang with would frequently oversleep the breakfast shift, and rather than spend beer money on food, they would simply wait for lunch and eat well and/or grab extras for later.  LAV wasn't a restricted diet, there would have been nothing stopping these men from doing that.  We're talking retired vets, some quite elderly, some quite ill, in a public home and not likely to have excessive spending money.  How often do you suppose these men were dining out in LA or buying food in the canteen rather than waiting for the next meal?   My college buddies probably only had 67% compliance with their meal plan, but more like 90% of their food was eaten in the dining hall.  This would make allowance for parties and 2am Dominoes deliveries, which I'm gathering the vets living in a chronic care facility in the 60's didn't really have much opportunity for.

My point being that is a huge stretch to go from "the men consumed around 80% of the potential meals provided" to "20% of the total food eaten was not the special diet".    This is what the Lancet editors did, and they did it with the 50% figure that included participants who weren't fully active in the study.   It would be far less of a stretch to presume that if they averaged 80% of all meals provided, they ate pretty close to all of their food from the special diet.  But Teicholz seized on this and repeated it as if fact, never mentions the extensive "control" that was implemented.  I'm going to call foul here.

One final point as regards "cheating" in this study.  Earlier in her description of the diets, Teicholz describes the control group as eating "normal foods".  Therefore anyone on that diet was basically adhering to their protocol regardless of their food source.  The control diet only differed from standard institutional fare in order that they could match the foods for palatability, etc.  If the men on the experimental diet didn't adhere, this would have simply lessened the severity of the "treatment" and perhaps reduced the effect observed.

Every meal eaten in the dining facility was in 100% compliance with the prescribed experimental diets.  

Really Bad Science vs. Well Controlled Unequivocal Amazing Definitive Research

Teicholz likes to use the plural of words a lot, to imply that there really is some group of studies out there from which she has plucked the one that best epitomizes her descriptions ... but there are many more where that came from!    And yet her talking points on this issue in terms of the really *good research* from the past decade boil down to one study.  Shai.  It is the only two year study mentioned in the main text of the book.

Shai is "well controlled" because instead of handing the subjects a diet book, they actually served these people in a cafeteria!   Imagine that!!  How is it Dayton/LAV is maligned as so poorly conducted then?  Your guess is as good as mine, unless Teicholz only read the Lancet critique .....

Below I've summarized the studies in terms of the basic study design and dietary intervention:

When constructing this table,  I was going to use the diet descriptions for Shai in the 2009 follow-on paper discussing adherence.  However these differed from the descriptions in the original paper used in the table.  The differences are that in the 2009 paper:  LF -- aim was 20-30% fat and 7-10% sat fat , Mediterranean -- aim was 40% fat mostly olive and nuts, LC -- aim was 100g carb/day max after 2 month induction.   

By Dayton's standard for adherence, the subjects in Shai only ate "study foods" around 25% of the time ... and we have no idea what they ate the other 75% of the time!!!    But worse than that, there is no indication whatsoever that the participants selected appropriate foods for their diet, none, or that they may have supplemented or substituted food brought from home.    Unlike the veterans, these folks ate in the company cafeteria alongside everyone else, with their usual friends and colleagues, etc.  The food choices were merely labeled for their convenience and there was no tracking whatsoever of what they chose.    One could argue that Dayton would have been better were the vets asked to record meals not eaten in the dining facility, but there's not a whole lot more they could have done.

The Shai trial?  Well, for starters, they could have had the subjects select from menus for a week and have them served only appropriate portions of compliant foods.  Did they at least record what they selected each day?  It doesn't appear so.    No ... in Shai we are to rely on self reported adherence and either four single-day food frequency questionnaires (selecting from a list and pictures of foods) or, for a subset, two 24-hour recalls.  The latter data, including any absolute values for the *reported* intake was not included in the original 2008 NEJM paper on this study.  They provided extensive information regarding "changes" in intakes and percents of total intake, but no absolute values.  Here is the intake determined by combining the data provided in the original paper and the 2009 Adherence paper.

What does this tell us about adherence?  Nothing certain, that's for sure, but we have no idea what these people were eating on these diets.  The differences between the FFQ data (original study, all participants) and the 24-hour recall (2009, about one-fifth of participants) is startling and demonstrates that neither of these methods was a particularly accurate representation of how much and of what they ate.   Further, the 24 hour recall data was also completed at the 1 month point with similar caloric intake reported.  The fact that on average the LC and LF subjects were gaining from 5 months onward yet still reporting considerable restriction is another red flag.

Lastly, as I noted below the comparison table, I had actually cut and pasted the diet descriptions from the 2009 paper into my table because I happened to have that paper open.  But then I noticed that the values differed from my memory of how they were stated in the original.  Sure enough, they did (and I used the original descriptions).  So I wonder if the low fat target was indeed 20-30% fat but they left off the "20 to" part when they realized the LFers reported eating slightly over 30% fat.  I wonder also if they didn't downgrade the 40% fat for the Meds when they were closer to 30.  The max carbs were increased to 120 grams because that was somewhat believable?

But as discussed in Part I, the dieters in this study didn't report complying with their assigned diets.  Especially the low carbers.  Couple this with the wildly inconsistent reports and the mis-match between reported intake and weight loss/gain, it's fair to say that this was a "poorly controlled" study on the Teicholz scale.   It's not quite as bad as handing-the-subjects-a-diet-book bad, but even that would be preferred if  the dieters then were required to weigh, measure and log their foods for several days at regular intervals to assess what they were eating.  While not individualized for quantity, the experimental diets in Dayton/LAV were actually analyzed for content directly or using tables.   The attendance at meals was a far better measure of adherence, because they knew they ate 100% compliant meals when they did.

Oh but wait, there's more!  Because (again you have to go to the 2009 paper on Shai to get this information)

.

When comparing the self-reported rating of extent of adherence during two years across the diet groups, the mean level dropped overall from 81% at the first month of intervention to 57% at 24 months, with no significant differences across the diet groups.  

Oh my ... that's not much better than what the Lancet chose to report about Dayton/LAV, only what the subjects consider adhering may not have been all that accurate as well.     Given that these were gainfully employed, relatively healthy, free-living (in likely private domicile, however many with spouses and/or children) middle aged people, how do you think this compares to those vets?  Even the meals served in the cafeteria weren't served in separate areas and they ate amongst their usual friends and colleagues.

It just boggles the mind that Teicholz has presented Shai as metabolic-wardly in nature, while dismissing Dayton/LAV as haphazardly administered.   She did this in the very same short interview with Feltham.  This is simply UNBELIEVABLE deceit.  As it turns out, Dayton was a superbly designed and implemented study according to the very guidelines she lays out in the book.  Yes, there was room for improvement, but it doesn't deserve the sneers Teicholz elicits in her interviews.  

A Note on Attrition

Attrition was quite significant in Dayton due to factors out of the control of investigators.  It may well be that without including those that withdrew or who were absent for a period between admissions the numbers weren't there to do the analyses.  But many of the "methodological" problems, weren't with the methodology at all.  Still, attrition impacted the results and their interpretation.  It is fair to question their analysis given this attrition, but not the design/implementation.

But there was attrition in Shai too.   I constructed a table from the flow chart in the original paper.   While the graphs in the initial paper would indicate monthly weighing, at each time point I included descending last major time point possible for weight.

Again, we get more detailed information in 2009 and learn that women had double the dropout rate (29%) of the men (14%) in this study.   They started with only 45 women (15, 20 and 10 women in LF, MED and LC respectively), which means 13 women did not complete the study.  Of the remaining 32 women that did, only 13 women were reportedly "successful" in losing more than 5% of their body weight.   How this might impact the outcomes depends on the distribution of these small numbers, and at the very least brings further doubt into the relevance of this study to the general population.  Incidentally, there's yet another paper out there from this trial discussing the success of their spousal engagement program.  What?  It turns out that they got the wives involved with a subset of the men in the study.  (Yes, to be blogged on at future date).   Gee, I wonder how that might have skewed things too?

They performed an intent-to-treat analysis on the primary outcome, weight.  This involved carrying forward the last measured value through all time points and thus including 104 or 109 values in their analyses at all time points.   Without the raw numbers, it is difficult to ascertain the impact on outcomes as well.

Some Conspiracy-Hypothesizing of My Own:

So the bottom line here is that Teicholz is basically lying to people, repeatedly, in her interviews and in the book.    Even IF, and that is a big if, the diets in Shai had been adhered to, the numbers are somewhat questionable due to attrition alone.  Couple that with the extremely iffy adherence, and it is clear that the label of "well controlled" (Teicholz meaning) is not deserved for Shai.

Given how obvious this is -- to anyone who read the methods section in Shai -- it becomes ever the more puzzling as to WHY this book was ever written and published and has such a huge marketing/publicity force behind it.

It would appear that most, if not all of this "good research" has been conducted by Atkins Nutritionals affiliated researchers (Volek & Phinney sit on the scientific advisory board) and/or with Atkins Foundation funding (Volek, Westman, Shai), not to mention by authors of the most recent Atkins book:  The New Atkins For a New You (2010, Westman, Phinney & Volek).  Between this and Chapter 10 essentially crediting Gary Taubes for single-handedly saving nutritional science from itself, with the help of copious quotations from and references to NuSI Advisory Board member Ronald "large LDL" Krauss, one wonders why this book was not entitled something like:

The Nu Atkins Revolution:  Forty Years and Still Fluffy!

Teicholz's delivery -- more polished and assured with each appearance -- has become a bit haunting.  As her "mistakes" mount, the probability that they could all be innocent or borne of ignorance has become infinitessimal.   For all its size and padding, TBFS is a mere shadow of its papa GCBC.  In Kindle locations, the text of her book numbers less than 50% of roughly 12K "locations", while the text of Taubes' tome makes up over 70% of about 13.5K.    Yet we are to believe this book's evolution began with an article on trans-fats a decade ago, consumed Teicholz life for the intervening decade, and has now come to fruition.

I'm not buying it.  It's no GCBC, especially as Gary saved her a lot of research by allowing her to duplicate many of his examples, etc.  Taubes only took 5 years to write his three theses.

No ... I'm guessing this got shelved at some point along the way, and sometime in 2012-13, it got exhumed and brought to the masses.   The book is so obviously biased as to be awkward if not downright embarrassing at times, and Teicholz has to go out on the publicity circuit and lie repeatedly.  She can't stop now.  What would motivate someone to do that?  I'm asking, because it makes no sense to me if there's no outside motivation here.  Ronald Krauss' name appears 91 times in the book.  He's got ties to the meat and dairy industries right?  Something isn't sitting right here ......

Oh ... and also, I don't think she did much if any of her own research for this book.  Because I'd like to believe that she's not a fundamentally dishonest person and that she believes she is telling the truth and telling this story accurately.    This would make sense -- the straight faces and unbelievably stupid "mistakes" (like the Hrdlicka thing) -- all of it would make sense in an innocent sort of way, if she trusted the sources of the information she's relating.   Then she's just a really horrible journalist and professionally inept.  Not the greatest thing to cop to, but I'm reminded how Taubes would prefer to be accused of scholarly incompetence than lying.

But this isn't 2007.  It's 2013 (the writing) and 2014 (the promoting) when GCBC has been all but totally discredited.  She has no doubt seen the criticisms but has chosen to label me a troll and a stalker on Twitter, rather than look into the issues raised.  I think she doesn't want to see what she'll find.

---

More on Nina Teicholz and The Big Fat Surprise:

• Ladies and Gentlemen I present .... The new Gary Taubes. Nina Tiecholz
• Book Review ~ Big Fat Surprise by Nina Teicholz
• No Big Surprise ... A Compendium of Errata Etc. from The Big Fat Surprise by Nina Teicholz