Nina Tei¢holz, Shai'ster ~ Part IV: Random Smoke and Mirrors

Shyster [ shīs-tər ]:  A person, especially a lawyer, who uses unscrupulous, fraudulent, or deceptive methods in business.

Shai'ster [ shī-stər ]:  A person, especially a science journalist, who is unscrupulous, fraudulent or deceptive in their representation of the Shai clinical trial.

---

Part I:  The Diets of Shai et.al. and Extrapolating from Weight Loss Studies

Part II: Applicability of RCTs to the General Population
Part III:  Well Implemented Clinical Trials

Related posts:  A Matter of Control, Control in Clinical Trials

DISCLAIMER:  What follows is in no way intended to be a review or analysis of the findings of the LA Veterans Study or the role of saturated vs. polyunsaturated fat in heart disease.  The purpose of this post is solely to discuss how randomizing and controlling was conducted in two studies, as described by Teicholz in her book and interviews vs. how it really happened.

---

To review from Part III:  A running theme in The Big Fat Surprise, and in practically every one of the various and several interviews with author Nina Teicholz, is this: 

1. Studies upon which the Diet-Heart hypothesis was advanced = Riddled with Methodological Problems 
2. Studies funded by the Atkins Foundation in the past decade = Gold Standard Well Controlled Paragons of RCTs 

The "poster study" for #1 is the LA Veterans Study: A Controlled Clinical Trial of a Diet High in Unsaturated Fat in Preventing Complications of Atherosclerosis. Dayton, et.al. 1969.

The "poster study" for #2 is the Israeli DIRECT Study: Weight Loss with a Low-Carbohydrate, Mediterranean, or Low-Fat Diet.  Shai, et.al. 2008.

---

The purpose of the Dayton/LAV study was to look the impact of fat quality (SaFA vs. PUFA) on cholesterol levels and heart disease.    In just about every interview Teicholz manages to get in the "in one study they didn't control for smoking" ... here's a little montage of three ...

Let's begin this discussion by looking further into what Teicholz says about controlling in clinical trials.  She gets the C in RCT routinely wrong, as discussed in Part III, confusing the control over implementation with controlling for experimental variables.  So imagine the big fat surprise it was to find that she mostly gets the R right, if not overstating the outcome a bit.  From the book:

If a large-enough study population is divided randomly into these two groups, they can theoretically be assumed to be the same in every relevant way. They should have the same age distribution, the same tendency to smoke or exercise, and be the same in a thousand other ways that researchers might never think to measure .....

... a clinical trial, by virtue of its very design, holds all these factors constant, regardless of whether the researchers have thought to account for them.    {Kindle Location 1301}

This is at the beginning of Chapter 4, The Flawed Science of Saturated versus Polyunsaturated Fats, wherein she goes on to discuss this problem of how they "failed to control for smoking" in the LA Vet study.  Did she write the beginning of the chapter in 2005 and the rest of it in 2013 or something?  Did she write the entire chapter?  Did ANYONE reviewing the draft manuscript not pick up on this inconsistency?    Later in the chapter discussing this ultimate of flawed studies:

A number of European scientists were more skeptical, and the editors of Britain’s oldest and most prestigious medical journal, The Lancet, wrote a withering critique. They cited such problems as the rate of heavy smoking being twice as high among the controls as it was in the experimental group I  ... {1360}

As an aside, that "withering critique" by these "skeptical scientists" opened with:

A CLOSE, if not directly causal, relationship exists between a raised serum-cholesterol level and the development of disease arising from atherosclerotic arterial degeneration. It seemed prudent, therefore, to explore the possibilities of lowering serum cholesterol levels by changes in diet both in advance of the development of overt signs of atherosclerosis and as part of the secondary preventive treatment of established disease.

I guess they weren't so skeptical of the relationship between serum cholesterol and heart disease ... something The Big Fat Surprise, not surprisingly, barely addresses.

To review, Dayton/LA Vet was a double-blinded randomized controlled trial.  After initial enrollment and screening, the 846 subjects were assigned to the two groups by simple randomization:  422 to the control diet (high saturated fat) and 424 to the experimental diet (high PUFA) which both mimicked the "standard American diet" at the time of 40% fat.

By Teicholz's own description, even if Dayton and his colleagues were total idiots and didn't think smoking was a factor worthy of consideration, the randomizing should have "controlled" for subject variability in smoking.   According to what Teicholz had just laid out in the book.   It should have, and it mostly did, but such are the breaks in the game of chance sometimes, especially when dealing with small numbers in a large group.  In a way, these researchers may have been victims of their own thoroughness rather than negligence in this regard.   

Did I mention the original paper is 63 pages long?  They presented detailed information on the subjects and presented copious data on those pages, 40 tables in all!  I've C&P'd and cropped a bit Tables 1 through 4, and included the full-page Table 5 just to give you an idea.   You can click to enlarge.

This kind of breakdown is included for cerebrovascular disease, medications, etc. ... including smoking.  There is the discussion on the methods of assessing smoking behavior and the outcome of the randomization along with the table of smoking frequencies:

Smoking habits were evaluated by questionnaire on the participants' entry into the study.  Through an oversight this information was not obtained from all subjects. Table 13 contains the responses to questions concerning current cigarette smoking at the time of entry.   A significant difference between the two groups is apparent in relation to the distribution of smoking habits.   However, the data in table 13 suggest that there was probably little or no difference with regard to the total amount of cigarette smoking.   Several other questions about smoking habits past and present revealed no other differences between the two groups; the responses are not tabulated in this report.

So ...  In her by-now-familiar fashion of leaving out any mention of the actual methodology, Teicholz leads to reader to believe they didn't bother to consider smoking.   This is bogus because it is specifically addressed and in rather great detail.  

If smoking had not been stratified and reported in such detail, it's quite possible the editorial board at The Lancet doesn't even mention it.   A lot has been made of the very heavy smokers -- over 2 packs per day -- and there being twice as many in the control group as the experimental group.  We're talking 20 men total here out of 846 ...  2.4% of the subjects ... 13 in one group and 7 in another ... just 6 subjects difference that's less than 1.5% of a study group.    Combined with to the 1-2 pack group combined there are 115 or around 14%.   There was a seriously heavy usage of cigarettes in this bunch where (of those 747 subjects reporting) only around 25% didn't smoke at all while 70% smoked on a daily basis.    Is there much difference between smoking 17 cigarettes a day vs. 23 a day?  I don't know.  But they did provide more detail than is provided in most studies that I've seen.  

In the Lancet editorial -- I'm not going to call it a critique as it was more a summary of the study and its strengths and weeknesses -- they wrote:

Randomisation proved effective [in producing groups of similar characteristics], except for the amount of cigarette smoking in the two groups.  Although the report does not stress it, the fact that 70 men in the control group, compared with 45 on the experimental diet, smoked more than one packet of cigarettes a day could be of great importance in interpreting their results.

So first of all, they specifically addressed what Teicholz herself discussed in the chapter opening ... that randomization should have effectively controlled for subject variability without the researchers specifically controlling for it in the design.    It is not a nit pick to point out that The Lancet cited stats for those smoking over a pack a day, giving hard numbers, where 2*45 = 90 not 70.  That two-times figure refers to the 2+ packs per day stratum.    So yes, The Lancet  cited issues with the distribution, but Teicholz got her figures from somewhere other than the editorial.  Sloppy, sloppy, copy, copy.

Far from "withering", The Lancet merely states that the researchers didn't stress the difference.  Teicholz would like the reader to think this meant that they didn't address it at all.  From the original Dayton paper:

Approximately 50 characteristics of each subject were recorded at the initial evaluation;  33 of these are shown in tables 1 through 14.  Because of the large number of characteristics evaluated, a difference in one of these (baldness) at the 97.5% confidence level and in another (distribution of cigarette smoking habits) at the 99% confidence level is entirely compatible with chance expectations in a fully randomized population. 

And later in their discussion they wrote:

The possibility that some variable other than diet accounted .for the outcome of the trial deserves careful consideration.  We have identified only one characteristic of the study groups which generates any suspicion of a difference in initial risk: cigarette smoking.  But any advantage of the experimental group resulting from differences in smoking habits must have been slight, judging from the data of table 13. [the table above]

I suppose this could be interpreted as giving the issue less than due consideration, but really the Lancet folks didn't say all that much more.  They said it wasn't stressed and that it could have factored into the outcome, perhaps moreso than the above paragraph might imply.  So then in the book, Teicholz tucks the fact that Dayton and colleagues did not just ignore the "critique".  They answered it in a letter of their own.   You find this in a footnote at the end of the chapter.

I: Dayton wrote a reply in The Lancet, in which he analyzes the smoking data and, based on a number of assumptions, asserts that it had “no net effect whatsoever” on the outcome of the trial (Dayton and Pearce 1970).   {1786}

This implies that Dayton essentially brushed them off, and given the way Teicholz writes and has portrayed the rest of the study, one is led to believe these assumptions are probably "cheating" in some manner.  Here are some excerpts from their response which began with ...

Sir,-- The leading article (Nov. 1, p. 939) in which you discussed our trial of a diet high in unsaturated fat made it clear that our report had not dealt adequately with at least one critical question. Specifically, your article suggested that the low incidence of atherosclerotic events in participants on the experimental diet might have been due to the chance inclusion of a smaller number of heavy cigarette smokers in that group than in the control group.  In order to satisfy ourselves and others on this point, we have undertaken further analysis of our results in relation to smoking habits. The results of this analysis, reported below, provide convincing evidence that differences in smoking habits could not have accounted for the favourable experience of subjects on the experimental diet.

So they did two additional analyses described in the letter.  First, they standardized the incidence rates per 100 man years and broke it all out by type of incident, etc.   This produced Table I below left (SD = sudden death, MI = myocardial infarction, CI = cerebral infarction).

As indicated in Table I, at any of the three levels of cigarette consumption examined, the incidence of clinical events attributable to atherosclerosis was lower in experimental subjects than in individuals on the control diet. Thus when cigarette consumption is the same, the effect of the experimental diet persists.

Perhaps expecting further criticism, they controlled for smoking a different way resulting in Table II above right.  They wrote:

Although Table I makes it clear that there was a dietary effect, whatever the inequalities of the smoking distribution, it is also desirable to determine whether the surplus of heavy smokers in the control group accounted, in part, for the more favourable experience of the experimental group.  Toward this end, we have developed estimates of "smoking adjusted" incidence rates-that is, estimates of the outcome which would have resulted if the subjects of each smoking stratum had been allocated in equal numbers to the control and experimental groups.

It is based on the outcome in Table II that they concluded, in full:

The resulting estimates, given in Table II, are nearly identical to the figures actually observed (bottom of Table I), the surplus of heavy smokers in the control group having been fully offset by an even larger surplus of moderate smokers (10-20 cigarettes per day) in the experimental group.  We conclude, therefore, that the uneven distribution of cigarette-smoking habits had no net effect whatsoever on the outcome of the trial.

Nina Teicholz goes out of her way to misrepresent these studies at seemingly every turn.  Remember, this is what she said:

I: Dayton wrote a reply in The Lancet, in which he analyzes the smoking data and, based on a number of assumptions, asserts that it had “no net effect whatsoever” on the outcome of the trial (Dayton and Pearce 1970).   

No, folks, what Dayton actually did do was a post hoc analysis, that I discussed in Control in Clinical Trials, where they controlled for smoking.   They didn't make a bunch of assumptions, they did statistical analyses.  I am not here to argue the quality of these or the conclusions reached.  The point is that it was done.  This is similar to how observational data is handled all the time where you isolate the effect of one variable by blocking and standardizing and comparing and seeing (a) if an effect persists when other variables are controlled for, and/or (b) doing a "what if" analysis and comparing that to the actual outcome.

At no point during Teicholz's "research" into this study was this 1970 information new.  As far as I can tell, there was no further challenge to this analysis, and if Teicholz's own "investigation" is any indication, any revisiting of these studies is a bunch of hot air.   Bottom line:

It is wrong for Teicholz to claim that smoking was not controlled for when interpreting the results of Dayton/LA Veterans.   

In retrospect a stratified randomizing strategy may have been more prudent, but between the randomization and the post hoc analysis it was controlled for in the full analysis.  There was one further publication that looked into the cancer occurrences (greater in the experimental group) in depth, and looked at smoking and the effect there as well.  To go into that one here would be overkill for this post, and I think I've made my point.  But I did want to put it "on the record" that there was still more, rather detailed, data and analysis on the cancer front.

You Couldn't Do That Kind of Trial Today .....

In the Amy Alkon interview from my clips, Teicholz seems besides herself a bit with how absurd it was for them not to control for smoking.    Forget all of the above for a bit let's look at Shai and smoking shall we?   Shai did not exclude smokers.  They used a stratified randomizing scheme where the strata were:

• Gender
• Age (above/below median)
• BMI (above/below median)
• Cardiac heart disease (Y/N)
• Diabetes (Y/N)
• Statins (none, < 1 yr , ≥ 1 yr)

... and I went back and double checked just to make sure I got that right.  So, although smokers made up 16% of the subjects at baseline, smoking wasn't specifically controlled for.  It looks like they lucked out with the allocation of the smokers between diets, but all we know there is a Yes/No ... as LA Veterans tells us, we could have an uneven distribution of degree of smoking, but that data is not provided and perhaps not even known.  

Should it have been?  Well, the argument could be made that since the primary outcome in this study was weight loss, perhaps not.  But since they were also assessing the health impact of these diets using metabolic markers influenced by smoking, it perhaps should have been.  They did randomize by CHD and diabetes and neither of those two conditions have much of an impact on weight loss per se.   I suppose an argument could be made that a history of either might impact the outcomes more that smoking might ... that's fair.   But if Shai is being portrayed as unequivocal "proof" of a dietary effect of fat, then let's drop the double standard here and insist that smoking be controlled for before we crown the Atkins diet as the best for every last ailment on the planet.  

Speaking of smoking though .....

... it DOES come up in the  2009 follow-on paper, that there was greater attrition in smokers at 25% vs. 14% for non-smokers (statistically significant, P < 0.04).   Unfortunately, in this rigorously-well-controlled-amazing-modern-day clinical trial, we were not provided with the breakdown for smoker attrition by diet-group.  We know that overall attrition was 10% LF, 15% MED and 22% LC ... if more smokers dropped out of the study, and more dieters dropped out of the LC group, perhaps more smokers dropped out of the LC group which could have impacted the outcome.  Of course this is pure *assumption* because .... well ... that's all we have to go by because we are not provided with the numbers (I do believe I'm repeating myself) or any indication that the authors even considered the effect this may have had on the outcome.  

The nice thing about electronic papers is that you can search them to make sure you didn't miss any discussion.    So I searched this paper for smok so as to find every reference to smoke(s), smoker(s) or smoking.  There is no analysis.  There were two cases where they "controlled for smoking" in a post hoc analysis to determine predictions for adherence.  It turned out that when controlling for smoking and other factors, higher initial BMI and lower weight loss at 6 months both remained predictors for dropping out.  Big whoop.  They also controlled for smoking and other factors and found that greater initial weight loss predicted successful completion.   Another huge snooze.   All that means is that if you lose a lot a quarter the way through that's all the more you can regain and still remain over 5% below baseline (the criteria for "success").  The weight change trajectories after 6 months were almost perfectly overlapped for the LC and LF diets.   

Other than reporting baseline demographics, smoking is not even mentioned in the original paper (that I'm convinced was the only one Teicholz read, if she did even that),  and the authors make no reference to how this may have impacted the outcomes of health measures.  Smoking can cause transient hyperglycemia (e.g. here) and effect blood pressure for example.  So IF any group lost more smokers this could have impacted these measures of metabolic health -- exactly how in an intent to treat analysis, is unclear.  They could have at least tabulated the drop-outs by diet assignment within the known confounders including heart disease and diabetes?   This is, after all, really really REALLY REALLY good clinical research.

Was this important?  Who knows, perhaps, perhaps not.  But without the numbers we don't know and therefore how can Teicholz say that this study shows anything??

But Wait!  There Was Some Heart Disease Analysis ...

I think this is the last of them, but there was yet another sub-study and publication to come out of DIRECT:   Dietary Intervention to Reverse Carotid Atherosclerosis.   For this they randomly selected 175 participants, out of which acceptably high quality diagnostic images were obtained for 140:  49 LF, 55 Med, 36 LC.  Adherence, demographics, etc, were all reportedly similar to the full cohort.   The primary outcome was carotid vessel wall volume (VWV) which correlates well with intima-media thickness (IMT)  but can give meaningful results over shorter time frames.

They concluded that regression occurred corresponding to weight loss and concurrent reductions in blood pressure, but there was no statistically significant difference between diets.  Therefore, it would be total folly to try to analyze this data.  Oddly enough, there were far more smokers in the lowest third of VWVs compared to the highest third.  This does not mean that smoking correlates with lower VWV, rather it may reflect that those with more serious heart disease may have heeded the warnings and quit smoking.   What does it all mean?  Again, who knows, and the supplemental material link is dead,  if it even held any clue.  

Concluding Thoughts

So in the DIRECT/Shai trial -- the one Teicholz has held up as supreme -- 

• If only 6% of the highest tertile smoked at baseline, and the smokers dropped out at higher rates?  Do we care?  
• If there were more than twice as many Mediterranean dieters than LCers in the lowest tertile (expected to see the least change) at baseline?  Matters?

Nah ... we don't really care about such minutia ... right?    Now 25 more heavy smokers out of 422 in the control group in Dayton.  That's a methodological nightmare.  Those were balanced out by 44 more moderate smokers in the experimental group?  Nah, can't go making assumptions.    Detailed accounting and controlling and adjusting to demonstrate that smoking didn't impact the results in Dayton?  Not good enough.  

But Shai ...

So I think I've laid out a solid case here again for the superiority of the Dayton/LA Vet study over Shai/DIRECT.  I mean there really is no contest.  This alone discredits Nina Teicholz and she really needs to be held to account to correct the record.