Nina Tei¢holz, Shai'ster ~ Part IV: Random Smoke and Mirrors

Shyster [ shīs-tər ]:  A person, especially a lawyer, who uses unscrupulous, fraudulent, or deceptive methods in business.
Shai'ster [ shī-stər ]:  A person, especially a science journalist, who is unscrupulous, fraudulent or deceptive in their representation of the Shai clinical trial.

DISCLAIMER:  What follows is in no way intended to be a review or analysis of the findings of the LA Veterans Study or the role of saturated vs. polyunsaturated fat in heart disease.  The purpose of this post is solely to discuss how randomizing and controlling was conducted in two studies, as described by Teicholz in her book and interviews vs. how it really happened.

To review from Part III:  A running theme in The Big Fat Surprise, and in practically every one of the various and several interviews with author Nina Teicholz, is this: 
  1. Studies upon which the Diet-Heart hypothesis was advanced = Riddled with Methodological Problems 
  2. Studies funded by the Atkins Foundation in the past decade = Gold Standard Well Controlled Paragons of RCTs 
The "poster study" for #1 is the LA Veterans Study: A Controlled Clinical Trial of a Diet High in Unsaturated Fat in Preventing Complications of Atherosclerosis. Dayton, 1969.

The "poster study" for #2 is the Israeli DIRECT Study: Weight Loss with a Low-Carbohydrate, Mediterranean, or Low-Fat Diet.  Shai, 2008.

The purpose of the Dayton/LAV study was to look the impact of fat quality (SaFA vs. PUFA) on cholesterol levels and heart disease.    In just about every interview Teicholz manages to get in the "in one study they didn't control for smoking" ... here's a little montage of three ...

Let's begin this discussion by looking further into what Teicholz says about controlling in clinical trials.  She gets the C in RCT routinely wrong, as discussed in Part III, confusing the control over implementation with controlling for experimental variables.  So imagine the big fat surprise it was to find that she mostly gets the R right, if not overstating the outcome a bit.  From the book:
If a large-enough study population is divided randomly into these two groups, they can theoretically be assumed to be the same in every relevant way. They should have the same age distribution, the same tendency to smoke or exercise, and be the same in a thousand other ways that researchers might never think to measure .....
... a clinical trial, by virtue of its very design, holds all these factors constant, regardless of whether the researchers have thought to account for them.    {Kindle Location 1301}
This is at the beginning of Chapter 4, The Flawed Science of Saturated versus Polyunsaturated Fats, wherein she goes on to discuss this problem of how they "failed to control for smoking" in the LA Vet study.  Did she write the beginning of the chapter in 2005 and the rest of it in 2013 or something?  Did she write the entire chapter?  Did ANYONE reviewing the draft manuscript not pick up on this inconsistency?    Later in the chapter discussing this ultimate of flawed studies:
A number of European scientists were more skeptical, and the editors of Britain’s oldest and most prestigious medical journal, The Lancet, wrote a withering critique. They cited such problems as the rate of heavy smoking being twice as high among the controls as it was in the experimental group I  ... {1360}
As an aside, that "withering critique" by these "skeptical scientists" opened with:
A CLOSE, if not directly causal, relationship exists between a raised serum-cholesterol level and the development of disease arising from atherosclerotic arterial degeneration. It seemed prudent, therefore, to explore the possibilities of lowering serum cholesterol levels by changes in diet both in advance of the development of overt signs of atherosclerosis and as part of the secondary preventive treatment of established disease.
I guess they weren't so skeptical of the relationship between serum cholesterol and heart disease ... something The Big Fat Surprise, not surprisingly, barely addresses.

To review, Dayton/LA Vet was a double-blinded randomized controlled trial.  After initial enrollment and screening, the 846 subjects were assigned to the two groups by simple randomization:  422 to the control diet (high saturated fat) and 424 to the experimental diet (high PUFA) which both mimicked the "standard American diet" at the time of 40% fat.

By Teicholz's own description, even if Dayton and his colleagues were total idiots and didn't think smoking was a factor worthy of consideration, the randomizing should have "controlled" for subject variability in smoking.   According to what Teicholz had just laid out in the book.   It should have, and it mostly did, but such are the breaks in the game of chance sometimes, especially when dealing with small numbers in a large group.  In a way, these researchers may have been victims of their own thoroughness rather than negligence in this regard.   

Did I mention the original paper is 63 pages long?  They presented detailed information on the subjects and presented copious data on those pages, 40 tables in all!  I've C&P'd and cropped a bit Tables 1 through 4, and included the full-page Table 5 just to give you an idea.   You can click to enlarge.

This kind of breakdown is included for cerebrovascular disease, medications, etc. ... including smoking.  There is the discussion on the methods of assessing smoking behavior and the outcome of the randomization along with the table of smoking frequencies:

Smoking habits were evaluated by questionnaire on the participants' entry into the study.  Through an oversight this information was not obtained from all subjects. Table 13 contains the responses to questions concerning current cigarette smoking at the time of entry.   A significant difference between the two groups is apparent in relation to the distribution of smoking habits.   However, the data in table 13 suggest that there was probably little or no difference with regard to the total amount of cigarette smoking.   Several other questions about smoking habits past and present revealed no other differences between the two groups; the responses are not tabulated in this report.
So ...  In her by-now-familiar fashion of leaving out any mention of the actual methodology, Teicholz leads to reader to believe they didn't bother to consider smoking.   This is bogus because it is specifically addressed and in rather great detail.  

If smoking had not been stratified and reported in such detail, it's quite possible the editorial board at The Lancet doesn't even mention it.   A lot has been made of the very heavy smokers -- over 2 packs per day -- and there being twice as many in the control group as the experimental group.  We're talking 20 men total here out of 846 ...  2.4% of the subjects ... 13 in one group and 7 in another ... just 6 subjects difference that's less than 1.5% of a study group.    Combined with to the 1-2 pack group combined there are 115 or around 14%.   There was a seriously heavy usage of cigarettes in this bunch where (of those 747 subjects reporting) only around 25% didn't smoke at all while 70% smoked on a daily basis.    Is there much difference between smoking 17 cigarettes a day vs. 23 a day?  I don't know.  But they did provide more detail than is provided in most studies that I've seen.  

In the Lancet editorial -- I'm not going to call it a critique as it was more a summary of the study and its strengths and weeknesses -- they wrote:
Randomisation proved effective [in producing groups of similar characteristics], except for the amount of cigarette smoking in the two groups.  Although the report does not stress it, the fact that 70 men in the control group, compared with 45 on the experimental diet, smoked more than one packet of cigarettes a day could be of great importance in interpreting their results.
So first of all, they specifically addressed what Teicholz herself discussed in the chapter opening ... that randomization should have effectively controlled for subject variability without the researchers specifically controlling for it in the design.    It is not a nit pick to point out that The Lancet cited stats for those smoking over a pack a day, giving hard numbers, where 2*45 = 90 not 70.  That two-times figure refers to the 2+ packs per day stratum.    So yes, The Lancet  cited issues with the distribution, but Teicholz got her figures from somewhere other than the editorial.  Sloppy, sloppy, copy, copy.

Far from "withering", The Lancet merely states that the researchers didn't stress the difference.  Teicholz would like the reader to think this meant that they didn't address it at all.  From the original Dayton paper:
Approximately 50 characteristics of each subject were recorded at the initial evaluation;  33 of these are shown in tables 1 through 14.  Because of the large number of characteristics evaluated, a difference in one of these (baldness) at the 97.5% confidence level and in another (distribution of cigarette smoking habits) at the 99% confidence level is entirely compatible with chance expectations in a fully randomized population. 
And later in their discussion they wrote:
The possibility that some variable other than diet accounted .for the outcome of the trial deserves careful consideration.  We have identified only one characteristic of the study groups which generates any suspicion of a difference in initial risk: cigarette smoking.  But any advantage of the experimental group resulting from differences in smoking habits must have been slight, judging from the data of table 13. [the table above]
I suppose this could be interpreted as giving the issue less than due consideration, but really the Lancet folks didn't say all that much more.  They said it wasn't stressed and that it could have factored into the outcome, perhaps moreso than the above paragraph might imply.  So then in the book, Teicholz tucks the fact that Dayton and colleagues did not just ignore the "critique".  They answered it in a letter of their own.   You find this in a footnote at the end of the chapter.
I: Dayton wrote a reply in The Lancet, in which he analyzes the smoking data and, based on a number of assumptions, asserts that it had “no net effect whatsoever” on the outcome of the trial (Dayton and Pearce 1970).   {1786}
This implies that Dayton essentially brushed them off, and given the way Teicholz writes and has portrayed the rest of the study, one is led to believe these assumptions are probably "cheating" in some manner.  Here are some excerpts from their response which began with ...
Sir,-- The leading article (Nov. 1, p. 939) in which you discussed our trial of a diet high in unsaturated fat made it clear that our report had not dealt adequately with at least one critical question. Specifically, your article suggested that the low incidence of atherosclerotic events in participants on the experimental diet might have been due to the chance inclusion of a smaller number of heavy cigarette smokers in that group than in the control group.  In order to satisfy ourselves and others on this point, we have undertaken further analysis of our results in relation to smoking habits. The results of this analysis, reported below, provide convincing evidence that differences in smoking habits could not have accounted for the favourable experience of subjects on the experimental diet.
So they did two additional analyses described in the letter.  First, they standardized the incidence rates per 100 man years and broke it all out by type of incident, etc.   This produced Table I below left (SD = sudden death, MI = myocardial infarction, CI = cerebral infarction).
As indicated in Table I, at any of the three levels of cigarette consumption examined, the incidence of clinical events attributable to atherosclerosis was lower in experimental subjects than in individuals on the control diet. Thus when cigarette consumption is the same, the effect of the experimental diet persists.
Perhaps expecting further criticism, they controlled for smoking a different way resulting in Table II above right.  They wrote:
Although Table I makes it clear that there was a dietary effect, whatever the inequalities of the smoking distribution, it is also desirable to determine whether the surplus of heavy smokers in the control group accounted, in part, for the more favourable experience of the experimental group.  Toward this end, we have developed estimates of "smoking adjusted" incidence rates-that is, estimates of the outcome which would have resulted if the subjects of each smoking stratum had been allocated in equal numbers to the control and experimental groups.
It is based on the outcome in Table II that they concluded, in full:
The resulting estimates, given in Table II, are nearly identical to the figures actually observed (bottom of Table I), the surplus of heavy smokers in the control group having been fully offset by an even larger surplus of moderate smokers (10-20 cigarettes per day) in the experimental group.  We conclude, therefore, that the uneven distribution of cigarette-smoking habits had no net effect whatsoever on the outcome of the trial.
Nina Teicholz goes out of her way to misrepresent these studies at seemingly every turn.  Remember, this is what she said:
I: Dayton wrote a reply in The Lancet, in which he analyzes the smoking data and, based on a number of assumptions, asserts that it had “no net effect whatsoever” on the outcome of the trial (Dayton and Pearce 1970).   
No, folks, what Dayton actually did do was a post hoc analysis, that I discussed in Control in Clinical Trials, where they controlled for smoking.   They didn't make a bunch of assumptions, they did statistical analyses.  I am not here to argue the quality of these or the conclusions reached.  The point is that it was done.  This is similar to how observational data is handled all the time where you isolate the effect of one variable by blocking and standardizing and comparing and seeing (a) if an effect persists when other variables are controlled for, and/or (b) doing a "what if" analysis and comparing that to the actual outcome.

At no point during Teicholz's "research" into this study was this 1970 information new.  As far as I can tell, there was no further challenge to this analysis, and if Teicholz's own "investigation" is any indication, any revisiting of these studies is a bunch of hot air.   Bottom line:
It is wrong for Teicholz to claim that smoking was not controlled for when interpreting the results of Dayton/LA Veterans.   
In retrospect a stratified randomizing strategy may have been more prudent, but between the randomization and the post hoc analysis it was controlled for in the full analysis.  There was one further publication that looked into the cancer occurrences (greater in the experimental group) in depth, and looked at smoking and the effect there as well.  To go into that one here would be overkill for this post, and I think I've made my point.  But I did want to put it "on the record" that there was still more, rather detailed, data and analysis on the cancer front.

You Couldn't Do That Kind of Trial Today .....

In the Amy Alkon interview from my clips, Teicholz seems besides herself a bit with how absurd it was for them not to control for smoking.    Forget all of the above for a bit let's look at Shai and smoking shall we?   Shai did not exclude smokers.  They used a stratified randomizing scheme where the strata were:
  • Gender
  • Age (above/below median)
  • BMI (above/below median)
  • Cardiac heart disease (Y/N)
  • Diabetes (Y/N)
  • Statins (none, < 1 yr , ≥ 1 yr)
... and I went back and double checked just to make sure I got that right.  So, although smokers made up 16% of the subjects at baseline, smoking wasn't specifically controlled for.  It looks like they lucked out with the allocation of the smokers between diets, but all we know there is a Yes/No ... as LA Veterans tells us, we could have an uneven distribution of degree of smoking, but that data is not provided and perhaps not even known.  

Should it have been?  Well, the argument could be made that since the primary outcome in this study was weight loss, perhaps not.  But since they were also assessing the health impact of these diets using metabolic markers influenced by smoking, it perhaps should have been.  They did randomize by CHD and diabetes and neither of those two conditions have much of an impact on weight loss per se.   I suppose an argument could be made that a history of either might impact the outcomes more that smoking might ... that's fair.   But if Shai is being portrayed as unequivocal "proof" of a dietary effect of fat, then let's drop the double standard here and insist that smoking be controlled for before we crown the Atkins diet as the best for every last ailment on the planet.  

Speaking of smoking though .....

... it DOES come up in the  2009 follow-on paper, that there was greater attrition in smokers at 25% vs. 14% for non-smokers (statistically significant, P < 0.04).   Unfortunately, in this rigorously-well-controlled-amazing-modern-day clinical trial, we were not provided with the breakdown for smoker attrition by diet-group.  We know that overall attrition was 10% LF, 15% MED and 22% LC ... if more smokers dropped out of the study, and more dieters dropped out of the LC group, perhaps more smokers dropped out of the LC group which could have impacted the outcome.  Of course this is pure *assumption* because .... well ... that's all we have to go by because we are not provided with the numbers (I do believe I'm repeating myself) or any indication that the authors even considered the effect this may have had on the outcome.  

The nice thing about electronic papers is that you can search them to make sure you didn't miss any discussion.    So I searched this paper for smok so as to find every reference to smoke(s), smoker(s) or smoking.  There is no analysis.  There were two cases where they "controlled for smoking" in a post hoc analysis to determine predictions for adherence.  It turned out that when controlling for smoking and other factors, higher initial BMI and lower weight loss at 6 months both remained predictors for dropping out.  Big whoop.  They also controlled for smoking and other factors and found that greater initial weight loss predicted successful completion.   Another huge snooze.   All that means is that if you lose a lot a quarter the way through that's all the more you can regain and still remain over 5% below baseline (the criteria for "success").  The weight change trajectories after 6 months were almost perfectly overlapped for the LC and LF diets.   

Other than reporting baseline demographics, smoking is not even mentioned in the original paper (that I'm convinced was the only one Teicholz read, if she did even that),  and the authors make no reference to how this may have impacted the outcomes of health measures.  Smoking can cause transient hyperglycemia (e.g. here) and effect blood pressure for example.  So IF any group lost more smokers this could have impacted these measures of metabolic health -- exactly how in an intent to treat analysis, is unclear.  They could have at least tabulated the drop-outs by diet assignment within the known confounders including heart disease and diabetes?   This is, after all, really really REALLY REALLY good clinical research.

Was this important?  Who knows, perhaps, perhaps not.  But without the numbers we don't know and therefore how can Teicholz say that this study shows anything??

But Wait!  There Was Some Heart Disease Analysis ...

I think this is the last of them, but there was yet another sub-study and publication to come out of DIRECT:   Dietary Intervention to Reverse Carotid Atherosclerosis.   For this they randomly selected 175 participants, out of which acceptably high quality diagnostic images were obtained for 140:  49 LF, 55 Med, 36 LC.  Adherence, demographics, etc, were all reportedly similar to the full cohort.   The primary outcome was carotid vessel wall volume (VWV) which correlates well with intima-media thickness (IMT)  but can give meaningful results over shorter time frames.

They concluded that regression occurred corresponding to weight loss and concurrent reductions in blood pressure, but there was no statistically significant difference between diets.  Therefore, it would be total folly to try to analyze this data.  Oddly enough, there were far more smokers in the lowest third of VWVs compared to the highest third.  This does not mean that smoking correlates with lower VWV, rather it may reflect that those with more serious heart disease may have heeded the warnings and quit smoking.   What does it all mean?  Again, who knows, and the supplemental material link is dead,  if it even held any clue.  

Concluding Thoughts

So in the DIRECT/Shai trial -- the one Teicholz has held up as supreme -- 
  • If only 6% of the highest tertile smoked at baseline, and the smokers dropped out at higher rates?  Do we care?  
  • If there were more than twice as many Mediterranean dieters than LCers in the lowest tertile (expected to see the least change) at baseline?  Matters?
Nah ... we don't really care about such minutia ... right?    Now 25 more heavy smokers out of 422 in the control group in Dayton.  That's a methodological nightmare.  Those were balanced out by 44 more moderate smokers in the experimental group?  Nah, can't go making assumptions.    Detailed accounting and controlling and adjusting to demonstrate that smoking didn't impact the results in Dayton?  Not good enough.  

But Shai ...

So I think I've laid out a solid case here again for the superiority of the Dayton/LA Vet study over Shai/DIRECT.  I mean there really is no contest.  This alone discredits Nina Teicholz and she really needs to be held to account to correct the record.  


charles grashow said…
The idiocy continues

My favorite slide and discussion starts at around 45 minutes into the presentation, looking at LDL-C relative to HDL-C. Direct observation from the Framingham heart study data: LDL-C does NOT predict cardiovascular risk as HDL-C rises (see the image below using international units mmol/l).

In other words, LDL-C becomes mostly irrelevant as HDL-C improves! To date ONLY a nutritional approach, specifically a well formulated low carb high fat (LCHF) diet reduces risk by raising HDL-C, and lowering triglyceride, consistent with the favorable markers observed in the original Framingham study. We also now know that LCHF diets also favorably improve cholesterol size and quality regardless of LDL particle count/concentration, based on modern advanced NMR lipid testing.
Ancestral Chemist said…
This is a good example of how a statician can make a graph look exactly how he wants it to look. Low HDL is a good indicator of CHD and metabolic syndrome. However, if the lower two lines were plotted on a separate graph, their slopes would look as steep as the upper lines. It's hard to tell on the graph, but it looks like the people with the highest HDL (~80 mg/dl) have a doubling of risk as LDL doubles from ~100 mg/dl to ~200 mg/dl, from hazard ratio ~.4 to ~.8. With lowest HDL (~25 mg/dl) risk doubles as LDL doubles from ~100 mg/dl to ~200 mg/dl, from hazard ratio ~1.5 to ~2.5.
Sorry, it's going to take more than that to discredit "everything you knew about SFA and serum cholesterol." I think I'll pass on the buttered steak and eat some EVIL whole grains and legumes, in the form of deliciously spiced dahl.
StellaBarbone said…
NMR lipoprotein testing? Has that been validated yet? Any studies that look at cardiac event endpoints? Anything that compares NMR lipoprotein results even to carotid artery intima? These were the best that I could come up with in a cursory search and they make it appear that NMR is a great moneymaker, but not quite ready for prime time.
charles grashow said…

Lipoprotein Particle Profiles by Nuclear Magnetic Resonance Compared with Standard Lipids and Apolipoproteins in Predicting Incident Cardiovascular Disease in Women

"There was essentially no reclassification improvement with adding LDLNMR particle concentration or apolipoprotein B100 to a model that already included the total/HDL cholesterol ratio and non-lipid risk factors (net reclassification index [NRI], 0% and 1.9%, respectively), nor did the addition of either variable result in a statistically significant improvement in the c-index.

In this prospective study of healthy women, CVD risk prediction associated
with lipoprotein profiles evaluated by NMR was comparable but not superior to standard lipids or apolipoproteins."

"In sum, CVD risk prediction associated with NMR lipoprotein profiles in
this large prospective cohort of women was comparable but not superior
to standard lipids or immunoassay-measured apolipoproteins. Thus, our
data support the use of standard lipids, in particular the total/HDL
cholesterol ratio, which are highly effective and readily available, for
routine CVD risk assessment."
carbsanity said…
We want Jimmy Moore exposes, Evie. This science-y stuff us getting tedious. Moar red meat! :-)
billy the k said…
"If you can eat two whole eggs for breakfast every day and still maintain a cholesterol level of under 220, go ahead." [And while butter wasn't recommended, frying them up in bacon fat WAS--"if you are partial to the flavor of bacon."]

Sounds good to me, Ancel. [Ancel & Margaret Keys. Eat Well and Stay Well. Rev.Ed. 1963. p.142-143.]
charles grashow said…

The effects of low-carbohydrate diets (≤45% of energy from carbohydrates) versus low-fat diets (≤30% of energy from fat) on metabolic risk factors were compared in a meta-analysis of randomized controlled trials.

Both low-carbohydrate and low-fat diets lowered weight and improved metabolic risk factors. Compared with participants on low-fat diets, persons on low-carbohydrate diets experienced a slightly but statistically significantly lower reduction in total cholesterol (2.7
mg/dL; 95% confidence interval: 0.8, 4.6), and low density lipoprotein cholesterol (3.7 mg/dL; 95% confidence interval: 1.0, 6.4), but a greater increase in high density lipoprotein cholesterol (3.3 mg/dL; 95% confidence interval: 1.9, 4.7) and a greater decrease in triglycerides (−14.0 mg/dL; 95% confidence interval: −19.4, −8.7). Reductions in body weight, waist circumference and other metabolic risk factors were not significantly different between the 2 diets. These findings suggest that low-carbohydrate diets are at least as effective as low-fat diets at reducing weight and improving metabolic risk factors. Low-carbohydrate diets could be recommended to obese persons with abnormal metabolic riskfactors for the purpose of weight loss. Studies demonstrating long-term
effects of low-carbohydrate diets on cardiovascular events were warranted.
Karin said…
I was shocked to see a review of this book in The Economist, which we subscribe to. I realized this must mean it's a pretty popular book. I guess I'm just wondering why. Is it just because it's a controversial topic? I've seen many books on the same topic come and go. It almost seems like old news to me now. I didn't see Eenfeldt's or Moore's book reviewed in The Economist (thankfully), so why this one? The mind boggles.
I'll take a shot at this one...

It comes down to two factors: timing and having a new face.

Now timing can be managed with some tactical thinking, but in the case of the individuals you mentioned, they are rather popular fixtures in the low carb sphere, which can dampen the superficial appeal to novelty when trying to re-pitch a paraphrased version of GCBC. When it comes to capitalising on controversy in the media, one needs a bit of subtly when attempting to repackage old ideas as something new. These individuals can't really reinvent themselves, so their pitch won't pack the same punch. In such situations, where a controversial idea has already been exhausted at a mainstream level, publications need new authors/fresher faces--with little background of visible biases--to be able to effectively pitch the same controversy again as something revolutionary.

Basically, they wanted a pristine new messenger for the same old stuff, and they got that with this author.
eulerandothers said…
These RCTs were short-term studies, right?
eulerandothers said…
I totally agree. 'What if it was all a big fat lie?' morphed into 'The Big Fat Surprise'

Someone - maybe Taubes - saw 'My Big Fat Greek Wedding' and said, 'That's got a nice ring to it, and it made a bundle, let's use the Big Fat part and run that up the flagpole...'

Next up, 'My Big Fat Tabby Cat' and 'The Big Fat Wall Street Trading Scandal,' followed by 'The Next Big Fat Food Network Star' and 'The Big Fat Conspiracy to Steal Your Retirement Savings.' Then, 'How Not to Raise a Big Fat Baby' and 'My Big Fat Nicotine Patch.'

Cover all the bases!
Pink Floyd said…
We have yet another Shai'ster on our hands, David Perlmutter. Check it out:
Next up, 'My Big Fat Tabby Cat' and 'The Big Fat Wall Street Trading Scandal,' followed by 'The Next Big Fat Food Network Star' and 'The Big Fat Conspiracy to Steal Your Retirement Savings.' Then, 'How Not to Raise a Big Fat Baby' and 'My Big Fat Nicotine Patch.' <- Evelyn, I think you got some quality material here.
carbsane said…
Nigel Kinbrum said…
Er, hello! Where are the results for the Mediterranean diet? :-/
carbsane said…
Watching after I reboot later ... got a plugin prob going on. But, in the meantime ... what's with the logo? Did he twist the brain down the midline? Huh??
Unknown said…
Find best Home based Business without any work, just invest and rest with your profit
Blogger said…
EasyHits4U - Your Free Manual Traffic Exchange - 1:1 Ratio, Business networking. FREE Advertising!