LYMEPOLICYWONK:
New Study Reveals Fatal Flaws in NIH Klempner Trial Statistical Analysis. Is this Error Human, Incompetence or Worse?
29th August 2012
A
new study by Allison DeLong, Barbara Blossom, Dr. Elizabeth Maloney,
and Dr. Steven Phillips entitled “Antibiotic Retreatment of Lyme disease
in Patients with Persistent Symptoms: A Biostatistical Review of
Randomized, Placebo-controlled, Clinical Trials” examined the quality of
the studies, statistical analysis and conclusions reached by the four
NIH-funded clinical trials for persistent Lyme disease. For this blog, I
am going to focus on one of them, the Klempner trial. It is the trial
most commonly cited by the Infectious Diseases Society of America to
support the IDSA guidelines recommendation that patients who remain ill
after treatment should not be retreated. The trial involved both a
seropositive arm (patients who tested positive for Lyme disease) and a
seronegative arm (patients who tested negative for Lyme disease).
So what was the fatal error in the Klempner study?
Researchers who design clinical trials define when a treatment is considered a success. Does the treatment work or not? How much does a patient have to improve before you can say the treatment helped? Do they have to return to perfect health? Clinical trials usually require that patients improve, but not that they return to perfect health. DeLong found that the Klempner trial set the level for determining treatment success excessively high. For instance, in the seronegative arm of the trial, treatment success required that patients not simply return to the norm for the general population, but instead surpass the norm by essentially one full standard deviation. That type of success is unheard of in clinical trials.
More specifically, the Klempner study used too high a level to determine success on something called the SF-36 scale. It used a measure that was not clinically meaningful. Let’s look at this in context. While other chronic diseases set the bar for improvement on the SF-36 between 2 to 5, the Klempner trial set the bar at roughly 7 and 9 in the seropositive arm and between 9 and 13 in the seronegative arm. This amount of improvement is far higher than the amount of improvement that could reasonably be expected among the chronically ill.
If the Klempner trial had used a clinically meaningful measure of success, say between 2 and 5, it would have had to have used much, much larger sample sizes, which it did not have. For instance setting the clinically meaningful measure of success at 2 would have required a sample of 400 or more per treatment arm. The sample sizes in Klempner were only 78 (seropositive) and 51 (seronegative)—a sample size which would not yield statistically significant results. When researchers select an excessively large treatment effect required for success, as was done here, the size of the sample required is reduced. The trade-off, however, is that the results are not clinically meaningful.
Many researchers say that small sample size trials are unethical because they can literally steer scientists down the wrong path. That’s what the Klempner study did. It was a waste of money, a waste of time, and it has led research down the wrong path for the last 10 years.
Was it a matter of researcher incompetence? DeLong and colleagues point out that studies on the level of improvement that is clinically meaningful for other chronic diseases had not been done at the time the Klempner trial was conducted. So the Klempner researchers did not know that they were using an excessively large treatment effect at the time they conducted the study. Does that take the researchers off the hook? Not really. First of all, once the studies on the way to measure success in chronic disease had been conducted, the Klempner team could have acknowledged a possible error.
Second, the Klempner team could have avoided “overstating” its findings and highlighted the potential for error. The ethical implications of exaggeration in science are understood by other scientific societies. For example, the code of conduct for the American Chemical Society states: “Public comments on scientific matters should be made with care and accuracy, without unsubstantiated, exaggerated, or premature statements.”
Instead, the Klempner study concluded flatly that retreatment was ineffective. The NIH press release headline read: “Chronic Lyme Disease Symptoms Not Helped by Intensive Antibiotic Treatment.” It further concluded that “ it is unlikely that a longer course of treatment or different antibiotic combination would result in greater improvement than what we found in these studies.” This conclusion was clearly beyond the scope of the study since it did not test other antibiotics or other durations of antibiotic therapy.
In short, rather than acknowledging the potential weaknesses of its findings and encouraging further research, the Klempner study overstated its conclusions and firmly shut the door on future science. Dr. Klempner himself then sat on the 2006 IDSA Lyme disease guidelines panel, which used his study to deny seriously ill patients any access to care. Research of fellow panel members on a guideline panel may be less likely to be critically challenged, and the IDSA guidelines are riddled with references from the panel members.
This guideline recommendation has caused untold suffering by patients, who are essentially told to simply live with the symptoms of chronic Lyme, which can cause disability equivalent to that of congestive heart failure. For example, 25% of over 4,000 patients who responded to a survey by LymeDisease.org have been on disability at some point in their illness. If a guidelines panel is going to preclude further care of patients this ill, they should be absolutely certain that further treatment will not be beneficial. And, they should err on the side of treatment. Denying patients access to care when treatment options are available results in unnecessary suffering.
All of this brings us back to what Congressman Smith referred to as the “lost decade” of research in Lyme disease. He was referring to the delay in publishing the Embers research trial which found persistent infection in monkeys after treatment using the Klempner protocol. The lack of scientific rigor in the Klempner trial coupled with the delay in publication of the Embers trial has caused the Lyme community enormous harm. Both studies were NIH funded, and, again we are left to wonder why the NIH and Dr. Phil Baker, who headed up Lyme research for the NIH, failed to properly oversee and follow-up on this research. Why aren’t they setting the record straight now that they have the information?
It brings to mind a comment that Dr. Stephen Barthold made at the congressional hearing chaired by Congressman Smith:
Delong, A. K., B. Blossom, et al. (2012). “Antibiotic retreatment of Lyme disease in patients with persistent symptoms: A biostatistical review of randomized, placebo-controlled, clinical trials.” Contemp Clin Trials.
Klempner, M., L. Hu, et al. (2001). “Two controlled trials of antibiotic treatment in patients with persistent symptoms and a history of Lyme disease.” N Engl J Med 345(2): 85-92.
Embers, M. E., S. W. Barthold, et al. (2012). “Persistence of Borrelia burgdorferi in Rhesus Macaques following Antibiotic Treatment of Disseminated Infection.” PLoS ONE 7(1): e29914.
My previous blog postings regarding the Embers monkey trials and their importance for Lyme patients can be found here.
The LYME POLICY WONK blog is written by Lorraine Johnson, JD, MBA, who is the Chief Executive Officer of LymeDisease.org, formerly CALDA. Contact her at lbjohnson@lymedisease.org.
Researchers who design clinical trials define when a treatment is considered a success. Does the treatment work or not? How much does a patient have to improve before you can say the treatment helped? Do they have to return to perfect health? Clinical trials usually require that patients improve, but not that they return to perfect health. DeLong found that the Klempner trial set the level for determining treatment success excessively high. For instance, in the seronegative arm of the trial, treatment success required that patients not simply return to the norm for the general population, but instead surpass the norm by essentially one full standard deviation. That type of success is unheard of in clinical trials.
More specifically, the Klempner study used too high a level to determine success on something called the SF-36 scale. It used a measure that was not clinically meaningful. Let’s look at this in context. While other chronic diseases set the bar for improvement on the SF-36 between 2 to 5, the Klempner trial set the bar at roughly 7 and 9 in the seropositive arm and between 9 and 13 in the seronegative arm. This amount of improvement is far higher than the amount of improvement that could reasonably be expected among the chronically ill.
If the Klempner trial had used a clinically meaningful measure of success, say between 2 and 5, it would have had to have used much, much larger sample sizes, which it did not have. For instance setting the clinically meaningful measure of success at 2 would have required a sample of 400 or more per treatment arm. The sample sizes in Klempner were only 78 (seropositive) and 51 (seronegative)—a sample size which would not yield statistically significant results. When researchers select an excessively large treatment effect required for success, as was done here, the size of the sample required is reduced. The trade-off, however, is that the results are not clinically meaningful.
Many researchers say that small sample size trials are unethical because they can literally steer scientists down the wrong path. That’s what the Klempner study did. It was a waste of money, a waste of time, and it has led research down the wrong path for the last 10 years.
Was it a matter of researcher incompetence? DeLong and colleagues point out that studies on the level of improvement that is clinically meaningful for other chronic diseases had not been done at the time the Klempner trial was conducted. So the Klempner researchers did not know that they were using an excessively large treatment effect at the time they conducted the study. Does that take the researchers off the hook? Not really. First of all, once the studies on the way to measure success in chronic disease had been conducted, the Klempner team could have acknowledged a possible error.
Second, the Klempner team could have avoided “overstating” its findings and highlighted the potential for error. The ethical implications of exaggeration in science are understood by other scientific societies. For example, the code of conduct for the American Chemical Society states: “Public comments on scientific matters should be made with care and accuracy, without unsubstantiated, exaggerated, or premature statements.”
Instead, the Klempner study concluded flatly that retreatment was ineffective. The NIH press release headline read: “Chronic Lyme Disease Symptoms Not Helped by Intensive Antibiotic Treatment.” It further concluded that “ it is unlikely that a longer course of treatment or different antibiotic combination would result in greater improvement than what we found in these studies.” This conclusion was clearly beyond the scope of the study since it did not test other antibiotics or other durations of antibiotic therapy.
In short, rather than acknowledging the potential weaknesses of its findings and encouraging further research, the Klempner study overstated its conclusions and firmly shut the door on future science. Dr. Klempner himself then sat on the 2006 IDSA Lyme disease guidelines panel, which used his study to deny seriously ill patients any access to care. Research of fellow panel members on a guideline panel may be less likely to be critically challenged, and the IDSA guidelines are riddled with references from the panel members.
This guideline recommendation has caused untold suffering by patients, who are essentially told to simply live with the symptoms of chronic Lyme, which can cause disability equivalent to that of congestive heart failure. For example, 25% of over 4,000 patients who responded to a survey by LymeDisease.org have been on disability at some point in their illness. If a guidelines panel is going to preclude further care of patients this ill, they should be absolutely certain that further treatment will not be beneficial. And, they should err on the side of treatment. Denying patients access to care when treatment options are available results in unnecessary suffering.
All of this brings us back to what Congressman Smith referred to as the “lost decade” of research in Lyme disease. He was referring to the delay in publishing the Embers research trial which found persistent infection in monkeys after treatment using the Klempner protocol. The lack of scientific rigor in the Klempner trial coupled with the delay in publication of the Embers trial has caused the Lyme community enormous harm. Both studies were NIH funded, and, again we are left to wonder why the NIH and Dr. Phil Baker, who headed up Lyme research for the NIH, failed to properly oversee and follow-up on this research. Why aren’t they setting the record straight now that they have the information?
It brings to mind a comment that Dr. Stephen Barthold made at the congressional hearing chaired by Congressman Smith:
There is overwhelming evidence in a
variety of animal species as well as humans that B. burgdorferi persists
without treatment, but the crucial question is does it survive
following treatment, and if so, do surviving spirochetes cause “chronic”
Lyme Disease or PLDS? These questions cannot be answered by speculative and expensive human clinical trials motivated by firmly held dogmatism.
Hear! Hear!
References:Delong, A. K., B. Blossom, et al. (2012). “Antibiotic retreatment of Lyme disease in patients with persistent symptoms: A biostatistical review of randomized, placebo-controlled, clinical trials.” Contemp Clin Trials.
Klempner, M., L. Hu, et al. (2001). “Two controlled trials of antibiotic treatment in patients with persistent symptoms and a history of Lyme disease.” N Engl J Med 345(2): 85-92.
Embers, M. E., S. W. Barthold, et al. (2012). “Persistence of Borrelia burgdorferi in Rhesus Macaques following Antibiotic Treatment of Disseminated Infection.” PLoS ONE 7(1): e29914.
My previous blog postings regarding the Embers monkey trials and their importance for Lyme patients can be found here.
The LYME POLICY WONK blog is written by Lorraine Johnson, JD, MBA, who is the Chief Executive Officer of LymeDisease.org, formerly CALDA. Contact her at lbjohnson@lymedisease.org.
