Bluster or Breakthrough? How the Public Should Evaluate Medical Reporting
A world of instant information and over-reporting can flood us with potentially vital but often trivial health information. Understanding the credibility of the study at the source can help you eliminate the static and recognize the relevant.
Recently, I saw a small study published in a very obscure medical journal covered on a major network evening news program. Involving only 20 patients, it examined the sexual side effects from Propecia®, the low dose version of finasteride used in men to treat/ prevent baldness. The report seemed alarmed about this issue in treatment for a cosmetic problem, implying that the results were shocking. However, nothing new can be drawn from this information. Finasteride has been known to cause significant sexual dysfunction side effects in men since it was marketed for symptomatic enlargement of the prostate over 20 years ago. Even this effect is not unexpected, since finasteride works by blocking effects of testosterone, the male hormone responsible for sex drive in both men and women. Mention of this tiny and inadequate article from an obscure journal on a network new program is beyond my understanding.
The basic issue is that people in the public need to look at these news stories with caution, realizing that some of them are encouraged, or even planted in the press, by supporters of the researchers with their own agenda. Medical research comes in many different forms, with a number of different evaluation techniques used. The important point everyone should remember is that the questions answered in a research project are determined by the questions posed at the beginning of the project. While additional questions are sometimes addressed, the originals are the ones from which you can draw reasonable conclusions.
Very important early medical research trials in the era after the end of World War II were epidemiological studies, which look at a very large group of patients, often whole national populations or ethnic groups, to determine the presence of disease or the risk factors for disease. The studies then assess the relationship between the presence of a given risk factor and selected diseases to determine if there is a correlation. These studies then follow the identified patients for a number of years to research relationships such as high blood pressure and the development of heart attack. As the patients begin to develop diseases, like heart attack or stroke, the researchers then will look at the various factors documented at the beginning of the study to see if their presence increased or decreased risk. Epidemiological studies have been done in many populations around the world, showing the influence of these factors in producing health problems in many ethnic groups.
The most famous of these studies in the United States may be the Framingham Heart Study, which began in Framingham, Massachusetts, in 1948, and has continued through to this day. The Framingham study was pivotal in the creation of our national interest in the causes of our heart attack problem by identifying certain risk factors, like elevated blood pressure, altered blood cholesterol, and smoking. The Framingham study has been crucial to our understanding of the risk factors that lead to the development of early heart disease in our population. It was extremely well designed and conducted, with the patients examined directly by the research study’s own physicians and the laboratory evaluations conducted by the study’s own laboratory. While Framingham did not treat any of these risk factors, its information spawned a number of treatment studies that addressed issues like high blood pressure or elevated cholesterol and showed the importance of treating these risks in follow up studies.
We do now recognize that as well designed and conducted as the Framingham Study is, it does have its weaknesses. The inhabitants of Framingham in 1948 were mostly Caucasian, so it does not reflect the ethnic diversity of the United States. Also, by focusing on heart and stroke risks, the study left many other major medical questions unanswered. Several of these issues have been addressed by other studies, with the Nurses Health Studies I and II being good examples. Enrolling several hundred thousand registered nurses across the entire United States, these studies collected health reports from participating nurses every two years, looking at questions like contraceptive use and disease, diet and colon cancer, and heart disease risk factors. Due to large numbers and broad age range, the study provides the diversity necessary to understand these issues.
As valuable as these studies are, they can only identify risk factors which suggest they are the root cause of health problems. When it comes to evaluating treatments, studies must recruit patients possessing the specific risk factor. Doctors then treat half of them with a medication to be studied and the other half with a placebo, to answer whether treatment of a risk factor can improve health. These treatment studies involve many thousands of patients and run for multiple years, often as many as three to five. Patients are closely followed to see if they develop heart attack or another of the expected consequences from the risk factors being treated. If the patients who received the real medication experience significantly fewer heart attacks or other potential problems than the patients who received the placebo, it is conclusive evidence that the medication is effective. It proves that it not only reduces the risk factors, such as blood pressure or cholesterol, but also that heart attack is caused by the risk factor.
Such studies, called placebo controlled double blind research, are the gold standard in proving the effectiveness of a new or not previously studied medication. Double blind means that neither the patients nor the medical staff and study personnel know which patients are on the real medication or the placebo, resulting in an accurate assessment of the treatment’s benefits or ill effects. These studies have shown the effectiveness of many treatments we now accept as important therapies for many conditions. These types of projects have also studied treatments showing that various medications can reduce some complications of diabetes, recurrent strokes, and depression. They can also examine certain invasive procedures, suchas evaluating whether the angioplasties that open blockages in the heart arteries are superior to management by medications without the angioplasty.
However, many times the opposite holds true and promising treatments turn out to be worthless. A number of these reports were greeted with anger and dismay by many when the study results disproved widely believed assumptions. One of the more obvious examples (and believe me there are a number of these) was the use of estrogen to prevent heart attack. The estrogen story provides a great example of why we must do placebo controlled studies.
Before the placebo controlled study, numerous observational studies suggested that taking estrogen for whatever reason protected women from developing early heart disease and heart attack. Observational studies look at large populations and follow them for years. These studies look at patient’s lifestyle practices like diet, exercise, and medications (such as vitamins, aspirin, and estrogen), taken not to treat a known medical problem but just to prevent the development of a problem. The observational studies that convinced many people of estrogen’s positive effects on heart disease looked at postmenopausal women and recorded many of their lifestyle issues − diet, smoking, exercise, medication use, and a wide variety of other personal factors. When they looked at the women who were regular users of estrogen after menopause, which was a distinct minority of only about 20%, this group had a significantly lower rate of heart attack than women who did not take estrogen. However, careful evaluation showed that these women who took estrogen and had lower heart attack rates also practiced much healthier lifestyles. They had better diets and noticeably higher rates of regular exercise as well as fewer smokers—hinting that the use of estrogen might simply be part of a healthier lifestyle at that time.
The only way to answer the question about whether taking estrogen protected against heart attack was to perform a double blind placebo controlled trial, having large groups of similar women take estrogen or a placebo. Some observers actually objected to the trial, stating that the observational evidence alone justified estrogen treatment of menopausal women. However, the results of the trial (actually several trials with slight differences) showed clearly that, not only did estrogen not prevent heart attack, but there were slight but clear increases in heart attack rate, stroke rate, and even breast cancer. These results were greeted with dismay by many people, but with grudging respect by physicians like me who felt that proper scientific research is needed before we plunge into using unsubstantiated therapies. There continues to be pushback against the policy of not using long-term estrogen replacement, more emotional rather than scientific criticism of the research itself. However, science is science, and we need to understand how our emotional desires are often not especially scientific. When it comes to applying new treatments to individual patients or large populations of patients, we need to stick to the science and control our emotional support for the treatments that the
science does not support.
There are many other instances, similar to the estrogen question, when the potential use of an incompletely studied therapy had great appeal to the scientific and general communities. These issues include vitamin pill benefits and whether adding a second drug to the cholesterol lowering statins would reduce heart attack further. Both presumptions had very good support from epidemiological and/or observational studies. Another potential treatment tested was that a vitamin rich diet lowered risk for heart attacks. However, when vitamins versus placebos were tested in people, the vitamins did not reduce heart attack.
Within the concept of adding a second cholesterol drug to the widely used and clearly beneficial statin drugs (Lipitor®, Crestor®, and simvastatin), was the observation that a persistently low level of the HDL cholesterol (good cholesterol) is related to an increased heart attack risk. People logically assumed that adding a second agent that raises the HDL could be added to regular statin therapy. This was researched with two separate classes of drugs that increased HDL cholesterol when added to a statin drug, one being the B vitamin niacin and the other coming from the fibric acid family of cholesterol drugs, versus a placebo. However, each of those potential therapies was shown, when added to a statin treatment, to be worthless in further reduction of heart attack. This is excellent proof that even as logical a treatment as this isn’t necessarily correct, since medicine is scientific, not logical.
Serum Homocysteine is another example of a potential treatment target that was espoused in the nutrition literature. Increased levels of homocysteine, a blood protein, was noted in patients with early heart disease. This association of two elevated facts does not show that they are related to one another, i.e. whether they are really cause and effect or are just coincidental. The issue was studied when patients were given either folic acid, a vitamin that lowers homocysteine in the blood, or a placebo. In the patients who received folic acid rather than placebo, the homocysteine levels dropped to normal. However, the heart attack rates between the two groups were exactly the same, refuting the assumption that there was a cause-and-effect relationship between homocysteine and heart disease. This proves that the elevated homocysteine was not the cause of the increased heart attack risk, but simply a marker that that increased risk was present.
In the last several years there has been a lot of interest in the question of whether lower levels of Vitamin D cause an increased risk for heart disease. There have been a number of observational studies that demonstrate an association between lower levels of Vitamin D and increased risk from heart disease. This has led some practitioners to institute Vitamin D replacement in patients who have lower levels of Vitamin D. However, no large scale treatment trial has shown that administering Vitamin D is superior to a placebo, while several smaller trials have not shown any success of such treatment. The necessary large scale research projects comparing Vitamin D therapy versus placebo are underway now, so an answer will be forthcoming in a few years. Since all previous vitamin therapy trials attempting to reduce heart attack have been unsuccessful, I think that caution should be observed until clear proof is provided through a placebo controlled, double blind treatment study.
Two other types of medical research projects are reported in the popular press, but should be viewed by the public with great skepticism. The first set is animal studies, usually performed in lab animals like mice and other rodents. I have seen a large number of lab rodent trials discussed in the mainstream press, with many of them being billed as a breakthrough research project. I am always mystified by this press coverage. It is true that rats and mice are mammals like we humans are, and research in them can be a valuable introduction for research concepts that may eventually impact humans. However, rodent research is extremely preliminary and only a tiny bit suggestive when it comes to human disorders. There have been numerous rodent studies that have been shown to have no applicability to humans after research progressed up into higher and more complicated mammals or to humans.
The public needs to remember that much of the attractiveness of rodent research is the ease and low cost with which it can be performed. The animals are bred and maintained for research purposes, and they can be studied and sacrificed as necessary for research. Some species have been developed as specific study targets, such as the spontaneously hypertensive rats bred to develop high blood pressure and to be used as the first animal to test new blood pressure lowering agents. However, success in this type of animal model is only slightly suggestive of applicability to humans, and therefore it needs to be regarded as very preliminary. Therefore, if you see an article or even a television news clip about some breakthrough rodent medical study, just ignore it.
Finally, the medical community and public alike should hold very cautious skepticism of a study type called meta analysis, an evaluation effort inside medical research that has become popular in the last 20 years. Meta analyses are attempts to answer medical research questions that have not been specifically addressed by a research project, and therefore don’t offer clear answers. The team will gather a variety of studies that may have information relevant to the question they hope to answer. Researchers then study the previous project’s reports and evaluate them, discarding many projects they believe are not applicable to their current research question. Frequently, the researchers will discard most of the available studies to concentrate on a few. Selected studies are mined to elicit information relevant to the question that they want to answer. They will then evaluate this information, ranking it as to how important it is in their quest. Conclusions are drawn after the findings are assessed and a statistical evaluation is completed.
While this sounds quite reasonable, it is fraught with problems, and many academic physicians recognize that. There is a saying in academic medicine, “meta analysis is to analysis as metaphysics is to physics.” It recognizes that metaphysics is a philosophical discipline, not a science like physics. The problem is simple and intrinsic to the whole design of the meta analysis process, it is fraught with subjectivity. Authors bring their own subjectivity to the process from the very beginning, choosing the questions to answer, but then ranking the studies that they think are valuable. Information from those studies is then ranked as authors see fit. So as you can see, the whole process significantly is contaminated by the judgments of the authors, since they choose the information that they are going to use in the study, and they always seem to find the result they expected. I have commented that I have never seen a meta analysis that didn’t find exactly what the authors expected to find. Therefore, all of us, both physicians and the public, need to regard any meta analysis with caution and as only preliminary information.
As we look carefully at research questions in modern medicine, we can see areas where our lives have been made longer and safer by properly done research. This properly done research has been able to define the importance of treating certain very common medical problems like high blood pressure and elevated blood cholesterol to reduce heart attacks and other problems. This properly done research has also been able to define when and how physicians should use interventions like surgery and angioplasty to treat heart attacks and strokes. These types of projects have also defined the proper spectrum of medications that patients with known heart disease need to take after their heart disease has appeared.
Also, the safety of many of the treatments, like statin therapy to prevent heart disease and stroke, has been clearly shown repeatedly, in these carefully done studies. A wide variety of treatments in cancer and many other disorders have also been defined clearly by this type of research. And of equal importance, properly done projects have shown that many treatments that were assumed to be useful, like vitamins, Omega-3 supplements, and others, have been shown to be of no value. It is important to note the studies and potential breakthroughs mentioned in the media but hopefully this brief discussion can assist in evaluating reports to ascertain which research is credible and which projects should be considered only suggestive and preliminary.
ABOUT THE AUTHOR
Robert Guthrie, MD is both a family physician and a general internist. He currently serves as Professor Emeritus of Emergency Medicine at the Ohio State University. He has published one book, numerous medical articles, has lectured extensively nationally and internationally, and been a long-term participant on local and national radio broadcasts. His column discusses a wide variety of health topics of interest to the general public. Questions, comments, or topic ideas may be sent to firstname.lastname@example.org