We have frequently advocated for evidence-based medicine (EBM) , that is, medicine based on judicious use of the best available evidence from clinical research, critically reviewed and derived from systematic search, combined with biomedical knowledge and understanding of patients' values and preferences. However, EBM risks being turned into pseudo-evidence based medicine due to systematic manipulation and distortion of the clinical research evidence base. Dr Wally Smith wrote about pseudo-evidence based medicine, which he defined as "the practice of medicine based on falsehoods that are disseminated as truth," in the British journal Clinical Governance (Smith WR. Pseudoevidence-based meidicne: what it is, and what to do about it. Clinical Governance 2007; 12: 42-52. Also see this post).
Now it appears that this issue is causing concern in the Cochrane Collaboration, the main voluntary international group promoting EBM. An article from December, 2011 in the Indian Journal of Medical Ethics outlined why we need to question the trustworthiness of the global clinical evidence base. (See Tharyan P. Evidence-based medicine: can the evidence be trusted. Ind J Med Ethics 2011; 8: 201-207. Link here.) It merits further review.
The Role of Vested Interests
Dr Tharyan, its author, emphasized that a major threat to the integrity of the clinical research data base is the influence on clinical research of those with vested interests in marketing particular products, e.g., drugs, devices, etc.
The motives for conducting research are often determined by considerations other than the advancement of science or the promotion of better health outcomes. Many research studies are driven by the pressure to obtain post-graduate qualifications, earn promotions, obtain tenured positions, or additional research funding; many others are conducted for financial motives that benefit shareholders, or lead to lucrative patents.
The Importance of Pervasive Conflicts of Interest
Early on, the author discussed how the distortion of the clinical research data base arises from the pervasive web of conflicts of interest in medicine and health care:
This hijacked research agenda perpetuates further research of a similar nature that draws more researchers into its lucrative embrace, entrenching the academic direction and position statements of scientific societies and academic associations. Funders and researchers are also deterred from pursuing more relevant research, since the enmeshed relationship between academic institutions and industry determines what research is funded (mostly drugs at the expense of other interventions), and even how research is reported; thus hijacking the research agenda even further away from the interests of science and society.
The article then goes on to show how the influence of vested interests may distort the design, implementation, analysis, and dissemination of research.
Distortion of Research Design and Implementation
The Research Question
Dr Tharyan noted
The majority of clinical trials conducted world-wide are done to obtain regulatory approval and a marketing licence for new drugs. These regulations often require only the demonstration of the superiority of a new drug over placebo and not over other active interventions in use. It is easier and cheaper to conduct these trials in countries with lower wages, lax regulatory requirements, and less than optimal capacity for ethical oversight. It is therefore not surprising that the focus of research does not reflect the actual burden of disease borne by people in the countries that contribute research participants, nor address the leading causes of the global burden of disease. Some 'seeding' trials, conducted purportedly for the purpose of surveillance for adverse effects, are often only a ploy to ensure brand loyalty among participating clinician-researchers.
Insufficient Sample Size
The article stated,
Many trials do not report calculations on which the sample size was estimated, often leading to sample sizes insufficient to detect even important differences between interventions (for primary, let alone secondary outcomes)
I would add that such small trials are particularly bad at detecting important adverse effects of the interventions being promoted.
Excessively Stringent Enrollment Criteria
As Dr Tharyan wrote,
Most RCTs funded by industry and academia are designed to demonstrate if a new drug works, for licensing and marketing purposes. In order to maximise the potential to demonstrate a 'true' drug effect, homogenous patient populations; placebo controls; very tight control over experimental variables such as monitoring, drug doses, and compliance; outcomes addressing short term efficacy and safety; and methods to minimise bias required by regulatory agencies are used to demonstrate if, and how, the drug works under ideal conditions.
The problem is that very few patients in clinical practice resemble those enrolled in such trials, so the generalizability of the trials' results is actually dubious. Ideally, clinicians practicing evidence-based medicine could refer to trials that include patients similar to those for whom they care.
Practical or pragmatic clinical trials are designed to provide evidence for clinicians to treat patients seen in day-to day clinical practice, and evaluate their effectiveness under 'real-world' conditions. These trials use few exclusion criteria and include people with co-morbid conditions, and all grades of severity. They compare active interventions that are standard practice, and in the flexible doses and levels of compliance seen in usual practice. They utilise outcomes that clinicians, patients, and their families consider important, such as satisfaction, adverse events, return to work, and quality of life). Recommendations exist on their design and reporting , but such trials are rare.
Comparisons to Placebo, not the Other Interventions Clinicians Might Realistically Consider
Per the article,
Industry sponsored trials rarely involve head-to head comparisons of active interventions, particularly those from other drug companies, thus limiting our ability to understand the relative merits of different interventions for the same condition.
The result may thus be a number of studies showing particular interventions appear to be better than nothing, but few if any studies that would help clinicians decide which intervention would be best for a particular patient.
Inappropriate Comparators
However,when studies are done comparing the intervention of interest to other active interventions, the details of the choice of comparators are often managed so that the comparators are likely to appear to be worse.
Even if active interventions are compared in industry-sponsored trials, the research agenda has devised ways in which the design of such trials is manipulated to ensure superiority of the sponsor’s drug. If one wants to prove better efficacy, then the comparator drug is a drug that is known to be less effective, or used in doses that are too low, or used in non-standard schedules or duration of treatment. If one wants to show greater safety, then the comparator is a drug with more adverse effects, or one that is used in toxic doses. Follow up, also, is typically too short to judge effectiveness over longer periods of time.
Meaningless Outcomes
A favorite tactic used in the design of trials influenced by vested interests is to choose outcomes that are likely to show results favorable to the product being promoted, but have no meaning for patients or clinicians. There are three ways this is commonly done.
The first is to use rating scales that are very sensitive to small perturbations, but not meaningful in terms of how patients feel or function:
The choice of outcome measures used often ensures statistically significant results in advance, at the expense of clinically relevant or clinically important results. Outcomes likely to yield clinically meaningless results include the use of rating scales (depression, pain, etc.). These scales yield continuous measures usually summarised by means and standard deviations, rather than the dichotomous measures clinicians use such as: clinically improved versus not improved. These rating scales, however extensively validated, are hardly ever used in routine clinical practice. A difference of a few points on these scales results in statistically significant differences (low p values), that have little clinical significance to patients.
Another is to use surrogate outcomes,
