Professional Documents
Culture Documents
unrecognised advantage of some forms of proportion of trials have remained unpub- CCTR and MEDLINE often contain
fetal monitoring during labour in reducing lished [18,19]. Furthermore, many trials more than one record from a single study,
neonatal seizures [9]. have been published in journals without and there are lags in adding new records
By the mid-1980s, the need to minimise being electronically indexed as trials, to both databases. The NLM filters are
the likelihood of being misled by the effects which makes them difficult to find. One probably not as efficient at excluding non-
of biases and the play of chance in reviews of the first steps in being able to adequate- trials as are the methods used to compile
of research evidence was being made ly review literature is that scientific CCTR. Furthermore, MEDLINE has
evident in articles [10–14] and textbooks contributions which predate digitalised more language restrictions than CCTR.
[15]. In 1988, regularly updated electronic information systems and trial indexing In brief, there is still no single repository
publication of systematic reviews and need to be ‘‘rediscovered and inserted into reliably showing the true number of
meta-analyses, along with bibliographies the memory system’’ [20]. Through the randomised trials. Similar difficulties apply
of randomised trials, began in the perina- 1990s, to identify possible reports of to trying to estimate the number of
tal field [16,17]. This provided a model for controlled trials, the Cochrane Collabora- systematic reviews and health technology
the inauguration of the international tion mobilised thousands of volunteers assessments (HTAs).
Cochrane Collaboration in 1993 to pre- around the globe to comb the major In Figures 2 and 3 we use a variety of
pare, maintain, and disseminate systematic databases, and to hand-search nondigita- data sources to estimate the numbers of
reviews of the effects of health care lised health literature, unpublished confer- trials and systematic reviews published from
interventions. ence proceedings, and books. The result of 1950 to the end of 2007 (see Text S1). The
this collaborative effort is the Cochrane number of trials continues to rise: although
Where Are We Now? Controlled Trials Register (CCTR) (now the data from CCTR suggest some fluctu-
called the Cochrane Central Register of ation in trial numbers in recent years, this
Despite this progress, the task keeps Controlled Trials). may be misleading because the Cochrane
increasing in size and complexity. We still The differences between the numbers of Collaboration virtually halted additions to
do not know exactly how many trials have trial records in MEDLINE and CCTR CCTR as it undertook a review and internal
been done. For a variety of reasons, a large (see Figure 2) have multiple causes. Both restructuring that lasted a couple of years.
N Stjernswärd J (1974) Decreased survival related to irradiation postoperatively in early breast cancer. Lancet 304: 1285-1286.
N Chalmers TC (1975) Effects of ascorbic acid on the common cold. An evaluation of the evidence. Am J Med 58: 532-536.
N Cochran WG, Diaconis P, Donner AP, Hoaglin DC, O’Connor NE, Peterson OL, Rosenoer VM (1977) Experiments in surgical
treatments of duodenal ulcer. In: Bunker JP, Barnes BA, Mosteller F, eds. Costs, risks and benefits of surgery. Oxford: Oxford
University Press. pp 176-197.
N Smith ML, Glass GV (1977) Meta-analysis of psychotherapy outcome studies. Am Psychol 32: 752-760.
N Hemminki E, Starfield B (1978) Routine administration of iron and vitamins during pregnancy: Review of controlled clinical
trials. Br J Obstet Gynaecol 85: 404-410.
N Hemminki E, Starfield B (1978) Prevention and treatment of premature labour by drugs: Review of controlled clinical trials. Br J
Obstet Gynaecol 85: 411-417.
N Chalmers I (1979) Randomized controlled trials of fetal monitoring, 1973–1977. In: Thalhammer O, Baumgarten K, Pollak A,
eds. Perinatal medicine. Stuttgart: Georg Thieme. pp 260-265.
N Policy Research Incorporated (1979) Medical Practice Information Demonstration Project. Bipolar disorder, a state of the
science report. Baltimore: Policy Research Incorporated.
N Editorial (1980) Aspirin after myocardial infarction. Lancet 1:1172-1173. [Published anonymously but written by Richard Peto.]
N Baum ML, Anish DS, Chalmers TC, Sacks HS, Smith H, Fagerstrom RM (1981) A survey of clinical trials of antibiotic prophylaxis
in colon surgery: Evidence against further use of no-treatment controls. N Engl J Med 305:795-799.
N Hampton JR (1982) Should every survivor of a heart attack be given a beta-blocker? Part I: Evidence from clinical trials. BMJ 285:33-36.
N Stampfer MJ, Goldhaber SZ, Yusuf S, Peto R, Hennekens CH (1982) Effect of intravenous streptokinase on acute myocardial
infarction: Pooled results from randomized trials. N Engl J Med 307: 1180-1182.
N Sacks HS, Chalmers TC, Berk AA, Reitman D (1985) Should mild hypertension be treated? An attempted meta-analysis of the
clinical trials. Mt Sinai J Med 52: 265-270.
N Yusuf S, Peto R, Lewis J, Collins R, Sleight P (1985) Beta blockade during and after myocardial infarction: An overview of the
randomized trials. Prog Cardiovasc Dis 27: 335-371.
Figure 2. The number of published trials, 1950 to 2007. CCTR is the Cochrane Controlled Trials Registry; Haynes filter uses the ‘‘narrow’’
version of the Therapy filter in PubMed:ClinicalQueries; see Text S1.
doi:10.1371/journal.pmed.1000326.g002
Even though these figures must be seen per week were being registered [23]. In remarkable in non-systematic (‘‘narra-
as more illustrative than precise, multiple 2007, the US Congress made detailed tive’’) reviews and case reports. Journal
data sources tell the same story: astonish- prospective trial registration legally man- publishing of non-systematic reviews, and
ing growth has occurred in the number of datory [24]. As WHO’s international the emergence of many journals whose
reports of clinical trials since the middle of clinical trials platform develops, it will sole product is non-systematic reviews, has
the 20th century, and in reports of become possible to generate a more far outstripped the growth of systematic
systematic reviews since the 1980s—and realistic picture of how many trials are reviews and HTAs, as impressive as the
a plateau in growth has not yet been being done. This registry draws together latter has been. And the number of case
reached. With a median of perhaps 80 standardised core data from all the trial reports—which can also provide impor-
participants per trial, the number of registries meeting specified quality criteria. tant new information such as adverse
people being enrolled in trials is likely to Registering full protocols and reporting effects—is far higher than the number of
be more than 2,000,000 per year [21]. trial results in these registries are the next trials or systematic reviews. Trials, system-
Prospective trial registration establishes a frontiers. atic reviews, and HTAs have undoubtedly
new genre of evidence repository: trials are had major impacts, including on clinical
registered in these databases at inception, How Close Are We to Archie guidelines: they are more likely to be cited
theoretically enabling an overview of all Cochrane’s Goal? and read than other study types [25].
published and unpublished trials. However, the staple of medical literature
In 2004, the International Committee In 1986 and 1987, Goldschmidt and synthesis remains the non-systematic nar-
of Medical Journal Editors (ICMJE, Mulrow showed how great the potential is rative review.
http://www.icmje.org/) announced that for error in reviews of health literature that Furthermore, we are a long way from
their journals would no longer publish were not conducted systematically [9,10]. having all relevant trials incorporated into
trials that had not been prospectively Looking at data such as those in Figure 3 good systematic reviews. The workload
registered [22]. Before this announcement, could provide the comforting illusion that involved in producing reviews is increas-
an average of 30 trials a week were being systematic reviews have displaced other ing, and the bulk of systematic reviews are
prospectively registered around the world. less reliable forms of information. Howev- now many years out of date [26]. The
Once the journal editors’ deadline came er, as Figure 4 shows, this is far from the median number of trials contained within
into force, more than 200 ongoing trials case. The growth has been even more individual systematic reviews has been
variously estimated at between six and 16 the trial to be necessary [30]. It is essential less common adverse effects. However, the
(Cochrane reviews now include an average that this requirement be more widely inclusion of all study types to answer all
of over 12 trials per review [27,28]; M adopted. And it is essential that reviews questions about the effects of treatments
Clarke, personal communication), but address questions that are relevant to would not necessarily provide better
many reviews have covered much the patients, clinicians and policymakers. quality information in every instance –
same territory. Thus, in the 30 years since Second, we may need to choose be- while it would unquestionably increase the
systematic reviews began in earnest, with tween elaborate reviews of a quarter of the time and resource requirement for re-
around 15 years of intensified and large- questions clinicians and patients have or views. While it is vital that reviews are
scale reviewing effort, only a minority of ‘‘leaner’’ reviews of most of what we want scientifically defensible, burdening those
trials have been assessed in systematic to know. The methodological standards preparing them with excessive require-
reviews. Given the triple constraint posed for systematic reviewing have been in- ments could result in having valid answers
by the growth in trials, the increasing creasing over time [28], and the evolution to relatively few questions.
complexity of review methods, and current of standards in the Cochrane Collabora- In particular, we need leaner and more
resources, Archie Cochrane’s vision will tion and in HTA has been remarkable. efficient methods of staying up-to-date
not be achieved without some serious The increase in steps and reporting with the evidence. Using current methods,
changes in course—in particular, with a required is reflected in the length of the Cochrane Collaboration has not been
greater concentration on Cochrane’s use reviews. Early Cochrane reviews could able to keep even half of its reviews up-to-
of the word ‘‘relevant’’. typically be printed out in 10 or 20 pages, date [31], and other organisations are in a
even when they incorporated several trials. similar predicament [32]. We need to
Where to Now? Today, it is not unusual for a review by a develop innovative methods to reduce the
health technology agency to run to several labour of updating, and provide what
First, we need to prioritise effectively hundred pages. Often the reviews are clinicians and patients need: an assurance
and reduce avoidable waste in the pro- longer than the combined length of the that a conclusion is not out of date, even if
duction and reporting of research evidence reports of all the included trials. not every later trial is included within
[29]. This has implications for trials as well A contributing factor here is the in- every analysis. It is also the responsibility
as systematic reviews. Some funders and creasing expectation for reviews to include of reviewer authors and journal editors to
others will now not consider supporting a study types other than randomised trials. ensure that every new systematic review
trial unless a systematic review has shown This will often be essential for detecting places itself clearly in context of other
References
1. Cochrane AL (1979) 1931–1971: a critical review, 11. Mulrow CD (1987) The medical review article: statement from the International Committee of
with particular reference to the medical profes- state of the science. Ann Intern Med 106: 485–488. Medical Journal Editors. Accessed on 24 April
sion. In: Medicines for the Year 2000. London: 12. L’Abbé KA, Detsky AS, O’Rourke K (1987) 2009 at: http://www.icmje.org/clin_trial.pdf.
Office of Health Economics. pp 1–11. Meta-analysis in clinical research. Ann Int Med 23. Zarin DA, Ide NC, Tse T, Harlan WR, West JC,
2. Lind J (1753) A treatise of the scurvy. In three parts. 107: 224–232. et al. (2007) Issues in the registration of clinical
Containing an inquiry into the nature, causes and 13. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, trials. JAMA 297: 2112–2120.
cure, of that disease. Together with a critical and Chalmers TC (1987) Meta-analysis of randomized 24. One Hundred Tenth Congress of the United States
chronological view of what has been published on the controlled trials. New Engl J Med 316: 450–455. of America. Food and Drug Administration Amend-
subject. Edinburgh: Printed by Sands, Murray and 14. Oxman AD, Guyatt GH (1988) Guidelines for ments Act of (2007) Accessed 17 May 2008 at:
Cochran for A Kincaid and A Donaldson. Accessed: reading literature reviews. Can Med Assoc J 138: http://www.fda.gov/oc/initiatives/&HR3580.pdf.
26 April 2009 http://www.jameslindlibrary.org/ 697–703. 25. Dijkers MPJM, The Task, Guidelines (2009) The
trial_records/17th_18th_Century/lind/lind_kp. 15. Jenicek M (1987) Méta-analyse en médecine. value of ‘‘traditional’’ reviews in the era of systematic
html. Évaluation et synthèse de l’information clinique et reviewing. Am J Phys Med Rehab 88: 423–430.
3. Duncan A (1773) Introduction. Medical and épidémiologique. St. Hyacinthe and Paris: EDI- 26. Shojania KG, Sampson M, Ansari MT,
Philosophical Commentaries. Volume First, Part SEM and Maloine Éditeurs. Coucette S, Moher D (2007) How quickly do
I. London: J Murray. pp 6–7. 16. Chalmers I (1991) The work of the National systematic reviews go out of date? A survival
4. Cummings MM (1981) The National Library of Perinatal Epidemiology Unit. One example of analysis. Ann Intern Med 147: 224–233.
Medicine. In: Warren KS, ed. Coping with the technology assessment in perinatal care. 27. Mallett S, Clarke M (2002) The typical Cochrane
biomedical literature: A primer for the scientist and Int J Technol Assess Health Care 7: 430–459. review. How many trials? How many partici-
the clinician. New York: Praeger. pp 161–173. 17. Starr M, Chalmers I, Clarke M, Oxman AD pants? Int J Technol Assess Health Care 18:
5. Barron BA, Bukantz SC (1967) The evaluation of (2009) The origins, evolution and future of The 820–823.
new drugs: current Food and Drug Administra- Cochrane Database of Systematic Reviews (CDSR). 28. Moher D, Tetzlaff J, Tricco AC, Sampson M,
tion regulations and statistical aspects of clinical Int J Technol Assess Health Care 25 Suppl 1: Altman DG (2007) Epidemiology and reporting
trials. Arch Intern Med 119: 547–556. 182–195. characteristics of systematic reviews. PLoS Med 4:
6. Banta D (2003) The development of health 18. Hopewell S, Loudon K, Clarke MJ, Oxman AD, e78. doi:10.1371/journal.pmed.0040078.
technology assessment. Health Policy 63: Dickersin K (2009) Publication bias in clinical 29. Chalmers I, Glasziou P (2009) Avoidable waste in
121–132. trials due to statistical significance or direction of the production and reporting of research evi-
7. Stjernswärd J (1974) Decreased survival related to results. Cochrane Database Syst Rev Issue 1. dence. Lancet 374: 86–89.
irradiation postoperatively in early breast cancer. 19. Lee K, Bacchetti P, Sim I (2008) Publication of 30. Danish Research Ethics Committee System
Lancet 304: 1285–1286. clinical trials supporting successful new drug (1997) Recommendation No. 20: Controlled
8. Chalmers TC (1975) Effects of ascorbic acid on applications: a literature analysis. PLoS Med 5: clinical trials - the influence of existing and newly
the common cold. An evaluation of the evidence. e191. doi:10.1371/journal.pmed.0050191. acquired scientific results on the research ethical
Amer J Med 58: 532–536. 20. Kass EH (1981) Reviewing reviews. In: evaluation. Copenhagen: Danish Research Ethics
9. Chalmers I (1979) Randomized controlled trials Warren KS, ed. Coping with the biomedical Committee System.
of fetal monitoring 1973–1977. In: literature: a primer for the scientist and the 31. Koch GG (2006) No improvement – still less than
Thalhammer O, Baumgarten K, Pollak A, eds. clinician. New York: Praeger. pp 79–91. half of the Cochrane reviews are up to date. XIV
Perinatal Medicine. Stuttgart: Georg Thieme. pp 21. Chan AW, Altman DG (2005) Epidemiology and Cochrane Colloquium, Dublin, Ireland.
260–265. reporting of randomised trials published in 32. Garritty C, Tsertsvadze A, Tricco AC,
10. Goldschmidt PG (1986) Information synthesis: a PubMed journals. Lancet 365: 1159–1162. Sampson M, Moher D (2010) Updating system-
practical guide. HSR: Health Services Research 22. International Committee of Medical Journal atic reviews: an international survey. PLoS ONE
21: 215–236. Editors (ICMJE). Clinical trial registration: a 5: e9914. doi:10.1371/journal.pone.0009914.