Systematic Reviews

and Meta-Analysis
Kuntjoro Harimurti
Center for Clinical Epidemiology and Evidence-Based Medicine
(CE-EBM)
Faculty of Medicine UI/Cipto Mangunkusumo Hospital
Jakarta

Reviews on medical
literatures
Overview (review article):
unsystematic

Systematic review:
in gathering, evaluating, presenting evidence
no formal statistical methods

Meta-analysis:
systematic review plus formal statistical
analysis

Review article
Systematic review
Meta analysis

Integrative Literature

Systematic vs. Unsystematic

Reviews
Unsystematic reviews
Not focused on one
interested topic
broader
Mechanisms of disease
No criteria for selection
of articles to be reviewed
bias
No assessment of
selected articles
Low applicability in
clinical practice

Systematic
Focused onreviews
only one or
two topics of interest
Diagnosis, etiology,
prognosis, and
treatment of disease
Explicit methods and
criteria for selection
(database, keywords,
types of articles)
Assessment of selected
articles
Highly applicable

Narrative/traditional reviews
(unsystematic)

Narrative/traditional reviews
(unsystematic)

Systematic reviews

Systematic review/Metaanalysis

Why
Systematic
Information
Reviews?
overload
Not all studies in
journals are good
in quality
(valid/unbiased)
Result from many
studies not
conclusive
(controversies
exist)

Systematically
search
Systematically
assessed the quality
of included studies
Systematically
combined to arrive at
conclusion

Medical evidence increasing at

epidemic rates:
we all need EBP skills to keep up-todate

MEDLINE 2010

2,000 articles / day

approx 75
new
trials
published
every day

Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

The research-toawareness gap

5,000?
per day

55 per
day

1,500
per day

Streptokinase for Myocardial

Infarction

Prophylactic antiarrhythmic drug

therapy in acute myocardial infarction
Teo et al (JAMA, 1993)
A systematic review of 51 RCTs of antiarrhythmic drugs in heart attack
involving 23,229 patients
660 deaths in patients allocated drugs
(lidocaine)
571 deaths in patients allocated to
control
89 deaths attributable to drugs

Deadly medicine:
why tens of thousands of heart
patients died in America's worst
drug disaster

At the peak of their use in the late 1980s, it

has been estimated that these drugs killed
asm many Americans every year as were
killed during the whole of Vietnam war.
The vast majority of the victims of these
drugs were treated outside controlled trials

Thomas J Moore, 1995

Steps on Systematic
Reviews
Formulating answerable question (PICO)
Extensive and comprehensive searching the
evidence
2 or more major databases + handsearching
Specific keywords
Selection criteria (inclusion and exclusion): types of
(primary) articles, year of publication, language

Critically appraise the evidence quality

assessment
2 or more assessors

Synthesis and analysis meta-analysis

Discuss and conclude the results

Components of SR/MA
Introduction
Methods
Results
Discussions and Conclusions

Introduction
State the reasons why the review is
needed
Based on problems in clinical setting
(high volume, high risk, high cost)

Any controversies in the literatures?

In treatment, choice of diagnostic
modalities, determination of
prognosis
Ended with objective of SR based on
clinical question (PICO)

Methods
How the individual studies searched
and selected
How to appraise/assess the individual
studies
How to combine (synthesis)

Searching the literatures

A comprehensive and reproducible literature
search is the foundation of a systematic
review
Search for published studies at least in
MEDLINE, EMBASE, and CINAHL.
Search for unpublished clinical trials in the
Cochrane Central Register of Controlled Trials
(CENTRAL) or other clinical trial database
(
www.clinicaltrial.gov)
Secondary/hand searching
No limitation on years and languages

Literature Search
Challenges
Database Bias - No single database is
likely to contain all published studies on a
given subject.
Publication Bias - selective publication of
articles that show positive treatment of
effects and statistical significance.
Hence, it is important to search for
unpublished studies through a manual
search of conference proceedings,
correspondence with experts, and a

Literature Review Challenges

(cont.)
English-language bias - occurs when
reviewers exclude papers published in
languages other than English
Citation bias - occurs when studies with
significant or positive results are
referenced in other publications, compared
with studies with inconclusive or negative
findings

Quality Assessment
The validity of a systematic review ultimately
depends on the scientific method of the
retrieved studies and the reporting of data
In systematic review to assess treatment
effect, RCTs are considered to be more
rigorous than observational studies
A review based on well-designed RCT will likely be
more valid and accurate than a review based on
observational studies or case reports

Quality Assessment
(cont.)
Quality assessment should be
performed by at least two assessors
The most common way to assess and
report study quality has been using a
composite, numerical scoring
instrument (Exp. Jadad score)

Jadad score

Randomization (2 points possible)

1 point if study described as randomized
Add 1 point if randomization method described and
appropriate (e.g. random numbers generated)
Deduct 1 point if randomization described and
inappropriate

Double-blinding (2 points possible)

1 point if study described as double-blinded
Add 1 point if method of double-blinding described
and appropriate
Deduct 1 point if double-blinding described and
inappropriate

Withdrawals (1 point possible)

Results
Result of search strategies (flow
diagram)
Result of quality assessment
Synthesis (summarized the effects)

Flow diagram
Potentially relevant
RCTs (n= ..)
n RCTs excluded with reason
RCTs retrieved for more
detailed information (n= ..)
n RCTs excluded with reason
Potentially appropriate
RCTs (n= ..)
n RCTs excluded with reason
RCTs in meta-analysis (n=..)

Quality Assessment

Summarized the effect

Forest plot:
Effect size with CI of each study represented by
weightened square
Combined effect (and CI) represented by diamond
Vertical line (line of no effect) 1 for OR/RR,
0 for mean difference
Vertical dotted line crossing combined effect to
assess heterogeneity

Test for heterogeneity

Cochrane Chi2 (p value), I2 (degree of heterogeneity)
Random effect model vs. Fixed effect model

a-analysis of RCTs with nominal outc

Study I 1992
Study II 1994
Study III 1995
Study IV 1995
Study V 1996
Study VI 1997
Study VII1 1999
Study VIII 2000
Combined
0.1

Favor drug

10

OR = 1

Favor placebo

a-analysis of RCTs, numerical outcom

Study I 1992
Study II 1994
Study III 1995
Study IV 1995
Study V 1996
Study VI 1997
Study VII1999
Study VIII 2000
Combined
-0.3

+0.3

Xe-xc=0
Favor drug

Favor placebo

Reporting publication bias

A funnel plot is used as a way to assess
publication bias in meta-analysis

CRITICAL APPRAISAL OF
SYSTEMATIC REVIEWS/METAANALYSIS

Is the review any good?

Q-FAST appraisal
Question Does the systematic review addres
focused question (PICO)?

Finding Did the search find all relevant evidence?

Appraisal Have been the studies critically
appraised?

Synthesis Have the results been synthesised with

appropriate summary tables and plots?

Transferability of results

Thank You