Medical evidence increasing at epidemic rates:

we all need EBP skills to keep up-to-date

Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

Medical evidence increasing at epidemic rates:

we all need EBP skills to keep up-to-date

approx 75 new
trials published
every day

Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

Medical evidence increasing at epidemic rates:

we all need EBP skills to keep up-to-date
MEDLINE 2010
2,000 articles / day

approx 75 new
trials published
every day

Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

About 10% of
published evidence
is worth reading
About 1/3 of
worthwhile
evidence is
eventually refuted
or attenuated
About 1/2 of
relevant evidence is
not implemented

Rapid critical appraisal

using GATE

Rod Jackson
University of Auckland, NZ
August 2011

Graphic Appraisal Tool for Epidemiological studies

Graphic Approach To Evidence Based Practice
Graphic Approach To Epidemiology

the GATE frame

the shape of every epidemiological study 8

British doctors

smoking status measured

smokers

Lung cancer

non-smokers

yes
no

5 years

Longitudinal (cohort) study

9

British doctors

smoking status measured

smokers

Lung function

non-smokers

normal
abnormal

Cross-sectional study
10

British doctors

Randomised to aspirin or placebo

aspirin

placebo

yes
Myocardial infarction
no

5 years

RCT
11

Middle-aged US women

Test applied
Mammogram positive

Mammogram negative

yes
Breast cancer
no

Clinical use of a diagnostic test

12

Middle-aged US women

Breast cancer

no Breast cancer

positive
mammogram
negative

Diagnostic test accuracy study

13

GATE: Graphic Appraisal Tool for

Epidemiological studies

1 picture, 2 formulas & 3 acronyms

14

14

One picture: the GATE frame

every epidemiological study hangs on the GATE frame

15

The 1st acronym = PECOT : the 5 parts

of every epidemiological study
P

Participants

Exposure Group

Time

Comparison Group

Outcomes

T
16

Lewis RT et al. Should antibiotic prophylaxis be

used routinely in clean surgical procedures: A
tentative yes. Surgery 1995;118:742-7.

17

Background. The incidence of surgical site infection (SSI)

after clean surgical procedure is regarded as too low for
routine antibiotic prophylaxis. But risk of SSI can be as high as
20%. We assessed the value of prophylactic cefotaxime in
patients stratified for risk of SSI in a double-blind RCT.
Methods. Patients having clean elective operations were
stratified for risk & randomized to receive IV cefotaxime 2 gm
or placebo before operation & followed for 4-6 weeks for SSI.
Results. The 378 of 775 patients who received cefotaxime
had 70% fewer SSIs than those who did not --Mantel-Haenszel
risk ratio (MH-RR) 0.31; 95 % CI 0.11 to 0.83. Benefit was
clear in the 616 low risk patients--0.97% versus 3.9% SSI
(MH-RR 0.25, CI 0.07 to 0.87, p = 0.018), but only a trend was
seen in 136 high risk patients--2.8% versus 6.1% SSI (MH-RR
0.48, CI 0.09 to 2.5).
Conclusions. The results indicate clear benefit for routine
antibiotic prophylaxis in clean surgical procedures. High risk
18
patients need more study.

19

1st critical appraisal task: describe studys design by

hanging on GATE frame using PECOT acronym

O
T

20

Participants
Study Setting

Eligible Participants

Participants
21

Lewis Trial

Participants
Study Setting: patients admitted to QE Hospital,
Montreal, Canada (1992-5?)
Eligible Participants: undergoing clean surgery or
simple cholecystectomy

P
Participants: 633 (?consecutive eligible patients)
22

Exposure & Comparison Groups

Exposure or
Intervention Group
(EG)

EG

CG

Comparison or
Control Group
(CG)

23

Exposure & Comparison Groups:

low risk group
633
316

Exposure or
Intervention Group
(EG):
2g cefotaxime IV preop

317

Comparison or
Control Group
(CG):
Identical placebo
IV
24

Exposure & Comparison Groups:

low risk group
633
316

317

Exposure or
Comparison or
Intervention Group
Control Group
(EG):
(CG):
308* 308*
2mg cefotaxime IV
Identical placebo
pre-op
IV
* With complete follow-up

25

Outcomes (O)

yes

Dis-ease
no

Outcomes (O)

d
26

633

Outcomes (O)
316

Surgical site
infection (SSI)

yes
no

317

a= 3 b=
12

Primary Outcome
(O)
27

Time (T)

incidence

prevalence
28

Time (T)
316

317

Outcome: SSI

incidence

T= time from
initiation of treatment
to end of follow-up

29

Study design: GATE frame & PECOT

Participants

Exposure Group

Time

Comparison Group

Outcomes
30

Lewis
Setting: QE Hospital, Montreal
Eligible: clean surgery or cholecystectomy
Participants

633
316

Exposure Group:
IV cefotaxime
Time:
Up to 6 wks post-op

317

308 308

12

Comparison Group:
IV placebo
Outcomes:
SSI
31

Questions?

32

The 1st formula: study analyses

Occurrence (risk) of disease

= Numerator Denominator
D
N

33

All epidemiological studies involve measuring

the OCCURRENCE of outcomes

Denominator (Participants)

Numerator (Outcomes)

Occ = ND

34

All epidemiological studies involve measuring

the OCCURRENCE of outcomes

Denominator (Participants)

During what period of time (T) was N

measured? (incidence)

Numerator (Outcomes)

Occ = ND (T?)

35

All epidemiological studies involve measuring

the OCCURRENCE of outcomes

Denominator (Participants)

T
At what point in time (T) was N measured?
(prevalence)

Numerator (Outcomes)

Occ = ND (T?)

36

The 1st formula:

Occurrence (risk) = Numerator Denominator

Exposed
Group

Comparison
Group

DE DC
T

NE

NC

T
37

2nd appraisal task: describe analyses by hanging

numbers on the GATE frame and calculating
occurrences in exposure & comparison groups
P
Denominator 1:
Exposure Group
(EG)
Numerator 1:
a

EG CG

a
c

Denominator 2:
Comparison Group
(CG)
Numerator 2: b

d
38

Occurrence = N D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a

EG CG

a
c

Exposure Group Occurrence:

EGO = a EG

b
d

Denominator 2:
Comparison Group
CG
Numerator 2:
b

Comparison Group Occurrence:

CGO = b CG
39

Occurrence = N D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a

EG CG

a
c

Exposure Group Occurrence:

EGO = a EG

b
d

Denominator 2:
Comparison Group
CG
Numerator 2:
b

Comparison Group Occurrence:

CGO = b CG
40

Occurrence = N D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a

EG CG

a
c

Exposure Group Occurrence:

EGO = a EG

b
d

Denominator 2:
Comparison Group
CG
Numerator 2:
b

Comparison Group Occurrence:

CGO = b CG
41

Calculate EGO & CGO for SSI in low risk group

P
Denominator 1:
Exposure Group
EG = 316
Numerator 1:
a=3

EG CG

a
c

b
d

Denominator 2:
Comparison Group
CG = 317
Numerator 2:
b = 12

EGO = 3/316= 9.5/1000 at 6

CGO = 12/317 = 37.9/1000
weeks
at 6 weeks
42
ITT (intention to treat) analysis

Calculate EGO & CGO for SSI in low risk group

P
Denominator 1:
Exposure Group
EG = 308
Numerator 1:
a=3

EG CG

a
c

b
d

Denominator 2:
Comparison Group
CG = 308
Numerator 2:
b = 12

EGO = 3/308= 9.7/1000 at 6

CGO = 12/308 =39/1000 at
weeks
6 weeks
43
OT (on treatment) or per-protocol analysis

Describing differences between

occurrences
Relative difference or Relative Risk = EGO CGO

Absolute Difference or Risk Difference = EGO - CGO

Number Needed To Treat (NNT) = 1 RD

44

Describing differences between

occurrences (SSI in low risk patients)
Relative difference or Relative Risk = EGO CGO

= 9.5/1000 37.9/1000 = 0.25

Absolute Difference or Risk Difference = EGO - CGO

= 9.5/1000 - 37.9/1000 = -28.4/1000

Number Needed To Treat (NNT) = 1 RD

= 1 (- 28.4 /1000) = - 1000/28.4 = 35

if 35 patients were given IV cefotaxime pre-op, there would be 1 fewer SSI up
to 6 weeks post-op
45

Study analyses

its all about EGO & CGO

46

Questions?

47

The 2nd acronym = RAMBO* : assessing bias

strength of study
P

Recruitment

Allocation

Maintenance

O
T
T

Blind or
Objective outcomes
measurement
48

48
* Paul Glasziou

The 2nd acronym = RAMBO* : assessing non

random error (i.e. bias)
P

Recruitment

Allocation

Maintenance

O
T
T

Blind or
Objective outcomes
measurement
49

* Paul Glasziou
49

3rd appraisal task: assess the degree of bias by

applying the RAMBO acronym
P

Recruitment

Allocation

Maintenance

O
T
T

Blind or
Objective outcomes
measurement
50

Study setting
Eligible people

P
P
E

RAMBO
were Recruitment processes
appropriate to study goals?

Study setting & eligibility criteria well

described?
e.g. Recruit random/representative sample
OR consecutive eligibles OR volunteers
from advertisements
Participants representative of eligibles?
Prognostic/risk profile appropriate to
study question?

51

RAMBO: A is for Allocation

Was Allocation to
EG & CG
successful?
RCT: Allocate randomly by
Cohort: Allocate by
investigators (e.g drugs) measurement (e.g. smoking)

EG CG

were EG & CG
similar at
baseline?

EG CG

O
52

EG CG

RAMBO

were Participants Maintained as

allocated?
did most participants remain in
allocated groups (EG & CG)
Participants &/or investigators blind to
exposure (and comparison exposure)?
Compliance high & similar in EG & CG?
Contamination low & similar in EG & CG?
Co-interventions low & similar in EG & CG?
Completeness of follow-up high & similar in EG
& CG?
53

RAMBO
Were outcomes measured
Blind or Objectively?

EG CG

O
T

If outcome measurements not

Objective (eg. automated or definitive)
were investigators Blind to exposure
(and comparison exposure)
54

The 4 (GATE) study biases

P

Recruitment bias

Allocation bias

Maintenance bias

Outcomes
Measurement bias
55

Questions?

56

The 2nd formula: assessing random error

Random error = 95% Confidence

Interval(1.96 x Standard Error)

57

4th appraisal task: assess degree of random

error in study findings using the 2 nd formula
Random error = 95% Confidence Interval
For the Outcome SSI (low risk group) :
EGO = 9.5/1000; (95% CI = 3.2 to 27.5)
CGO = 37.9/1000; (95% CI = 21.8 to 65)
EGOCGO = 0.25 (0.07 to 0.88)
EGO-CGO = -28.4 (-52 to -4.8)
NNT = -35 (-19 to -211)
58

Excel CATs & paper Gate-lites

There is a GATE for every study design

www.epiq.co.nz
59

59

Final appraisal task: search for & appraise

SRs / meta-analyses using 3rd acronym
(FAITH)

Find appropriate studies?

Appraise selected studies?
Include only valid studies?
Total-up (synthesise) appropriately?
Heterogeneity adequately addressed?
60

Systematic Reviews
There are 4 Cochrane SRs on this topic
and the findings are not consistent

61

Using GATE
as a framework for evidence
based practice

The first 4 steps of EBP

1. Ask a focused question.
2. Access (systematically search for)
epidemiological evidence to help answer question.
3. Appraise evidence found for its validity, effect
size, precision (ideally all the relevant evidence)
4. Apply the evidence:
a. amalgamate the valid evidence with other
relevant information (patient/community values,
clinical/health issues, & policy context) and make an
evidence-based decision; and
b. act (implement) the decision in practice

EBP Step 1: Ask- turn your question

into a 5-part PECOT question
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! another therapy or no treatment
Outcome (e.g. a disease you want to prevent or
manage)
Time frame (over which you expect a result)

EBP Step 2: Access the evidence use

PECOT to choose search terms
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! another therapy or no treatment
Outcome (e.g. a disease you want to prevent
or manage)
Time frame (over which you expect a result)
65

EBP Step 3: Appraise the

evidence
using the best evidence from
epidemiology to help inform decisions
more critically (using GATE)
more systematically (using FAITH)

EBP Step 4: APPLY the evidence by: a.

AMALGAMATING the relevant information & making
an evidence-based decision: the X-factor

X-factor: making evidence-based decisions

Evidence

Clinical / health
considerations

Patient / community
preferences
Policy issues

Xpertise: putting it all together the art of

practice