You are on page 1of 6

Educational Resource Column

University of Colorado Department of Psychiatry


Evidence-Based Medicine Educational Project

Robert E. Feinstein, M.D., Brian Rothberg, M.D., Neil Weiner, M.D.


Daniel M. Savin, M.D.

I n July 2005, the University of Colorado Department of


Psychiatry Evidence-Based Medicine (EBM) Project
began to investigate whether formal educational interven-
lative effectiveness of a curriculum with educational inter-
ventions. The goal is to significantly increase resident
knowledge, skills, and clinical use of EBM with psychiatric
tions could help residents develop a positive attitude to- outpatients. The project includes curriculum development,
ward EBM, acquire EBM knowledge and skills, and fa- implementation of four educational interventions, devel-
cilitate the daily use of EBM with patients in a psychiatric opment of two attitude questionnaires, development and
outpatient residency-training site. We developed our cur- validation of a psychiatry EBM knowledge exam, and cur-
riculum and teaching approach by reviewing the world lit- riculum evaluation using a pre/post-intervention design.
erature, three published EBM curricula (1–3), and addi- This article describes the curriculum, study instruments,
tional EBM curricula presented at American Association and preliminary evidence about the program’s effective-
of Directors of Psychiatric Residency Training (AADPRT) ness using data from 37 psychiatry residents who volun-
meetings. In December 2006, we found no published com- tarily enrolled in the study.
prehensive EBM curricula from any specialties reporting
effectiveness data. Psychiatry-EBM Curriculum
Two common approaches to EBM education found in The resident curriculum is progressive over 3 years
the literature include the use of problem/case-based learn- (postgraduate years 1–3) and follows a traditional EBM
ing methods (4–9) and adult learning theory (10). These approach of teaching the 6As (Assess a patient, Ask
approaches emphasize active learning to link knowledge
and skill with clinical practice. In 1995, a systematic review TABLE 1. Clinically Integrated Teaching Methods which
of 102 trials (8) revealed that didactic approaches had little Improve Attitudes, Knowledge, and Use of EBM in
Clinical Practice
effect on physicians’ clinical practice. A recent 2004 meta-
analysis (9) reviewing 18 studies using standalone EBM ‘‘Real time’’ question formulation and literature searching
teaching methods demonstrated that these methods im- (Coomarasamy et al, 2004)
proved EBM knowledge, but failed to change attitudes to- EBM daily ward teaching rounds (Coomarasamy et al, 2004)
EBM journal club based on queries from the ward or
ward EBM or foster clinical use of EBM with patients. outpatient clinics (Coomarasamy et al, 2004; Hatala et
Table 1 reviews empirically validated, clinically integrated al, 2006)
teaching methods (9, 11–14), utilized by nonpsychiatric EBM teaching rounds based on case presented in clinical
rounds (Coomarasamy et al, 2004)
colleagues, fostering clinical use of EBM.
8-week session based on problems encountered in clinical
We believe our study may be unique as the first pro- practice (Coomarasamy et al, 2004)
spective psychiatry study designed to research the cumu- 2-hour EBM ward rounds, every other week, based on
patients currently being treated (Coomarasamy et al,
2004)
Received February 18, 2007; revised June 22 and September 2, 2007; 2-week internal medicine EBM elective rotation (Akl et al,
accepted October 1, 2007. The authors are affiliated with the De- 2004)
partment of Psychiatry at the University of Colorado Health Sciences ‘‘Real time’’ evidence-based general medical attending month
Center. Address correspondence to Robert E. Feinstein, M.D., Se- (Korenstein et al, 2002; McGinn et al, 2002)
nior Associate Dean of Education, University of Colorado Denver EBM in morning report (Hatala et al, 2006)
School of Medicine, Mail Stop F523, 13001 East 17th Place, Room
E1330, Aurora, CO 80045; Robert.Feinstein@ucdenver.edu (e-mail). EBM⳱evidence-based medicine
Copyright 䊚 2008 Academic Psychiatry

Academic Psychiatry, 32:6, November-December 2008 http://ap.psychiatryonline.org 525


EVIDENCE-BASED MEDICINE EDUCATIONAL PROJECT

a clinical question, Acquire the information, Appraise the dex case.” In the subsequent hours, the resident discusses
information, Apply the information with a patient, Assess with faculty any knowledge gaps in using the 6As. Together
the outcome with the patient) (3–5). Residents learn and they set individualized learning goals. By the end of this
practice assessing a patient, asking an EBM question using session, the resident will have selected an outcome mea-
the PICO-QQ format (Population, Interventions, Com- sure, from a rating scale book (17) or CD, which will be
parison group, Outcome, Question type, Quality of the used with the index patient. The resident is also introduced
study), acquiring the information using online library to the EBM Reminder Survey and completes the first sur-
searching of textbooks and the world literature, appraising vey detailing usage of EBM practices during the session.
the literature using EBM mathematics, applying the liter-
ature with a patient, and assessing patient outcomes. All Intervention 2: Outcome Measure at the Index Pa-
courses are highly interactive and use computers, mini-lec- tient’s Second Visit. During the second visit with the
tures, problem/case-based learning, and resident teaching. index patient, the resident explains, negotiates, and begins
The PGY-1 course consists of three weekly, 2-hour ses- use of an outcome measure with the patient. Using the
sions, designed to introduce EBM and create interest in EBM Reminder Survey, the resident details EBM activi-
the 3As (Assess a patient, Ask a question, Acquire the ties during this and all subsequent patient visits.
information), while avoiding EBM mathematics. The con-
Intervention 3: Four Additional Resident Cases:
tent addresses the question, “What is EBM and why use
EBM Reminder Surveys. After completing the initial
it?” as well as assessing, asking, and acquiring articles from
index patient visit, the resident chooses four additional
online searches.
cases. A staff member from medical records attaches the
The PGY-2 course meets weekly and consecutively for
EBM Reminder Survey to the front of the additional
6 hours. Two sessions are devoted to practicing the 3As.
charts for the next five visits of each patient. After each
Residents choose one randomized controlled trial to eval-
patient visit, the resident completes a survey. Over many
uate. Three sessions focus on critical appraisal skills and
months, each resident can complete a total of 25 surveys.
an introduction to EBM mathematics (1–6). The last ses-
sion covers outcomes and application of information with Intervention 4: 1-Minute-Preceptor. Once PGY-3
patients, with consideration of patient values and prefer- residents begin working with their index cases, four EBM
ences. attendings, who precept all PGY-3 residents, begin using
The PGY-3 course consists of 25 75-minute classes. The a 1-minute preceptor “microskills” (14, 18, 19) approach,
first 15 sessions solidify use of the 6As and EBM mathe- which is supportive of EBM. The attending tries to elicit
matics. Faculty members model an ideal case for two ses- a commitment from the resident regarding a case formu-
sions. In 13 sessions, residents present their own cases to lation and treatment plan, probe the resident’s thinking
practice and teach all 6As. With faculty support, residents and evidence supporting all decisions, teach something
lead critical appraisals of therapy, use EBM mathematics, new about EBM, reinforce what was done correctly, and
and learn to use outcome measures. The remaining 10 ses- correct mistakes.
sions use the 6As with practice guidelines, systematic re-
views, and a wider variety of outcome measures. Practice Environment for Implementation
guidelines are critically appraised, using the AGREE (15) A critical component of curriculum implementation in-
assessment. Meta-analyses/systematic reviews are critically volves availability of faculty knowledgeable in EBM. With
appraised using guidelines described elsewhere (4–6, 16). only one expert EBM teacher, we formed a four-member
EBM faculty self-teaching group, designed to help all fac-
Clinically Integrated Teaching Activities ulty learn EBM and prepare to co-teach EBM courses. We
Use of EBM in daily psychiatric care is strongly empha- also scored exams and supervised in the outpatient clinic
sized in the PGY-3 outpatient year. Midway into the PGY- using microskills. These attendings all work in the outpa-
3 course, we add four educational interventions designed tient department, a major teaching site for all residents.
to facilitate residents’ clinical use of EBM. Close resident-faculty working relationships may have in-
fluenced the 100% willingness of 37 eligible residents to
Intervention 1: Index Case and EBM Reminder Sur- enroll, with only one resident opting out of the clinical
vey. Residents in PGY-3 participate in a 3-hour session. component of the study. Senior psychiatric research and
In the first hour, the resident sees a patient called the “in- clinical faculty volunteered to take, and completed, an

526 http://ap.psychiatryonline.org Academic Psychiatry, 32:6, November-December 2008


FEINSTEIN ET AL.

early version of the EBM exam, revealing a very receptive consultation with Dr. Ramous. The 14 open-ended ques-
environment. tions attempt to reproduce the 6As clinical thinking used
during a patient encounter. The exam is administered to
Curriculum Study residents with an open time frame of 60–120 minutes.
Based on the literature and prior experience teaching To validate the exam and scoring rubric, at random we
EBM to residents, we assumed that residents would have chose 8 out of 20 exams previously completed by PGY-4
little prior exposure to EBM. We hypothesized that atti- or PGY-5 child fellows not enrolled in the study but re-
tudes toward EBM and EBM knowledge, as measured by ceiving EBM instruction from the first author. Initially, rat-
the Colorado Psychiatry Evidenced-Based Medicine Ex- ers independently scored the same exam and compared
amination (CP-EBM Exam), would increase progressively results on each of 14 items. We discussed all items on
with this curriculum. After six sessions, we expected resi- which our scorings disagreed. After three rounds, using
dents would do more searching using PICO-QQ. After 25 three different exams, we agreed on all answers. Interrater
sessions, we hypothesized that R3 scores on the CP-EBM reliability was obtained using our four investigators, scor-
Exam would substantially increase. Furthermore, we hy- ing two rounds of five additional exams. Raters were
pothesized that application of EBM with patients would blinded to resident and resident year. One common exam
not significantly increase until PGY-3 residents had a 3- was embedded in each group of five exams for each rater,
hour teaching session, saw patients, completed the EBM and raters were blinded to the common exams. Reliability
Reminder Survey, and received feedback from attendings was estimated using average interrater correlations on the
using the 1-minute preceptor. To evaluate these hypothe- total score and by the intraclass correlation. The interrater
ses, we conducted a prospective pre/post-intervention reliability for the two rounds of five tests was 0.95, and the
study examining changes in resident attitudes/beliefs, intraclass correlation was 0.93. Given these high reliabili-
knowledge, and clinical use of EBM at multiple time ties, we were comfortable having a single investigator score
points. resident exams. Preliminary EBM scores are reported by
number of sessions, while pre/post-improvement in scores
Instrumentation is reported in the Results section.
Three instruments were prospectively designed to assess To validate the exam, psychiatry EBM experts were
program impact: the EBM Attitudes and Beliefs Survey; identified using the AADPRT list of e-mail addresses. All
the EBM Reminder Survey, designed to prompt and track experts identified themselves as EBM teachers in their re-
resident use of EBM with patients; and an EBM knowl- spective psychiatry residency training programs. Five ex-
edge exam. perts volunteered to complete the exams via the honor sys-
The EBM Attitudes and Beliefs Survey is a 72-item tem following the same test conditions offered to residents.
questionnaire assessing prior computer use and experi- Two investigators scored the expert exams. With 226 as a
ence, prior experience with EBM, perceived importance of perfect score, four experts scored in the range of 211–221,
using EBM, perceived confidence in EBM skills, and per- and one expert scored 178.
ceived barriers or facilitators to EBM use. The attitude The CP-EBM Exam, scoring rubric, and both survey in-
questionnaire is administered before and after each course struments are available from the first author.
and after clinical teaching interventions.
The EBM Reminder Survey is placed on the front of Sample. After 1 year of implementation, preliminary
patient charts, prompting residents to use EBM during results of our attitudes and beliefs survey consist of data
clinical care. Nine questions ask residents to detail their from four resident groups, totaling 37 residents in all 4
use, or nonuse, of EBM practices after each of 25 visits. years. Table 2 details the number for each year, the number
Investigators are blinded to which residents complete the of residents who took and completed the attitudes and be-
surveys and any identifying patient information, except for liefs survey at different points in time, and the numbers
the initial index patient visit. from our sample that were available for our preliminary
The Colorado Psychiatry Evidenced-Based Medicine analysis.
Examination (CP-EBM Exam) is a 14-question exam as-
sessing EBM knowledge and skills. Based on the Fresno Analyses. Attitude questionnaire responses are sum-
Test of Competence in Evidence-Based Medicine (20), the marized in the Results section, using parametric statistics
psychiatry version and scoring rubric were developed in and pre/post analyses employing paired t tests. Given the

Academic Psychiatry, 32:6, November-December 2008 http://ap.psychiatryonline.org 527


EVIDENCE-BASED MEDICINE EDUCATIONAL PROJECT

limited sample with full data, results should be considered In the questionnaire domain of residents’ level of con-
preliminary and cautiously interpreted. fidence for performing 26 EBM skills, mean responses of
Preliminary analysis of performance on the CP-EBM residents prior to completing any sessions (PGY-1 and
Exam’s knowledge portion was based on a group of seven PGY-2), after six sessions (PGY-1–PGY-3), and after 25
PGY-2 residents. Results of these exams were analyzed by sessions (PGY-3 and PGY-4) indicate an increase in con-
comparing means across groups based on number of EBM fidence on all skills. Paired t tests were conducted to ex-
sessions attended using ANOVA. A paired t test estimated plore whether changes after six sessions (PGY-1 and PGY-
improvement in knowledge for the seven residents with 2) and between 6 and 25 sessions (PGY-3) were significant.
exam scores prior to receiving any EBM training and after For significant results, see Table 3. Because sample sizes
completing six EBM sessions. are small, these results are tentative at best.
Fourteen residents completed data prior to and after
Preliminary Results
completing six sessions. The mean reported level of con-
fidence increased significantly (meanpre⳱1.29, meanpost⳱
EBM Attitudes and Beliefs Survey 2.29, p⳱0.01) only for their ability to “construct a well-
Before any EBM session, all residents reported using formulated (six-part PICO QQ) clinical question.” Be-
the computer at least once a day at work and a few times cause only four residents had complete data after 6 and 25
a week at home. Residents in PGY-1 indicated more fre- sessions, no preliminary statistical analyses are appropri-
quent computer usage than either those in PGY-2 or PGY- ate. Even in this sample of four, an examination of areas
3 who had completed six EBM sessions. Likewise, a larger where level of confidence exceeded 2 standard deviations
percentage of PGY-1 residents had some prior experience in the pretest group, four is considered noteworthy. On
with epidemiology/statistics/EBM, some formerly working average, residents participating in 25 sessions gained more
as research assistants or researchers. than 2 standard deviations, compared with their level of
In the questionnaire domain of importance of EBM for confidence after participating in six sessions, in their level
patient care, six items were analyzed. These items assessed of confidence to perform six skills: understand the basic
reading and understanding the psychiatric literature, self- statistical concepts; clinical epidemiology and study design/
questioning about optimal patient care after each diag- methodology; determine the effectiveness of a treatment
nostic assessment and routine visit, finding and applying intervention by calculating risk/benefit numbers; write/
the pertinent literature to the patient’s care, using risk/ summarize a concise, thoughtful, one-page critical ap-
benefit calculations to make treatment decisions, and using praisal of a therapy; determine if a relevant study can be
measurable outcomes or rating scales to follow patient applied to a patient; and utilize librarians and EBM su-
progress. Paired t tests of differences in level of impor- pervisors as needed.
tance, measured on a 5-point scale (1⳱not important, This questionnaire also included 14 items about EBM
5⳱extremely important) before and after six sessions practices. Some questions were anticipated to change
(PGY-1/PGY-2) and before and after 25 sessions (PGY- based on postgraduate year alone (e.g., number of patients
3) showed no statistical differences. Residents rated that seen each day), whereas others were thought to be subject
use of the literature, skills in finding and applying the lit- to change as a result of improved EBM skills (e.g., number
erature, and self-questioning about best patient care to be of article searches each week). Comparing residents after
“very important.” Residents found using risk/benefit cal- six sessions, only the number of background searches done
culations to make treatment decisions and rating scales to each week significantly changed, with the number of lit-
follow patient progress “important.” erature searches declining. After 25 sessions, residents

TABLE 2. Number and Level of Residents Completing Attitude-Belief Survey at Various Points in Training

Posttest after Posttest after Pre/Post Total


Year in Residency n Pretest six sessions 25 sessions Scores Available
PGY-1 9 9 9 — 9
PGY-2 15 14 15 — 14
PGY-3 6 — 5 5 4
PGY-4 7 — — 7 —
Total 37 23 29 12 27

528 http://ap.psychiatryonline.org Academic Psychiatry, 32:6, November-December 2008


FEINSTEIN ET AL.

showed an increase of 2 pretest standard deviations in the The effects of our clinically integrated interventions in
“number of treatment decisions changed due to EBM in PGY-3 are awaiting further data and analysis.
the last month,” from a mean of 0.75 to 2.5 (with 0⳱none, Keeping these limitations in mind, we discuss our results
1.5⳱1–2, 3.5⳱3–4, 5.5⳱5–6, 7⳱7 or more). These results according to our initial assumptions and hypotheses. First,
are summarized in Table 3. our assessment of prior exposure to EBM confirms our
assumption that current residents in PGY-2 and PGY-3
Colorado Psychiatry EBM Exam have little experience. On the other hand, current residents
A perfect score on the Colorado Psychiatry EBM Exam in PGY-1 have greater experience with epidemiology/sta-
is 226 points. We scored 17 resident examinations. Thir- tistics/EBM and more experience working as research as-
teen residents were in PGY-2, who received six EBM sistants/researchers. We are not sure whether this is a local
teaching sessions; two residents were in PGY-4 or 5, who phenomenon or whether it reflects a national trend of in-
received 10 teaching sessions; and two residents were in creasing EBM teaching in medical school. It has clear im-
PGY-3, who received 25 teaching sessions. Descriptive sta- plications for the starting point in the curriculum of the
tistics for groups receiving six sessions (13 PGY-2), 10 ses- future, because it reinforces the need for a knowledge ex-
sions (2 PGY-4–5), and 25 sessions (2 PGY-3) indicate amination to appropriately match teaching to resident
that means increase with number of sessions, as expected needs.
(106.7 after six sessions, 144.5 after 10 sessions, and 195 Current attitudes data provide no obvious progression
after 25 sessions). This sample of 17 residents was too in perceived importance of EBM in patient care. This un-
small for statistical analyses (n⳱13, 2, and 2, respectively), anticipated result may be explained by the high level of
and therefore our data are only suggestive of trends we importance ascribed to EBM prior to participation in the
expected.
program. Changes in resident level of confidence reflect
Because we are reporting preliminary data, we only
our hypothesis with specific areas of confidence paralleling
scored seven out of the 13 PGY-2 EBM exams that were
curriculum topics.
taken. A paired t test was estimated to assess improvement
Results from our attitudes survey indicate that when
in resident knowledge after participating in six EBM ses-
combining confidence and knowledge data, six sessions ap-
sions. Of seven residents, scored with both pre- and post-
pear sufficient to introduce residents in PGY-2 to EBM
test data, performance improved significantly from a mean
and facilitate their knowledge and use of literature and
of 37.0 (SD⳱23.9) to a mean of 102.6 (SD⳱49.9), a sig-
PICO-QQ searching. Likewise, 25 sessions appear suffi-
nificant improvement (t⳱2.814, p⳱0.03).
cient to extend PGY-3 residents’ level of confidence in
their ability to understand EBM concepts/study design, de-
Discussion
termine the effectiveness of a treatment using EBM math-
Our experience suggests that an effective, comprehen- ematics, write a critical appraisal, apply EBM with a pa-
sive EBM curriculum can be developed and delivered to tient, and regularly utilize librarians and EBM supervisors
psychiatric residents. Data on the effectiveness of the cur- as needed.
riculum is generally favorable, although the limited num- Knowledge of EBM, as measured by the Colorado Psy-
ber of residents completing questionnaires and exams re- chiatry EBM Exam, indicates that even a limited, six-ses-
quires that all results be interpreted as very preliminary. sion curriculum can result in substantial improvement in

TABLE 3. EBM Attitudes and Beliefs Survey Results

Number of Sessions
Domain Year in Residency Pretest Posttest n t* p
Confidence performing six-part PICO QQ PGY-2 0 6 14 3.02 0.01
Number of background searches completed PGY-2 0 6 12 2.42 0.03
Number of treatment decisions PGY-3 6 25 4 2.11 0.12

EBM⳱evidence-based medicine
PICO QQ format⳱Population, Interventions, Comparison group, Outcome, Question type, Quality of the study
*paired t test

Academic Psychiatry, 32:6, November-December 2008 http://ap.psychiatryonline.org 529


EVIDENCE-BASED MEDICINE EDUCATIONAL PROJECT

searching skills. These trends in knowledge acquisition a systematic review of 102 trials of interventions to improve
lead us to cautiously suggest that the sequence and depth professional practice. Can Med Assoc J: 1995; 153:1423–1431
9. Coomarasamy A, Khan KS: What is the evidence that post-
of instruction appears appropriate.
graduate teaching in evidence based medicine changes any-
The final tasks and results of our efforts to facilitate thing? A systematic review. BMJ 2004; 329:1017–1021
resident use of EBM skills with patients in daily practice 10. Smith CA, Ganschow PS, Reilly BM, et al: Teaching residents
are awaiting future data on the effect of these components. evidence-based medicine skills: a controlled trial of effective-
ness and assessment of durability. J Gen Int Med 2000;
We would like to thank Gretchen Guiton, Ph.D., for her invalu- 15:710–715
able support with survey designs and statistics. 11. Akl EA, Izuchukwu IS, El-Dika S, et al: Integrating an evi-
dence-based medicine rotation into an internal medicine res-
idency program. Acad Med 2004; 79:897–904
References
12. Korenstein D, Dunn A, McGin T: Mixing it up: integrating
evidence-based medicine and patient care. Acad Med 2002;
1. Keitz SA, Owens TA, Chard C: Co-Directors: Teaching. Lead- 77:741–742
ing, Practicing EBM. Duke University, Durham, NC, April 13. McGinn T, Seltz M, Korenstein D: A method for real-time
2003. Available at http: www.mclibrary.duke.edu/limited/EBM evidence-based general medical attending rounds. Acad Med
workshop/index.html 2002; 77:1150–1152
2. How to Teach Evidence-Based Clinical Practice Workshop. 14. Hatala R, Keitz SA, Wilson MC, et al: Beyond journal clubs:
McMaster University, Hamilton, Ontario, Canada, June 13, moving toward an integrated evidence-based medicine curric-
2004. Available at http://clarity.mcmaster.ca/date_loc.php ulum. J Gen Int Med 2006; 21:538–541
3. Feinstein RE: Evidence-based medicine, in Psychosomatic 15. The AGREE Collaboration: Appraisal of Guidelines for Re-
Medicine. Edited by Blumenfield M, Strain J. Lippincott, Wil- search and Evaluation AGREE Instrument. Available at
liams & Wilkins, Philadelphia, 2006, pp 881–897 www.agreecollaboration.org
4. Sackett DL, Straus SE, Glasziou P, et al: Evidence-based med- 16. Mayer D: Meta-analysis and systematic reviews, in Essential
icine: how to practice and teach EBM, 3rd ed. Edinburgh, UK, Evidence-Based Medicine. Mayer D. Cambridge University
Churchill Livingstone, 2005 Press, Cambridge, UK, 2004, pp 319–333
5. Guyatt G, Rennie D: User’s Guide to the Medical Literature: 17. Sajatovic M, Ramirez LF: Rating Scales in Mental Health,
Essentials of Evidence-Based Clinical Practice. Chicago, 2nd ed. Hudson, Ohio, Lexi-Comp Inc, 2003
American Medical Association Press, 2002 18. Neher JO, Gordon KC, Meyer B, et al: A five step micro skills
6. Greenhalgh T: How to Read a Paper: Basics of Evidence- model of teaching. J Am Board Fam Pract 1992; 5:419–424
Based Medicine, 3rd ed. London, BMJ Books, Blackwell Sci- 19. Parrot S, Dobbie A, Chumley H, et al: Evidence-based office
entific Publishing Ltd, 2006 teaching: the five-step micro skills model of clinical teaching.
7. Green ML: Evidence-based medicine training in internal Fam Med 2006; 38:164–167
medicine residency training: a national survey. J Gen Int Med 20. Ramos KD, Schafer S, Tracz SM: Validation of the Fresno
2001; 15:129–335 test of competence in evidence-based medicine. BMJ 2003;
8. Oxman AD, Thomson MA, Davis DA, et al: No magic bullet: 326:319–321

530 http://ap.psychiatryonline.org Academic Psychiatry, 32:6, November-December 2008

You might also like