The Journul @Continuing Educution in rhe Hedth Projiessions, Volume 24, pp. 20-30. Printed in the U.S.A.

Copyright 02004 The Alliance

for Continuing Medical Education, the Society for Medical Education, the Society for Academic Continuing Medical Education, and the Council
on CME, Association for Hospital Medical Education. All rights reserved.

Original Article
elearning: A Review of Internet-Based
Continuing Medical Education
Rita Wutoh, MD, MPH, Suzanne Austin Boren, MHA, and E. Andrew Balas, MD, PhD
Abstract
Introduction: The objective was to review the effrct oflnternet-based continuing medical edu-
cation (CME) interventions on physician pegormance and health care outcomes.
Methods: Data sources included searches of MEDLINE (1966 to January 20041, CINAHL
(1982 to December 2003), ACP Journal Club (1991 to JulyIAugust 2003), and the Cochrane
Database of Systematic Reviews (third quartel; 2003). Studies were included in the analyses
if they were randomized controlled trials of Internet-based education in which participants
were practicing health care professionals or health professionals in training. CME interven-
tions were categorized according to the nature of the intervention, sample size, und other
information about educational content and form&.
Results: Sixteen studies met the eligibility criteria. Six studies generated positive changes in
participant knowledge over traditional formats; t v d y three studies showed a positive change
in practices. The remainder of the studies showed no difference in knowledge levels between
Internet-based interventions and traditional formats f o r CME.
Discussion: The results demonstrate that Internet-based CME programs are just as efective
in imparting knowledge as traditional formats of CME. Little is known as to whether these
positive changes in knowledge are translated into changes in practice. Subjective reports of
change in physician behavior should be confirmed through chart review or other objective
measures. Additional studies need to be peqormed to assess how long these new learned behav-
iors could be sustained. elearning will continue to evolve as new innovations and more
interactive modes are incorporated into learning
Key Words: Computer-assisted instruction, computers, continuing medical education (CME),
continuing professional development, family physicians, graduate medical education, Internet,
randomized controlled trials

Dr. Wutoh: Assistant Professor, Howard University
College of Pharmacy, Nursing and Allied Health Sciences,
Division of Allied Health Sciences, Washington, DC;
Ms.Boren: Associate Director, Center for Health Care
Quality, School of Medicine, University of Missouri,
Columbia, Missouri; and Dr. Balm: Dean and Professor,
School of Public Health, Saint Louis University, St. Louis,
Missouri.
This study was supported in part by a grant from the
Agency for Health Care Research and Quality (ROI
HS 10472).
Reprint requests: Rita Wutoh, MD, MPH, Assistant
Professor, Howard University College of Pharmacy,
Nursing and Allied Health Sciences, Division of Allied
Health Sciences, 6th and Bryant Streets, NW, Washington,
DC 20059.
Introduction
,4lthough health professionals dutifully attain con-
1.inuing medical education (CME) credits, the
CME goal of optimizing physician performance
and outcomes has not been met.’,*Despite increases
in knowledge evidenced by testing before and
after CME activities, most CME courses fail to
change physician practices.’-3Eckdahl et al.4sug-
gest that physicians’ knowledge decreases after
graduation from medical school, and new medical
information fails to be integrated into routine
practice. These shortcomings in teaching and Prac-
tice have long been recognized and addressed by

20
 Wutoh et al.
the development of innovative and interactive
health education programs.*s5
Nontraditional formats such as problem-
based small-group learning and electronic edu-
cation have had limited success in imparting
knowledge and changing the behavior of the
health care profe~sionall-~; yet one of the latest
innovations holds promise in the area of Internet-
based CME.6 Lower costs and c ~ n v e n i e n c eare ~,~
advantages sometimes associated with Internet-
based education, but there are others. Internet-
based programs are becoming more interactive,
often using multimedia formats to transfer knowl-
edge to meet individual preferences,8-10and the
newer Internet formats for learning can comple-
ment and reinforce traditional medical teach-
ing.*,11,12 Although these approaches offer poten-
tial for improvement, there are many claims that
Internet-based CME is no more effective than tra-
ditional methods in imparting knowledge.I2,l3
There are few randomized controlled trials of
the effects of computer-assisted instruction in
medical education; most studies are demonstra-
tions or comparative r e p o r t ~ . ' ~ J ~ J ~
In the United States, access to technology has
increased dramatically during the past 20 years,2
and the learning formats of health professionals
have slowly changed with this access. In 2001, only
2.7% of physicians used the Internet for CME.16
More recently, on-line use by physicians was esti-
mated at 31 %, with CME the second most common
use of the Internet and younger female physicians
the most active category of physician-learners.
Physicians are searching the Internet to solve
patient pr0b1ems.I~If Internet programs are not
effective, then ways for improvement need to be
identified. The question herein is whether Internet-
based CME increases the level of knowledge over
traditional formats and whether Internet-based
formats of leaming can be more effective in chang-
ing behavior.
The literature offers few review articles found
on Internet or distance learning for medical edu-
~ a t i o n . ' ~Those
, ' ~ that are available indicate that dis-
tance learning courses compare favorably with
21
classroom instruction and on-line participants
express high satisfaction.12Few studies examine
whether on-line programs change health profes-
sional behavior.I2In addition, few studies focus on
individual learning preterences; instead, they
group and average participant commentary and
scores, and there is little documentation of gender
and racial differences in regard to learning pref-
erences and computer use with regard to contin-
uing In addition and contrasting
with the recent Harris et a1.18 report that young
women are surpassing men in their use of on-line
CME, Bernhardt et al.19 revealed no association
between age, sex, or race/ethnicity and a health pro-
fessional's perceived effects of a World Wide
Web-based continuing education program. The
health professionals working in a university sys-
tem preferred traditional classroom learning for-
mats over Web-based continuing education. l 9
Practical Significance
With a dearth of randomized controlled trials and
a lack of individualized learning
there are many gaps in the research of Internet-
based CME. There is a need for more analysis of
current programs and randomized controlled tri-
als to determine whether current versions of on-
line CME are effective in changing behavior to
ensure provision of the latest standards of care. This
article reviews scientific studies of Internet-based
CME to answer three questions: Are Internet-
based CME programs effective? Are Internet-
based CME programs more effective than formal
CME? What is particularly effective within
Internet-based CME in changing physician behav-
ior or patient outcomes?
Methods
In the systematic review of the literature, three
major sources of original research were explored.
These sources included randomized controlled
trials, meta-analyses of randomized controlled tri-
als, and pre/post studies of clinical interventions.
 Internet-Based Continuing Medical Education
Eligibility Criteria
The eligibility criteria for studies in this review were
(1) randomized controlled clinical trials, meta-
analysis, or retrospective study comparing the out-
comes of specific educational interventions and
control interventions in a group of health care pro-
fessionals and (2) focus on Internet, Web, or soft-
ware applications for the education and training of
health care professionals in the areas of nursing,
dentistry, pharniacy, allied health, or medicine.
Data Sources
Systematic database searches were conducted
electronically to identify eligible studies. Initially,
MEDLINE (1966 to January 2004), CINAHL
(1982 to December 2003), ACP Journal Club
(1991 to July/August 2003), and the Cochrane
Database of Systematic Reviews (third quarter,
2003) were searched. Each Internet-related word
and health care specialty, including the following
words, was combined without restriction to lan-
guage: education, graduate education, continuing
education, nursing education, graduate nursing
education, continuing nursing education, medical
education, graduate medical education, continu-
ing medical education, pharmacy education, grad-
uate pharmacy education, continuing pharmacy
education, dental education, graduate dental edu-
cation, continuing dental education, health edu-
cation, hospital education department, distance
education, Internet, Web, or software. Manual
searches were not performed.
Study Selection and Evaluation
We included only those articles that met the fol-
lowing criteria: randomized controlled trial, meta-
analysis, or retrospective study of Internet- or
Web-based educational interventions (such as
electronic mail [e-mail] and curriculum modules);
studies that did not coerce participants; and par-
ticipants who were practicing health care profes-
sionals or health care professionals in training.
22
Data Extraction
We used a standardized abstraction format to col-
lect data from each article that met the eligibility
criteria. For each trial, the abstractor noted the inter-
vention (persons targeted, timing and periodicity,
rules shaping the intervention), sample, primary
measures of effect, and reported differences in
process and outcome (observed ratios for the inter-
vention and control groups, p values). Studies
were analyzed to assess which intervention
reported a positive outcome or improved perfor-
mance or indicated negative or inconclusive results.
The analyzed interventions were grouped into one
of the following three categories, according to
intervention type: e-mail, World Wide Web, and
hybrid (CD-ROM plus Web).
Results
Literature searches identified 86 studies; however,
70 studies did not meet OUT criteria for inclusion.
Twelve studies focused on patient education,
40 studies used computer software, and 18 stud-
ies were not an eligible study design. A total of 16
studies met all of the inclusion criteria (Table 1).
Fifteen studies used an objective assessment of
knowledge through the use of pre- and post-testing
of both intervention and control groups for e-mail
or Web-based programs.11.20 Six of 16 studies tested
knowledge levels weekly or by unit (immediately)
before conducting final examinations.s~8~'2~'5~20~21
Only one study conducted post-testing4 to 6 months
after the intervention had been conducted.22
In addition to objective assessments, 10 stud-
ies included subjective evaluations as part of their
assessment.3~x~11~12~20-25 Some subjective items
included enhancement of learning through multi-
media, ease of use, and learning preferences of Web
modules over print materials or didactic instruc-
tion.10,12,23,24
In addition, negative comments were
elicited in a few studies regarding barriers to using
the e-mail or the Web, such as time constraints and
ease of use. Three studies reported compli-
10323-25
ance with Web one of these studies exam-
 Table 1 Description of Studies Included
Study, Year Intervention
Dev et al., 199915 Web with educational content
on eating disorders
Maki et al., 20002" Lecture group vs. Web group
with required activities and
mastery quizzes
Bell and Mangione, Web-based tutorial vs. self-study
20009 using content-equivalent print
materials to teach care after
myocardial infarction
Huntley and Conrad, Introduce new communications
1994'O technologies+-mail, news,
and gopher-into dermatology
course
Goldberg and Neuroscience course using a
McKhann, 200012 software application and
Internet vs. a conventional
lecture format
Chan et al.. 1999' Intervention group: discussion
group with a facilitator and
two psychiatrists vs. control
group
Didactid
Interactive/
Effects Sample Mixed
Model 2: significant differences were 61 university Interactive
seen between intervention and control students
groups on the Body Shape Questionnaire
(F = 5.78, p < .02) and the Eating Disorders
Inventory Drive for Thinness
(F = 4.29, p = .044)
Web students performed better on the unit 277 university Mixed
examinations than lecture students; performance students
on cumulative final examinations did not show
any effect of the differential use of FAQ pages
Number of question answers and guideline 166 medical Mixed
passages viewed were associated with a residents
significantly higher post-test score; viewing
graphic evidence was not associated with higher
5s.
post-test scores (p = .93); female gender and 2
prior low experience with the Web were a
correlated with significantly less use of
randomized controlled trials
16% felt computer literacy was enhanced; 88 second-year Interactive
82% appreciated access to information; medical school
42% felt time constraints; 25% found students
communication forms intimidating
Normalized test scores were over 5 points 40 university Mixed
higher for students in the multimedia students
sessions (p < .Ol)
There was no statistically significant difference 23 physicians Interactive
between the pre- and post-test scores for the
two groups (p = .5 1); there was no interaction
effect between the allocation group and time on
the MCQ scores (p = .33)
 Table 1 continued Description of Studies Included
~ ~ ~~ ~~

Didactid
Interactive/
Study, Year Intervention Effects Sample Mixed

Kemper et al., 2002' Curriculum: case-based modules, Immediate intervention group improved significantly 537 health care Interactive
listserv discussion group, more than the waiting list group on all three professionals
hypertext links to resources outcomes: knowledge (3.0 vs. 1.4; p < . O I L (RD, MD, NP, etc.)
confidence (2.6 vs. 0.6; p < .Ol), and
communication practices (0.21 vs. -0.1; p < . O l )
3
s
Marshall et al., 20018 On-line case discussion vs. More intervention participants thought they had 40 family Interactive 2n,
control group become aware of new techniques or relevant physicians 7
research (58.8%) and had made changes in their 2
practice (64.7%) than those in the comparison
group (38.5% and 30.8%, respectively) 5
Lipman et al., 20015 Didactic group vs. Internet group Case analysis grades from external reviewers were 127 medical Interactive c?s
higher in the group with Internet component vs. students a.
x
h) traditional course (3.0 vs. 2.6, respectively; p < .005)
P Immediate scores on post-tests were similar between 162 family Interxctiw
E.
s
Bell et al., 20002? Printed materials or Web-based 09
tutorial system on myocardial paper and Web groups (p > .2), but Web groups had practice and
infarction greater learning efficiency (p = .04): Web groups internal medicine %
n,
were more satisfied with learning (p < .0001); residents
3
n
after 4-6 months, knowledge of Web groups had %
!
decreased to the same extent as paper (p = .12) trl
Carr et al., 199914 Computer learning module and Post-instruction scores between intervention groups 58 medical Mixed 3
interactive small-group seminar showed no significant differences among written, students on 2
!g.
practical, or combined scores; intervention groups ENT rotations s
showed significant improved performance on written
and practical examinations; students' comments were
very positive
Mehta et al., 1 99825 Web-based oncologic curricular No statistically significant difference between the 164 second-year Mixed
modules in basic science and pretests and post-tests between the two groups, medical students
clinical correlation but improvements were seen individually between
pre- and post-tests; 50% of both cohorts reported
that the Web had enhanced their learning
 Table 1 continued Description of Studies Included

Didactid
Interactive/
Study, Year Intervention Effects Sample Mixed

Komolpis and Johnson, Orthodontic Web site was
200224 developed (RCT)
Barden et al., 2OOOZ7 Self-study vs. direct face-to-face
intervention vs. telehealth
teaching
Grundman et al.. 200026 Print vs. multimedia
Curran et al., 20OOz1 CD-ROM and Web to teach
dermatologic office procedures
No statistically significant difference between mean 103 second-year Mixed
test scores or mean test times between conventional dental students
and Web-based groups (p < .05)
Post-tests showed no significant differences between 42 physical Mixed
didactic and telehealth students, but significant therapists
differences were seen between self-study and
telehealth groups and self-study and didactic groups
for all five skills (p < .05)
Multimedia group: students spent more time on 121 first-year Mixed
material and after analysis still performed better medical students
than those using print materials (p < .001); there was
a correlation between the eye information and time
spent on material (r2 = .61, p < .0001); 78% students
preferred multimedia version to print and thought 2
their learning was more effective with multimedia
Physicians in experimental groups using computer- 52 physicians Interactive
K
mediated instruction performed significantly better
on knowledge evaluations of dermatologic office
procedures (p = .OOO); participants were satisfied
with the instruction

ENT = otorhinolaryngology; FAQ = frequently asked questions; MCQ = multiple-choice questions; MD = medical doctor; NP = nurse practitioner;
RCT = randomized controlled trial; R D = registered dietitian.
 Internet-Based Continuing Medical Education
ined the use of rewards and compliance to assess
motivation of the ~ t u d e n tThis
. ~ study also exam-
ined when the study aid was used in relation to the
final e~arnination.~ The assessment provided some
insight as to the level of comfort with the study aid
before examinations, utility of the study aid, and
the motivation for use of the study aid.
Fourteen studies were randomized clinical
trials and two studies were modified quasi-exper-
iments.20%21 The overwhelming majority of these
studies were conducted among professional stu-
dents, but three studies involved physicians as
participants.2*'x2'Only one study examined the
correlation of Web use with gender and previous
low experience with the Web.9 This study found
female gender and low previous Web experience
to be negatively correlated with use of the Inter-
net intervention9 In addition, three studies exam-
ined whether participants believed that Web
instruction had changed their practice behav-
iors.',8,21Marshall et al.x found that 64.7% of on-
line case discussion participants believed that they
had changed their practice behaviors compared
with 30.8% of control group participants. Kemper
et a1.l reported that intervention groups achieved
significant improvements in communication prac-
tices compared with controls.
E-mail
Only three studies used e-mail or listserv for CME
interventions.2.8,'0The first study, conducted in
1994, was designed to introduce electronic com-
munication tools to medical students. The Inter-
net tools used were e-mail, newsgroup, and gopher
(a bulletin board interface).1° E-mail was best
used to communicate on a one-to-one basis
between student and teacher. Newsgroup pro-
vided the dissemination of notes, announcements,
class quizzes, etc. to be posted for several indi-
viduals to access. Lastly, gopher was used for
posting course schedules, previous examinations,
syllabi, and any other material for extended peri-
ods of time. Ninety percent of participants used
their e-mail accounts. From the 63% of question-
26
naires completed at the last lecture, 93% had acti-
vated their e-mail; of those, 87% used their
e-mail and 81% used news and only 65% used
gopher. However, on written commentary, the
majority of respondents indicated an apprecia-
tion for access to information. Seven of 44 felt that
they had enhanced computer literacy. When ques-
tioned on the negative aspects of using Internet
tools, respondents cited time constraints, intimi-
dation, and difficulty getting started."'
Four years later, in 1998, a trial using prob-
lem-based small-group learning via the Internet
was conducted with family physicians. Twenty-
three family physicians participated in the inter-
vention using a listserv service. These physicians
had been in practice for an average of 13 years,
with only 1.3 years experience using the Web.
There was no statistically significant difference
between the intervention group and the control
group. Moreover, there was no statistically sig-
nificant difference between the pretests and the
post-tests (no difference with the intervention).
Finally, there was no significant interaction effect
for group allocation on post-test scores. How-
ever, this study demonstrated that it was feasible
and inexpensive to organize problem-based small-
group learning using the Internet.8
After another 3 years, a similar study was
conducted using family physicians who had
already been using a listserv to communicate on
various medical issues. This study used on-line case
discussions over the course of 2 weeks per case
to get family practitioners to access the latest
research and to change their practices in preven-
tive medicine. The intervention group tended to be
younger, with an average of 3 years less in prac-
tice. On post-testing, more intervention group
participants had accurate knowledge on seven of
eight items. Both groups improved their preven-
tive measure screening over the course of the
intervention; however, intervention group mem-
bers had a greater sense that they were aware of
newer techniques and research, and 64.7% felt
that they had made changes in their practice com-
pared with the control group.*
 Wutoh et al.
World Wide Web
The rest of the studies, 13 in all, incorporated the
Web into the curricula for health professional stu-
d e n t ~ . ~ , ~ ,One ~ ~ study,
, ~ ~ in
, ~1999,
~ , ~initially
~ - ~ ~
used CD-ROM multimedia programs, but the
program was later converted to the Web with edu-
cational content on eating disorder^.'^ In Dev et
al.'sls study, participants were female students at
risk of developing eating disorders from two large
universities on the west coast. The study used a
combination of media such as newsgroup, edu-
cational materials, individual exercises, and a
journal over an 8- to 10-week period for self-
monitoring of attitudes and behaviors to create a
supportive community of young women and to
change behavior that could place a young woman
at risk of developing an eating disorder. The first
model using a CD-ROM showed a small statisti-
cally significant risk reduction and a compliance
of 53%. Compliance was low because the process
of accessing the program was cumbersome owing
to low numbers of computers with CD drives and
having to download the program each time. l5
Models 2 through 4 used converted versions of
the program on the Web. Ease of use improved
dramatically as communication occurred via Web-
based newsgroups. Thus, compliance with the pro-
gram increased to 63.5%. In addition, significant dif-
ferences were seen between intervention and control
groups at baseline and follow-up psychiatric mea-
sures on the Body Shape Questionnaire and the
Eating Disorders Inventory. Model 3 implemented
a more structured Web intervention than model 2;
readings, postings, and journal entries were required.
However, compliance with newsgroups require-
ments was higher than in model 2. Initial analyses
showed decreases in weight concerns and disordered
eating behaviors and attitudes.15
Web versus Print
Six randomized controlled trials evaluated Web
interventionscompared with print materials for con-
veying knowledge about a myriad of medical top-
27
i c ~ . ~ Three , ~ ~of ,the~ six
~ studies
, ~ ~ found
~ ~ that
Web-based interventions allowed participants to
score higher on post-tests than those who used print
material^.^*^^^^*^ The results on learning efficiency,
however, were inconclusive. Two of the studies
showed that learning efficiency was higher in the
Web intervention groups.22,28 Another study found
that the Web group spent more time on the
A third study found no difference between the Web
and print groups with regard to time spent learning.%
On written commentaries by participants, Web-
based interventions were generally ranked higher in
value than paper materials in three s t ~ d i e s . ~ ~ ~ ~ ~
Many participants using Web interventions pre-
ferred multimedia versions and felt a greater satis-
faction with Web learning in three s t ~ d i e s . ~ ~ , ~ ~ , ~ ~
Web versus Didactic
Six studies compared Web-based programs with
didactic or traditional lecture fonnats.5~12,'4,20,2"25,27
Only one study showed a statistically significant
overall or final positive advantage of a Web-based
learning program over a didactic or lecture for-
mat.I2This study showed normalized virtual learn-
ing environment scores being consistently higher
than mean normalized conventional lecture hall
scores for all lectures (p < .01).12Atwo-wayanaly-
sis of variance demonstrated an effect for the type
of lecture format received [F(1,144) = 6.696,
p < .01].12In three other studies, a positive outcome
of Web courses over didactic instruction was seen
on a unit b a s i ~ ~ however,
~ ' ~ . ~ ~on; final examina-
tions, the advantage was lessened, and both inter-
ventions were equally effective in improving per-
formance on final examinations.5~12~14~20~2s~27
Subjective commentary from all studles was favor-
able toward Web intervention^.^^^^^^^,^^,^^,^^ Stu-
dents remarked that Web courses were more effec-
tive and that Web courses enhanced
Discussion
Several randomized controlled trials of Internet-
based CME programs show that they are as effec-
 Internet-Based Continuing Medical Education
Lessons for Practice
Internet-based CME programs are as
effective as selected other CME in
imparting knowledge.
Little is known as to whether positive
changes in knowledge are translated
into changes in practice.
Additional studies need to be per-
formed to assess how long newly
learned behaviors are sustained.
tive as selected other CME in transferring knowl-
edge to the health provider.2,3~s~14~20~22-2s~27
Three
studies report participants feeling that they had, as
a result, changed their practice behavior^.^,^,^'
Another study, however, indicated that changes in
the behavior of participants were not sustained
beyond 4 to 6 months after the intervention.22The
behaviors and knowledge levels of participants
decreased to the levels of traditional didactic for- should provide the user with medical information
mats.22This brings the following questions to
mind: Can Internet-based CME programs change
behavior? Do practitioners need to enrol in mini-
residencies to have sustained change in behavior?
It is possible that as Internet-based CME pro-
grams apply the same curricula as traditional for-
mats of CME, they apply the same deficiencies,
rendering both programs ineffective. As such,
new, innovative, interactive programs need to be
created and tested to see if they are effective.
Several influences drive CME. These influ-
ences need to be examined if change is to be
implemented and Internet-based CME is to gain
larger support. For example, attending meetings
is still the most preferred format for physicians
obtaining CME in the United States.I6There are
social, economic, and marketing influences
involved in the domination of this format. These
28
influences must be examined to counter them and
promote more effective formats of learning.
There are three steps to undertake in examining
Internet-based CME:
1. Examine what has been successful in tra-
ditional formats of CME and incorporate
them into Internet-based CME.
2. Perform more studies using sound scien-
tific design (e.g., randomized controlled
trials) to examine if Internet-based pro-
grams are effective in changing physician
behavior using objective measures of
assessment.
3. Design additional Internet-based educa-
tional programs that cater to individual
needs based on market analysis.
There are many advantages to accessing
Internet-based CME programs. Aggressive mar-
keting needs to be initiated to encourage more
health professionals to overcome barriers to tech-
nology. Moreover, Internet programs should be
customized to the individual; Internet-based CME
of interest to the user in a format that suits the user.
In the customization of services and medical infor-
mation for the health care provider, it may be pos-
sible to increase knowledge and change behavior
to produce the best outcome for the patient.
References
1. Davis D, Thomson O’Brien M, Freemantle N,
Wolf F, Mazmanian P, Taylor-Vaisey A.
Impact of formal education. Do conferences,
workshops, rounds and other traditional con-
tinuing education activities change physician
behavior or health care outcomes? JAMA
1999; 282(9):867-874.
2. Chan D, Leclair K, Kaczorowski J. Problem-
based small group learning via the Internet
among community family physicians: a ran-
 Wutoh et al.
domized controlled trial. MD Computing
1999; 16(3):54-58.
3. Kemper K, Amata-Kynvi A, Sanghavi D,
Whelan J, Dvorkin L, Woolf A, Samuels R,
Hibberd P. Randomized trial of an Internet
curriculum on herbs and other dietary supple-
ments for health care professionals. Acad Med
2002; 77(9):882-889.
4. Eckdahl C, Karlsson D, Wigertz 0, Forsum U.
A study of the usage of a decision-support
system for infective endocarditis. Med Inf
Internet Med 2000; 25( 1):1-1 8.
5. Lipman A, Sade R, Glotzbach A, Lancaster C,
Marshall M. The incremental value of
Internet-based instruction as adjunct to class-
room instruction: a prospective randomized
study. Acad Med 2001; 76(10):1060-1064.
6. Slotte V, Wangel M, Lonka K. Information
technology in medical education: a nationwide
project on the opportunities of the new tech-
nology. Med Educ 2001; 35:990-995.
7. Maki WS, Maki RH. Evaluation of a Web-
based introductory psychology course: 11.
Contingency management to increase use of
on-line study aids. Behav Res Methods
Instrum Computers 2000; 32(2):240-245.
8. Marshall JN, Stewart M, Ostbye T. Small-
group CME using e-mail discussions. Can it
work? Can Fam Physician 2001; 4737-563.
9. Bell DS, Mangione C. Design and analysis of
a Web-based guideline tutorial system that
emphasizes clinical trial evidence. Proc AMIA
Annu Symp 2000; 56-60.
10. Huntley AC, Conrad SJ. Internet tools in the
medical classroom. Med Educ 1994;
28508-5 12.
11. Merisotis JP, Phipps RA. What’s the differ-
ence? Outcomes of distance vs. traditional
classroom-based learning. Change 1999;
3 1:13- 17.
12. Goldberg HR, McKhann GM. Student test
scores are improved in a virtual learning envi-
ronment. Adv Physiol Educ 2000;
23( 1):59-66.
13. Adler MD, Johnson KB. Quantifying the liter-
ature of computer-aided instruction in medical
29
education. Acad Med 2000;
75( 10):1025-1028.
14. Carr MM, Reznick RK, Brown DH.
Comparison of computer-assisted instruction
and seminar instruction to acquire psychomo-
tor and cognitive knowledge of epistaxis man-
agement. Otolaryngol Head Neck Surg 1999;
121(4):430-434.
15. Dev P, Winzelberg AJ, Celio A, Taylor CB.
Student bodies: psycho-education communi-
ties on the Web. Proc AMIA Annu Symp
1999; 510-514.
16. Brown TT, Proctor SE, Sinkowitz-Cochran
RL, Smith TL, Jarvis WR. Physician prefer-
ences for continuing medical education with a
focus on the topic of antimicrobial resistance:
Society for Healthcare Epidemiology of
America. Infect Control Hosp Epidemiol
2001 ; 22:656-660.
17. Casebeer L, Bennett N, Kristofco R, Carillo
A, Centor R. Physician Internet medical infor-
mation seeking and on-line continuing educa-
tion use patterns. J Contin Educ Health Prof
2002; 22:33-42.
18. Harris JM, Novalis-Marine C, Harris R.
Women physicians are early adopters of on-
line continuing medical education. J Contin
Educ Health Prof 2003; 23:221-228.
19. Bernhardt JM, Runyan CW, Bou-Saada I,
Felter EM. Implementation and evaluation of
a Web-based continuing education course in
injury prevention and control. Health
Promotion Pract 2003; 4(2): 120-1 28.
20. Maki RH, Maki WS, Patterson M, Whittaker
PD. Evaluation of a Web-based introductory
psychology course: I. Learning and satisfac-
tion in on-line versus lecture courses. Behav
Res Methods Instrum Computers 2000;
32(2):230-239.
21. Curran VR, Hoekman T, Gulliver W, Landells
I, Hatcher L. Web-based continuing medical
education (11): evaluation study of computer-
mediated continuing medical education. J
Contin Educ Health Prof 2000; 20:106-119.
22. Bell DS, Fonarow GC, Hays RD, Mangione
CM. Self-study from Web-based and printed
guideline materials. A randomized, controlled
 Internet-Based Continuing Medical Education
trial among resident physicians. Ann Intern
Med 2000; 132(12):938-946.
23. Baumlin KM, Bessete MJ, Lewis C ,
Richardson LD. EMCyberSchool: an evalua-
tion of computer-assisted instruction on the
internet. Acad Emerg Med 2000;
7(8):959-962.
24. Komolpis R, Johnson RA. Web-based ortho-
dontic instruction and assessment, J Dent
Educ 2002; 66(5):650-658.
25. Mehta MP, Sinha P, Kanwar K, Inman A,
Albanese M, Fahl W. Evaluation of Internet-
based oncologic teaching for medical students.
J Cancer Educ 1998; 13(4):197-202.
30
26. Grundman J, Wigton R, Nickol D. A con-
trolled trial of an interactive, Web-based virtu-
al reality program for teaching physical diag-
nosis skills to medical students. Acad Med
2000; 75(0ct S~ppl):S47-S49.
27. Barden W, Clarke H, Young N, McKee N,
Regehr G. Effectiveness of telehealth for
teaching specialized hand-assessment tech-
niques to physical therapists. Acad Med 2000;
75( 1O):S43-S46.
28. Papa FJ, Aldrich D, Schumacker RE. The
effects of immediate online feedback upon
diagnostic performance. Acad Med 1999;
74( 10):S 16-S 18.