COMMENTARY

Safe Electronic Health Record Use Requires

a Comprehensive Monitoring
and Evaluation Framework
Dean F. Sittig, PhD tial safety hazards that result from inadequate design,
development, implementation, or use of EHRs. Koppel and
David C. Classen, MD Kreda6 state that many EHR vendors legally limit the abil­
ity of their clients to publically report these types of prob­

R
ECENT PASSAGE OF THE AMERICAN REINVESTMENT
lems. Such a national EHR hazard reporting system,
and Recovery Act (ARRA) increases pressure on
described in detail by Walker et al,5 could increase aware­
health care practitioners and organizations to imple­
ness of safety concerns among users as well as help ven­
ment currently available electronic health records
dors identify items that need their attention. This report­
(EHRs). Research and experience gained to date show that
ing system might be operationalized through the new
such implementation efforts are difficult, costly, time-
patient safety organization statute and associated Agency
consuming, and fraught with many unintended conse­
for Healthcare Research and Quality (AHRQ) common
quences.1 Evaluation of these systems after implementa­
reporting formats.8
tion suggests that they do not routinely meet safety standards
of other safety-critical industries.2 The aggressive timeline Enhanced EHR Certification
proposed in the ARRA bill means that a large number of prac­
The organizations responsible for certifying EHRs should
titioners and health care organizations will soon be attempt­
add the following types of checks to their current certifica­
ing a monumental feat without the time or ability to cus­
tion criteria. All EHR vendors should have to demonstrate
tomize these systems to their local workflows.3
that they follow good software engineering practices, in­
In a previous article, we proposed 8 dimensions or “rights”
cluding performing hazard analyses of their products, de­
of EHR safety that addressed the complex social, technical,
signing for safety, documenting these designs, and verify­
and personal issues associated with EHR use.4 The goal of
ing that their systems work as designed.7 All systems should
this article is to describe an approach for a comprehensive
have their user interfaces tested for usability. All vendors
EHR monitoring and evaluation framework (ie, the eighth
should be prepared to demonstrate that they have success­
dimension of EHR safety). Without such a comprehensive
fully addressed all critical software issues identified within
framework, safe and effective EHR use cannot be ensured.
the national hazard reporting and investigation system within
This proposed framework has 5 essential components: (1)
the previous year at any time. In addition, each EHR should
ability for practitioners and organizations to report patient safety
be “load tested” to simulate its response time when mul­
events or potential hazards related to EHR use5,6; (2) enhanced
tiple users are accessing the system simultaneously. More­
EHR certification that includes specific assurances that good
over, each vendor should be required to present data col­
software development procedures7 have been followed along
lected from multiple implementations describing their
with evidence that previously reported adverse events and haz­
system’s reliability and response time as implemented. In
ards have been addressed; (3) self-assessment, attestation, test­
summary, EHR vendors must demonstrate that their appli­
ing, and reporting by both clinicians and health care organi­
cations have been designed for safety, developed correctly,
zations that all 8 dimensions of safe EHR use have been
work as designed, and had all their defects fixed.
addressed; (4) local, state, and national oversight in the form
of an onsite, in-person accreditation of EHRs as implemented Self-assessment of EHR Use
and used by clinicians in the health care setting; and (5) a
A national, vendor-independent, federally funded and man­
national EHR-related adverse event investigation board that
aged EHR oversight organization should develop a self-
reviews incident reports and has the authority to investigate.
Author Affiliations: University of Texas at Houston—Memorial Hermann Center
Reporting EHR Safety Issues for Healthcare Quality and Safety, School of Health Information Science, Univer­
sity of Texas Health Science Center, Houston (Dr Sittig); and Department of In­
Currently, there is no clear organization or other entity for ternal Medicine, University of Utah School of Medicine, and CSC, Salt Lake City
health care practitioners to report adverse events or poten- (Dr Classen).
Corresponding Author: Dean F. Sittig, PhD, University of Texas at Houston—
Memorial Hermann Center for Healthcare Quality and Safety, University of Texas
See also p 452. School of Health Information Sciences, 6410 Fannin St, UTPB 1100.43, Houston,
TX 77030 (dean.f.sittig@uth.tmc.edu).

450 JAMA, February 3, 2010—Vol 303, No. 5 (Reprinted) ©2010 American Medical Association. All rights reserved.

Downloaded from www.jama.com at Houston Academy of Medicine on February 2, 2010


assessment guide for all individual EHR users as well as the
organizations responsible for implementing them. This guide
for users should include at a minimum approximately 25
common actions that a user should be capable of perform­
ing (eg, look up a patient by name, medical record num­
ber, or review the 3 most recent laboratory test results), the
organization’s EHR downtime and reactivation proce­
dures, and any EHR-related adverse events or potential haz­
ards the user has been directly involved in. This self-
assessment tool could be developed and implemented
similarly to the way the AHRQ/LeapFrog EHR/clinical de­
cision support assessment tool has been.9
In addition, each organization should perform and docu­
ment a more extensive review of their clinical information
systems on a yearly basis. This review should address each
of the 8 dimensions of safe EHR use that includes hard­
ware and software, clinical content, user interfaces, user train­
ing and authorization procedures, clinical workflow and com­
munication, organizational policies and procedures,
compliance with state and federal rules and regulations, and
periodic measurements of system activity.
The results of these self- and organizational assessments
should be submitted to a central clearinghouse that would
facilitate the creation and dissemination of national EHR
safety benchmarks.
Onsite Accreditation of EHRS as Implemented
and Used
Although the first 3 proposals will most likely help to sig­
nificantly improve the safety and effectiveness of EHR us­
age nationwide, there is still no substitute for periodic, un­
announced, random, onsite inspections of EHR systems.
These inspections could be conducted as part of ongoing
organization-wide, quality and patient safety accreditation
activities by organizations, such as The Joint Commission
or local state departments of health. Once again, these in­
spections would address all 8 dimensions of safe EHR use.4
Organizations could be given 6 months to address all of the
reviewers’ concerns or face significant financial penalties.
National EHR Adverse Event Investigation Board
As EHRs begin to play an increasingly important role in pa­
tient care on a large-scale, community-wide basis, the Of­
fice of the National Coordinator for Health Information Tech­
nology (ONC) could create a National EHR Adverse Event
Investigation Board, similar to the National Transporta­
tion Safety Board or the Commercial Aviation Safety Team,10
to investigate serious EHR-related adverse events or haz­
ards that involve complex, sociotechnical interactions be­
tween clinicians or patients and software systems. The find­
ings of these investigations could be reported to the ONC
and made public so that patients, other vendors, health care
organizations, and practitioners can learn from them. This
board would be provided with unlimited access to all as­
pects of the EHR, including, for example, key system back­
©2010 American Medical Association. All rights reserved.
Downloaded from www.jama.com at Houston Academy of Medicine on February 2, 2010
COMMENTARY
ups, change logs, and minutes from the local EHR safety over­
sight committees. This board would have the authority to
investigate all serious EHR-related adverse events or near
misses.
Conclusions
In conclusion, President Obama has taken an important step
toward improving the clinical computing infrastructure of
the US health care delivery system by stating the goal of all
citizens having access to an EHR. However, the extremely
aggressive timeline in the ARRA stimulus package places
enormous pressure on health care practitioners and their
organizations to rapidly implement EHRs. Such rapid imple­
mentations could lead to significant patient safety events.
To help ensure that these implementations are as safe and
effective as possible, this 5-stage proposal to create a com­
prehensive EHR monitoring and evaluation framework
should be considered. Using such a framework, the ONC,
or another designated national body, could begin to pro­
vide the oversight needed to ensure that all EHR implemen­
tations are safe and effective and to provide the mecha­
nisms to help health care organizations using these systems
to deliver the highest quality, lowest cost, and safest health
care possible.
Financial Disclosures: Dr Classen is an employee and stockholder of CSC (Salt Lake
City, Utah), a technology services company. Dr Sittig reported no financial disclo­
sures.
Funding/Support: Dr Sittig is supported in part by grant R01-LM006942 from the
National Library of Medicine.
Role of the Sponsor: The National Library of Medicine had no role in the prepa­
ration, review, or approval of the manuscript.
Additional Contributions: Hardeep Singh, MD, MPH (VA HSR&D Center of Ex­
cellence, Michael E. DeBakey Veterans Affairs Medical Center, and Baylor Col­
lege of Medicine, Houston, Texas); and Eric J. Thomas, MD, MPH (University of
Texas at Houston—Memorial Hermann Center for Healthcare Quality and Safety
and Department of Medicine, University of Texas Medical School, Houston), pro­
vided their valuable comments on an earlier version of the manuscript, for which
they received no compensation.
REFERENCES
1. Sittig DF, Ash JS. Clinical Information Systems: Overcoming Adverse
Consequences. Sudbury, MA: Jones & Bartlett; 2009.
2. Bhansali PV. Universal software safety standard: ACM SIGSOFT Software En­
gineering Notes. http://doi.acm.org/10.1145/1095430.1095440. Accessed Janu­
ary 6, 2010.
3. Sittig DF, Ash JS, Zhang J, Osheroff JA, Shabot MM. Lessons from “unex­
pected increased mortality after implementation of a commercially sold comput­
erized physician order entry system.” Pediatrics. 2006;118(2):797-801.
4. Sittig DF, Singh H. Eight rights of safe electronic health record use. JAMA. 2009;
302(10):1111-1113.
5. Walker JM, Carayon P, Leveson N, et al. EHR safety: the way forward to safe
and effective systems. J Am Med Inform Assoc. 2008;15(3):272-277.
6. Koppel R, Kreda D. Health care information technology vendors’ “hold harm­
less” clause: implications for patients and clinicians. JAMA. 2009;301(12):1276­
1278.
7. Leveson N. Safeware: System Safety and Computers. Indianapolis, IN: Addison-
Wesley Professional; 1995.
8. Sanghvi AA. Patient safety and quality improvement act finalized. J Med As­
soc Ga. 2009;98(1):53-54.
9. Kilbridge PM, Welebob EM, Classen DC. Development of the Leapfrog meth­
odology for evaluating hospital implemented inpatient computerized physician or­
der entry systems. Qual Saf Health Care. 2006;15(2):81-84.
10. Pronovost PJ, Goeschel CA, Olsen KL, et al. Reducing health care hazards:
lessons from the Commercial Aviation Safety Team. Health Aff (Millwood). 2009;
28(3):w479-w489.
(Reprinted) JAMA, February 3, 2010—Vol 303, No. 5 451