Available online at www.sciencedirect.

com

Nurs Outlook xxx (2017) 1e10

www.nursingoutlook.org

Twenty years of staffing, practice environment, and

outcomes research in military nursing
Patricia A. Patrician, PhD, RN, FAANa,*, Lori A. Loan, PhD, RN, FAANa,
Mary S. McCarthy, PhD, RN, FAANb, Pauline Swiger, RN, MSN, CNLc,
Sara Breckenridge-Sproat, PhD, RNd, Laura Ruse Brosch, PhD, RNe,
Bonnie Mowinski Jennings, PhD, RN, FAANf
a
Department of Family, Community Health & Systems, University of Alabama at Birmingham, School of Nursing, Birmingham, AL
b
Madigan Army Medical Center, Tacoma, WA
c
University of Alabama at Birmingham, School of Nursing, Birmingham, AL
d
Regional Health Command Europe, Sembach, Germany
e
Office of Research Protections (ORP) Headquarters, US Army Medical Research and Materiel Command, Fort Detrick, MD
f
Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA

article info abstract

Article history: Background: Two decades ago, findings from an Institute of Medicine (IOM) report
Received 29 March 2017 sparked the urgent need for evidence supporting relationships between nurse
Revised 28 June 2017 staffing and patient outcomes.
Accepted 29 June 2017 Purpose: This article provides an overview of nurse staffing, practice environment,
and patient outcomes research, with an emphasis on findings from military
studies. Lessons learned also are enumerated.
Keywords: Method: This study is a review of the entire Military Nursing Outcomes Database
Nursing sensitive indicators (MilNOD) program of research.
Outcomes Discussion: The MilNOD, in combination with evidence from other health care
Practice environment studies, provides nurses and leaders with information about the associations
Staffing between staffing, patient outcomes, and the professional practice environment
of nursing in the military. Leaders, therefore, have useful empirical evidence to
make data-driven decisions. The MilNOD studies are the basis for the current
Army nursing dashboard, and care delivery framework, called the Patent Car-
ingTouch System.
Conclusion: Future research is needed to identify ideal staffing based on workload
demands, and provide leaders with factors to consider when operationalizing
staffing recommendations.
Cite this article: Patrician, P. A., Loan, L. A., McCarthy, M. S., Swiger, P., Breckenridge-Sproat, S., Brosch, L.
R., & Jennings, B. M. (2017, -). Twenty years of staffing, practice environment, and outcomes research in
military nursing. Nursing Outlook, -(-), 1-10. http://dx.doi.org/10.1016/j.outlook.2017.06.015.

* Corresponding author: Patricia A. Patrician, Department of Family, Community Health, & Systems, University of Alabama at Bir-
mingham School of Nursing, NB 324, 1720 Second Avenue South, Birmingham, AL 35294-1210.
E-mail address: ppatrici@uab.edu (P.A. Patrician).
0029-6554/$ - see front matter 2017 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.outlook.2017.06.015
2 Nurs Outlook xxx (2017) 1e10
Introduction
In the two decades since the release of the 1996 Insti-
tute of Medicine (IOM) report that underscored the lack
of empirical evidence about the contributions of nurse
staffing to the quality of patient care (Wunderlich,
Sloan, & Davis, 1996), researchers have attempted to
document empirically the contribution of nursing to
patient care outcomes in a variety of settings. A
compelling body of evidence suggests that staffing is
associated with patient outcomes in clinically mean-
ingful ways (Kane, Shamliyan, Mueller, Duval, & Wilt,
2007; Lang, Hodge, Olson, Romano, & Kravitz, 2004;
Page, 2004). Higher nurse to patient ratios, a higher
proportion of registered nurses (RNs), and higher total
nursing care hours have been linked to lower patient
mortality (Aiken, Clarke, Sloane, Sochalski, & Silber,
2002; Kane et al., 2007; Lang et al., 2004; Needleman,
Buerhaus, Meattke, Stewart, & Zelevinsky, 2002),
decreased length of stay, and a lower likelihood of
patient complications, such as nosocomial infections
and pressure injury (Kovner & Gergen, 1998; Lang et al.,
2004; Needleman et al., 2002).
The research on adverse events such as medication
errors and patient falls as they relate to nurse staffing
is less conclusive. Lang et al. (2004) conducted a sys-
tematic review of the nurse staffingepatient outcomes
research from 1980 to 2003 and did not find evidence
attributing nurse staffing to a number of patient safety
outcomes, including falls, medication errors, and pa-
tient injuries. Other investigators reported insignifi-
cant associations between staffing patterns and falls
(Bolton et al., 2007; Lake & Cheung, 2006), medication
errors (Cho, Ketefian, Barkauskas, & Smith, 2003; Lang
et al., 2004; Mark & Belyea, 2009), or both (Mark,
Hughes, Belyea, Bacon, Chang, & Jones, 2008).
Conversely, other researchers found higher risk of falls
with fewer nursing care hours per day and a lower
proportion of RNs (Dunton, Gajewski, Taunton, &
Moore, 2004) and increased medication errors with
lower staffing levels (Blegen & Vaughan, 1998).
The context of hospital care, or the practice envi-
ronment, also has been found to influence outcomes.
The nursing practice environment is defined as the
conditions under which nurses work and includes
consideration of leadership, resource availability, and
collaboration with physicians (Lake, 2002). There is
ample empirical evidence that favorable professional
nursing practice environments are associated with low
levels of nurse burnout, higher job satisfaction, less
nurse turnover, and more positive patient outcomes to
include lower mortality and higher satisfaction (Aiken
& Sloane, 1997; Aiken, Clarke, Sloane, Lake, & Cheney,
2008; Aiken, Sloane, Lake, Sochalski, & Weber, 1999;
Kazanjian, Green, Wong, & Reid, 2005; Patrician,
Shang, & Lake, 2010b).
In parallel with this body of evidence from the
civilian sector, military nurse researchers began a
journey to explore the associations among nurse
staffing, nursing practice environment, and patient
safety in military hospitals. This article provides an
overview of staffing, nursing practice environment,
and patient outcomes research in the military over the
past two decades, introduces the lessons learned from
this body of research, and identifies future directions
for this line of inquiry related to health care in the
military.
The Case for Military Nursing-Specific
Research
Distinctive Features of Nursing Staff in Military
Hospitals
Although many of the findings about nurse staffing and
the nursing practice environment discovered from
studies conducted in civilian hospitals are applicable to
military nursing, there are important differences be-
tween civilian and military health care. Similar to the
civilian sector, nursing personnel working in military
hospitals may be RNs, licensed practical or vocational
nurses (LPNs and LVNs), or nursing assistants (NAs).
Less than half of the RNs and LPN/LVNs are active duty
military officers or enlisted personnel. Department of
Defense (DoD)/federal civilians account for most
workforce, with additional nurses provided by contract
and military reserve nursing personnel. Compared
with civilian hospital nurse staffing models, military
staffing models typically include a lower proportion of
RNs and a higher proportion of LPNs/LVNs, and more
assistive personnel, such as medics, corpsmen, and
NAs. Findings have suggested that the military rank
structure may facilitate more positive working re-
lationships between nurses and physicians in military
health care settings compared with civilian settings
(Foley, Kee, Minick, Harvey, & Jennings, 2002; Patrician
et al., 2010b).
Compared with their federal civilian counterparts
and nurses working in civilian hospitals, military RNs
are younger, the proportion of men is greater, and
there is more ethnic diversity (Patrician et al., 2010b,
2011a, 2011b). More military RNs have bachelors,
masters, and doctoral degrees as well as advanced and
specialty training (e.g., critical care nursing, operating
room nursing) than civilian nurses (Spratley, Johnson,
Sochalski, Fritz, & Spencer, 2000). Although federal
civilian nurses may choose to move into identified
leadership positions, leadership education and skills
are essential for military nurses because they are ex-
pected to assume leadership roles and advance in rank.
Military courses and competitive selection to attend
civilian universities for advanced degrees are typical
avenues for higher education.
Although baccalaureate education is associated
with better patient outcomes (Aiken, Clarke, Cheung,
 Nurs Outlook xxx (2017) 1e10
Sloane, & Silber, 2003), so is nursing experience
(Blegen, Vaughn, & Goode, 2001). In the military, as in
civilian settings, new graduates with less nursing
experience are more likely to provide care at the
bedside.
Another unique characteristic of military nurses is
that each nurse assumes a dual role: one as a nurse and
another as a military service member. Military training
requirements may remove nursing personnel from
their assigned patient care unit for an hour here and
there or for many months at a time, creating a more
dynamic work environment as staff is augmented or
changed (Breckenridge-Sproat, Johantgen & Patrician,
2011). Military deployments also displace nursing
personnel from their assigned role. Moreover, manda-
tory relocation of military personnel approximately
every 3 years has significant implications on the defi-
nition and measurement of staff turnover.
Distinctive Features of the Military Health System
Although civilian hospitals report data to external
stakeholders such as the Centers for Medicare and
Medicare Services, American Hospital Association, in-
surance providers, and other entities, military hospi-
tals generally have not participated in large public
database collection initiatives. The lack of standard-
ized and accessible military health data is a challenge
for investigators who are trying to study nurse staffing
and patient outcomes. Primary data collection for
nurse staffing and patient outcome studies often is cost
prohibitive. The need for a database containing mili-
tary data related to nurse staffing and patient out-
comes was a catalyst to creating the Military Nursing
Outcomes Database (MilNOD).
Early Studies Reflecting the Military Nursing
Practice Environment
Before research focused on nurse staffing and patient
outcomes, military nurse scientists began to explore
the culture and dynamics of the nursing practice
environment within the military health care system.
These studies are sparse, and the findings are incon-
sistent. For instance, in the latter part of the 1990s,
investigators conducted a study at a single military
medical center; they found poor nurseephysician
communication and low autonomy in military nursing
practice environments (Anderson, Maloney, Oliver,
Brown, & Hardy, 1996; Maloney, Anderson, Gladd,
Brown, & Hardy, 1996). By contrast, Foley et al. (2002),
who studied two different military hospitals, found
nursing practice environments characterized by high
autonomy, control over practice, clinical expertise
among the nursing staff, and collegiality with
physicians.
The first systemwide investigation within army
hospitals examining the effect of nursing practice
3
environments on nurses job satisfaction, emotional
exhaustion, intent to leave the military, and ratings of
care quality was conducted by Patrician et al. (2010b).
These investigators collected data from all 23 army
hospitals. On average, nurses working in these military
settings rated their practice environment as favorable,
reflecting average scores that represented ratings be-
tween those reported for Magnet and nonmagnet
hospitals on the Practice Environment Scale. Patrician
et al. (2010a, 2010b) also found that more poorly rated
military nursing practice environments were associ-
ated with nursing job dissatisfaction, intent to leave,
emotional exhaustion, and poor quality of care ratings.
Beginnings of a Military Program of Research
on Staffing, Practice Environment, and
Outcomes
Linking Nursing Care Quality to Patient Outcomes
Studies performed by military nurse researchers
following the 1996 IOM report (Wunderlich et al., 1996)
demonstrate an ongoing commitment to investigate
associations between nursing care quality and patient
outcomes. These early studies parallel efforts by the
Agency for Healthcare Research and Quality (then the
Agency for Health Care Policy and Research) promoting
patient outcomes research (Mitchell & Durenberger,
1990) and the American Nurses Associations (ANA)
work to create a nursing care report card for use in
acute care settings (ANA, 1995), which ultimately
evolved into the National Database of Nursing Quality
Indicators (Montalvo, 2007).
Military nurse researchers interested in patient
outcomes saw the opportunity to align military
research with the ANA report card efforts, motivating
them to seek funding from the TriService Nursing
Research Program (TSNRP) to assess the feasibility of
collecting various data elements from existing military
data sources related to patient outcomes. The first was
a pilot study designed to determine the feasibility of
collecting the ANA nursing quality indicators at one
army medical center (Jennings, Loan, DePaul, Brosch, &
Hildreth, 2001; Loan, Jennings, Brosch, DePaul, &
Hildreth, 2003). From a sample of 5,082 patient re-
cords on five inpatient units, nursing care quality
scores were calculated by evaluating compliance with
patient education, skin care, patient safety, and central
line management standards of care. Nurse satisfaction,
based on survey responses from 73 nursing staff
members, also was assessed. Along with demon-
strating that ANA nursing quality indicators could be
collected using existing data, analyses found that
staffing mix (RNs, LPNs, and unlicensed staff) and total
nursing care hours per patient day were positively
correlated at a significant level with various measures,
including quality scores (Jennings et al., 2001; Loan
et al., 2003).
4 Nurs Outlook xxx (2017) 1e10
A New Approach to Collecting and Analyzing Nurse
Staffing and Patient Safety Data
The second foundational study funded by TSNRP was
known as the Army Nursing Outcomes Database
(ANOD) demonstration project, undertaken in collabo-
ration with experts from the California Nursing Out-
comes Coalition (CalNOC), now known as the
Cooperative Alliance for Nursing Outcomes (CALNOC)
(Aydin et al., 2004; Brosch, 2005). The ANOD project had
many features that were instrumental to future military
nurse staffing studies. First, ANOD was designed to use
data collection, analysis, and reporting techniques that
would yield a more granular assessment of the rela-
tionship between staffing and patient outcomes. Sec-
ond, ANOD investigators wished to determine if data
collection could be standardized on 24 clinical nursing
units from two large army medical centers. Believing
that assessment of staffing at the shift and unit levels
and the real-time occurrences of safety events might
better serve to detect meaningful associations, the use
of aggregated hospital-level data was replaced with
unit-level data collection for every 8-hr shift. A third
product from the ANOD study was developing pro-
cesses for producing staffing and outcome reports at the
unit, section, and hospital levels and delivering these to
hospital nurse leaders who also were trained about how
to use them for decision making. Collecting and storing
data to allow for future secondary analysis of staffing,
patient and nurse outcomes, and the practice environ-
ment became a reality.
A fourth consideration by ANOD investigators was
to examine the feasibility and usefulness of collecting
nurse staffing and patient and nurse outcome indicator
data using the CalNOC methodologies (Aydin et al.,
2004; Donaldson, Brown, Aydin, & Bolton, 2001). As
part of the ANOD project, a first of its kind, shift-level
relational database was constructed. This distinction
is noteworthy; the aim was to construct and test a
system to collect data more closely synchronized to the
timing of adverse events, such as falls.
Results from the ANOD project demonstrated, and
army nursing leaders concurred, that it was feasible to
implement a standardized, shift-level nurse staffing and
patient outcome measurement methodology. Moreover,
the reports generated with the ANOD data enabled nurse
leaders to compare nursing care quality to internal and
external benchmarks, on a unit-by-unit, section-by-
section, or hospital-by-hospital basis (Brosch, 2005).
The MilNOD Studies
Proof of Concept for Data Quality, Usefulness, and
Generalizability
The MilNOD project began in 2002 (Loan, Brosch,
McCarthy, & Patrician, 2005). The first aim was to
expand the army database and ANOD processes by
implementing the new MilNOD at seven inpatient
hospital sites across the army, navy, and air force. The
second aim was to sustain a high-quality database to
support evidence-based clinical and administrative
decision making as well as to conduct research related
to nurse staffing effectiveness and patient safety. The
MilNOD project included database development, data
specifications, and a series of studies derived from
MilNOD data.
Needleman et al. (2002) noted we need to better
understand the factors influencing both staffing
levels and mix of personnel in hospitals (p. 1045).
The MilNOD afforded a way to derive a better un-
derstanding of nurse staffing in military hospitals. It
is the bedrock of the database that continues to
enable military nurse leaders and military nurse re-
searchers to trace and analyze daily variations in
staffing and examine the effect of staffing on patient
safety and outcomes. Unique to the MilNOD approach
was the decision to use the unit level for data
collection and analysis. Civilian outcomes re-
searchers advocated a movement away from focusing
only on hospital-level data and toward focusing on
unit-level data (Mitchell & Shortell, 1997). Unit-level
data better support operational groups, such as de-
partments and patient care units, that assume ulti-
mate responsibility for outcomes (Whitman, Kim,
Davidson, Wolf, & Wang, 2002). MilNOD in-
vestigators believed the opportunity to advance
measurement precision resided in the ability to cap-
ture valid and reliable nurse staffing and measures of
clinical workload, along with daily patient care out-
comes at the patients bedside.
Good data are essential to guide clinical and
administrative decision making and support rigorous
research. Thus, the initial MilNOD work focused on
developing processes to ensure sustainment of high-
quality data collection. Early efforts focused on
rigorous assessment of the completeness, accuracy,
and quality of data collected for each indicator in the
database. Along with the indicators that were previ-
ously collected in the ANOD, for example, falls and
pressure ulcer prevalence, the investigators expanded
the MilNOD to include two additional indicatorsd
nurse-committed medication errors and nursing staff
needlestick injuriesdand two additional potential
explanatory variablesdpatient turnover and patient
acuity. The investigators developed data definitions
and collection methodologies for these indicators and
variables.
Along with developing the database, the early
MilNOD work also provided direct benefits to nurse
leaders at participating hospitals. For example, chief
nurse executives received quarterly MilNOD
reports reflecting nurse staffing effectiveness and
patient safety. These data and reports satisfied
The Joint Commission staffing effectiveness
requirements.
 Nurs Outlook xxx (2017) 1e10
Relationships Between Staffing, Outcomes, and the
Practice Environment
During the next 4 years, 2003 to 2006, six additional
military hospitals were added to the study for a total
of 13, and analyses were conducted to explore the
associations between nurse staffing as well as patient
and nurse outcomes at the shift level. Nurse staffing,
patient census, patient turnover, and patient acuity
indicators were collected from each unit and entered
into a spreadsheet that was submitted monthly to the
central MilNOD database. Patient falls, nurse-
committed medication errors, and needlestick in-
juries were obtained from hospital and occupational
health reports. Pressure injury prevalence and re-
straint use data were collected by conducting bian-
nual prevalence surveys at each of the participating
hospitals. Nursing personnel completed annual sur-
veys about nurse education, experience, and job
satisfaction. The annual survey also included
administration of the Practice Environment Scale of
the Nursing Work Index (PES-NWI; Lake, 2002).
Annual patient surveys included satisfaction with
pain management, patient education, and care as
measured by The Patient Satisfaction with Nursing
Care Questionnaire (Jacox, Bausell, & Mahrenholz,
1997). A summary of findings from the 13 military
hospitals that participated in the MilNOD during the 4
years follows.
Safety Improvements Over Time
On average, statistically significant improvements
were demonstrated in fall rates, nurse-committed
medication error rates, and hospital-acquired pres-
sure injury (HAPI; National Pressure Ulcer Advisory
Panel, 2016). Data collection became routine, reports
were being used by hospital nurse leadership to
monitor patient safety and staffing effectiveness, and
several best practices were adopted by all hospitals,
such as using the Braden scale (Bergstrom, Braden,
Laguzza, & Holman, 1987) to predict patient pressure
injury risk.
Shift-Level Analyses
The shift-level analyses were conducted using
Bayesian hierarchical logistic modeling techniques to
model the odds of one or more adverse event(s) (i.e.,
separate analyses for falls, falls with injury, nurse-
committed medication errors with and without harm,
and needlestick injuries) occurring on a shift. As ex-
pected, with most adverse events, lower RN skill mix
and/or lower levels of total nurse staffing were asso-
ciated with the adverse events. Patient acuity was
higher when falls and nurse-committed medication
errors occurred. Nurse staff occupational categories
(military, civilian, military reserve, or contract) showed
that civilian staff was a proxy for higher experience
levels. Higher levels of experienced staff on a shift were
associated with fewer falls, medication errors, and
5
needlestick injuries. Additional details of the shift-
level staffing and adverse events are reported else-
where (Patrician et al., 2011a; Patrician, Pryor, Fridman,
& Loan, 2011b).
Practice Environment Findings
There was a strong association between the PES-NWI
and nurse job satisfaction. MilNOD investigators
found differences between the PES-NWI responses of
RNs, LPNs, and NAs, with NAs tending to score much
higher than RNs and LPNs. For this reason, an RN and
LPN-only measure of PES-NWI was recalculated and
aggregated to the unit level and used for all analyses. In
this sample of military hospitals, job satisfaction was
strongly associated with total nursing care hours per
patient shift, even when controlling for the effects of
the PES-NWI scores.
Nursing Personnel Job Satisfaction
Nurse job satisfaction was dependent more on ones
position in the unit rather than on the staffing or pa-
tient turnover rates. Military LPNs and military unli-
censed personnel were the least satisfied with their
jobs, and DoD civilians were most satisfied. These
findings may be explained by the additional duties the
military are required to perform, such as working extra
hours when civilian nurses call in sick. Theoretically,
the military are supposed to be available 24 hr a day,
7 days a week.
The Practice Environment and Workload Intensity:
Testing Mediation and Moderation Effects
The most recent study was a secondary analysis of
MilNOD data using 111,500 shifts from 57 units in 13
hospitals and 1,586 surveys from nurses who worked in
these hospitals. The focus of these efforts was based on
examining workload intensity, which was conceptu-
alized as patient turbulence and patient acuity. Patient
turbulence was measured by the admissions, dis-
charges, and transfers (ADT) that occurred on a shift.
Investigators calculated an ADT index based on the
patient census at the beginning of the shift. As an
example, an ADT of 0.50 meant that 50% of the patients
were transferred or discharged during a shift and
replaced by patients who were admitted during that
shift. A daily measure of patient acuity from the mili-
tarys Workload Management System for Nursing was
also included (Wolgast, Taylor, Garcia, & Watkins,
2011). These data were used to examine how work-
load intensity and the quality of the nursing practice
environment (i.e., scores on the PES-NWI) influenced
the relationship of staffing on patient outcomes using
mediation and moderation analysis. In this study, the
investigators also evaluated the staffing effect on HAPI
using prevalence data from different time points
before the date the HAPI was discovered. Higher LPN
care hours 3 days before the prevalence assessment
were associated with lower HAPI rates. HAPI was
6 Nurs Outlook xxx (2017) 1e10
significantly lower with higher nursing care hours
worked by LPNs (Patrician et al., 2017). These associa-
tions were not detected when the LPN staffing was
aggregated to higher levels (i.e., to the week and
month). Workload intensity, measured by the ADT
index, was associated with nearly all adverse events,
but patient acuity was not. The ADT index partially
mediated the effect of LPN hours on HAPI.
When assessing whether staffing had different ef-
fects on outcomes, based on practice environment
(PES-NWI) scores, there was only one noteworthy
interaction. In medical and surgical units, better
practice environments and higher total nursing care
hours were both associated with fewer falls. There
was a significant interaction of practice environment
scores with total nursing care hours per patient per
shift and patient falls, indicating that staffing has a
stronger effect on falls in better nursing practice
environments.
Moving Beyond MilNOD
The MilNOD research findings provided military nurses
and leaders with information about staffing effective-
ness and the professional practice of nursing in the
military. Leaders were informed and able to make de-
cisions based on high quality and useful data.
In addition, the MilNOD studies were the basis for
the current armywide nursing dashboard, an impor-
tant component of the armys care delivery framework
called the Patent CaringTouch System (PCTS)
(Breckenridge-Sproat et al., 2015). The goals of the PCTS
are to inspire exceptional nursing care by supporting
improved patient outcomes, reducing variations in
practice within the Army Health System, and
decreasing nursing turnover (Horoho, 2011). To address
low scores on the Nursing Participation in Hospital
Affairs scale of the PES-NWI noted in MilNOD research
findings, the PCTS incorporates unit-, hospital-, and
system-level nursing practice councils, an aspect of
shared governance that has empowered all nursing
personnel on the team to participate in professional
and hospital improvements.
Most metrics selected for use in the PCTS were
based on those used in the MilNOD projects (Loan,
Patrician, & McCarthy, 2011; Patrician, Loan,
McCarthy, Brosch, & Davey, 2010a; Patrician et al.,
2011a, 2011b). Nurses in senior leadership positions
and nurse leaders at the bedside use these metrics to
identify early signals of increased or decreased staffing
effectiveness and evaluate unit-, hospital-, or army-
level changes in nursing structure or process by
monitoring associated outcomes. The research find-
ings along with routinely collected longitudinal data
are used as a basis for quantifying success of new
improvement efforts and fostering accountability for
the nursing care quality at all levels (Breckenridge-
Sproat et al., 2015). Findings from deep-dive
evaluations of these data that are related to nurse
staffing are used to formulate and justify staffing
models and inform day-to-day decisions, such as
considering trade-offs between patient outcomes and
nursing requirements when adequate staffing is in
question.
Research-Related Lessons Learned
The research team has learned several important les-
sons over the course of this work. These lessons
pertain to operational definitions, levels of measure-
ment, model specification, and lessons about imple-
mentation of measurement strategies to aid in quality
improvement work.
First, we cannot influence what we do not measure.
We have shown significant improvements in measures
tracked in the 13 MilNOD hospitals during the period of
active data collection. We believe that comparing the
data among units and hospitals and sharing best
practices are what really work to improve care. We
cannot corroborate this belief, however, because it was
beyond the scope of our studies to measure the in-
terventions that took place to change practice. We did
track the use of evidence-based practices to prevent
pressure injury and saw change over time. At the start
of our work, only 3 of the 13 hospitals consistently used
a pressure injury risk assessment tool. By the end of
the first year of the study, 100% of the hospitals were
using the Braden scale.
The second lesson is that definitions matter. Before
the MilNOD series of studies, the military hospitals
used various operational definitions of adverse events
to include falls. If we cannot agree on standardizing
nomenclature, we have no basis for comparisons.
This is a critical consideration if the nursing profes-
sion is to progress in developing nurse-sensitive
indicators.
The third lesson pertains to the unit of analysis.
From the beginning, ANOD and the MilNOD were
developed to collect and analyze data at the shift level.
This choice was predicated on the belief that it is
important to evaluate the effects of staffing and
adverse events at the closest time point to the event, as
with discrete adverse events tied to a shift (e.g., falls,
medication administration errors). We found only one
other study in which the investigators measured
staffing at the shift level. Needleman et al. (2011) found
increased inpatient mortality associated with shifts
where the staffing levels were below what was rec-
ommended by the staffing planning methodology.
They also found increased patient turnover (ADT)
associated with increased patient mortality. Thus,
shift-level data are superior to data aggregated at the
hospital level if the goal is to understand these discrete
phenomena.
Through the studies based on the MilNOD, we have
learned that patient acuity is not a good measure of
 Nurs Outlook xxx (2017) 1e10
nursing workload. This may be, in part, because acuity
is used to plan staffing in the military hospitals. A more
precise measure of acuity, used in the study by
Needleman et al. (2011), may be a ratio of required
nursing care hours to actual care hours. The mea-
surement of workload intensity as it affects the need
for additional staffing has been elusive in the field of
health services research in nursing; military research
shows that ADT is superior to census alone in
measuring staffing effectiveness.
Fourth, modeling some of our rare events, such as
falls and medication errors, is extremely difficult
because of the analytic problems caused by the
numerous shifts where these rare events do not occur.
Rare events complicate statistical analyses and
require different types of approaches, including Pois-
son and negative binomial models (Loan & Patrician,
2005).
Fifth, we need to incorporate implementation sci-
ence methods to effect improvements in care (Tabak,
Khoong, Chambers, & Brownson, 2012). Implementa-
tion science offers ideas about strategies for
improving nursing-sensitive patient outcomes.
Although we shared results quarterly with nursing
leaders at each site, we do not have a good account of
whether (or how) they used the data to make changes.
We used some outreach methods, such as newslet-
ters, to share best practices to assist with knowledge
building; these methods were helpful but difficult to
sustain.
The Future of Military Nursing Staffing,
Practice Environment, and Outcomes
Research
Although many studies, including those discussed in this
article, demonstrate the positive effects of more staffing,
there is still insufficient evidence to determine the ideal
staffing needed to ensure the delivery of high-quality
care (Griffiths et al., 2016). Most staffing and outcomes
research aggregates data to the hospital level; minimal
research is conducted at the unit or shift levels. In addi-
tion, because of perceived and actual methodological
restrictions and the presence of unmeasured or poorly
measured variables, the results of staffing research are
difficult to translate into specific practice recommenda-
tions (Griffiths et al., 2016). One goal of future military
staffing research may be focused on identifying the ideal
number of staff and best skill mix to maintain high-
quality care. Several steps could help achieve this goal.
First, existing workload measurement systems
should be revalidated. Workload measures such as
nursing care hours per patient day and total nursing
care hours per patient day, which are used frequently to
determine staffing needs, could be incorrect or biased
toward underestimating the actual need for nursing
care (Twigg, Gelder, & Myers, 2015). For example, the
7
demands associated with medication administration
are complex, begin before, and last far beyond the
scheduled administration time (Jennings, Sandelowski,
& Mark, 2011). Validation of existing and newly
emerging workload systems, therefore, needs to
continue, as the business of nursing is complex and
demanding (Swiger, Vance, & Patrician, 2016). The
validation of these workload measurement systems can
only occur in light of the research identifying variables
that impact nursing workload. For example, workload
generated from organizational demands and the effects
of patient turnover are often left out of measurement
systems (Jennings, Sandelowski, & Higgins, 2013;
Krichbaum et al., 2007; Myny et al., 2012; Park, Dunton,
& Blegen, 2016) and may confound the results of staff-
ing and outcomes studies when left unaccounted.
Second, the use of contemporary statistical methods
and the study of big data generated from electronic
health records may advance this area of research.
Mining large data sets may provide opportunities to
identify associations between staffing and outcomes
that were previously unknown (Topaz & Pruinelli,
2017). The capture of data as an event happens may
also provide insights that exceed those generated from
the shift-level data seen in the MilNOD studies.
Next, measurement instruments would benefit from
modernization. For example, the PES-NWI was devel-
oped in the early 2000s with samples collected during
the original Magnet hospital studies of the 1980s (Lake,
2002). The PES-NWI is a common covariate used in
staffing research, and although this measure has un-
dergone modifications for use in specific settings, the
overall instrument remains unchanged. The context in
which nursing care is delivered and the complexity of
that care, however, has changed substantially since the
late 1980s (Ebright, 2010; Ebright, Patterson, Chalko, &
Render, 2003; Myny et al., 2011; Suter et al., 2012).
Further development and refinement of the PES-NWI
would ensure we are not missing important factors
that contribute to the delivery of high-quality care, yet
remain unmeasured by the current instrument.
Finally, learning about measures of individual nurse
performance and effectiveness may also be important
in identifying the ideal number and skill mix of nursing
staff required to provide quality care. Similar to the
way performance and compliance is tracked for inde-
pendent licensed providers, such as physicians and
nurse practitioners, individualized measures of per-
formance could lead to a direct measure of nurse
effectiveness. Measures commonly studied to deter-
mine individual nurse effectiveness, such as educa-
tion, experience, and licensure, are often used as
indirect measures of nursing competence (Griffiths
et al., 2016). More concrete measures of individual
competence and performance, such as identification of
deteriorating patient status and taking appropriate
follow-on actions, could be tracked electronically. The
goal of collecting individual performance measures
would not be to place blame but instead to identify top
8 Nurs Outlook xxx (2017) 1e10
performers and support underperformers with addi-
tional resources. This philosophy is in line with high
reliability organizations that gather and display quality
measures, reward staff for reaching and sustaining
quality goals, and work to identify unsafe conditions
before patients are harmed (Griffith, 2015).
Conclusions
Although the understanding of nurse staffing as it re-
lates to quality care has advanced over the past 20
years, the complexity of the care environment and the
patients themselves leave much to be learned about
ideal nurse staffing. The program of research repre-
sented by the MilNOD, its predecessors, and the armys
Patient CaringTouch System illustrate how military
nurses and funding by the TSNRP have ensured mili-
tary beneficiaries benefit from military unique knowl-
edge and evidence-based practice. Not only has
MilNOD ensured that the military kept pace with in-
vestigations of nurse staffing and quality care in the
civilian sector, but studies based on MilNOD data are
able to show the benefit of shift-level datada largely
unprecedented accomplishment.
Acknowledgments
This project was funded by the TSNRP, Uniformed
Services University of the Health Sciences (grants
N97-011, N01-016, N02-P06, N03-P07, N10-C01, and
N13-P13); however, the information or content and
conclusions neither do necessarily represent the offi-
cial position or policy of nor should any official
endorsement be inferred by the TSNRP, Uniformed
Services University of the Health Sciences, the
Department of Defense, or the U.S. government.
references
Aiken, L. H., Clarke, S. P., Cheung, R. B., Sloane, D. M., & Silber, J.
H. (2003). Educational levels of hospital nurses and surgical
patient mortality. JAMA, 290, 1617e1623.
Aiken, L. H., Clarke, S. P., Sloane, D. M., Lake, E. T., & Cheney, T.
(2008). Effects of hospital care environment on patient
mortality and nurse outcomes. Journal of Nursing
Administration, 38, 223e229.
Aiken, L. H., Clarke, S. P., Sloane, D. M., Sochalski, J., & Silber, J. H.
(2002). Hospital nurse staffing and patient mortality, nurse
burnout, and job dissatisfaction. JAMA, 288, 1987e1993.
Aiken, L. H., & Sloane, D. M. (1997). Effects of organizational
innovations in AIDS care on burnout among urban hospital
nurses. Work and Occupations, 24, 453e477.
Aiken, L. H., Sloane, D. M., Lake, E. T., Sochalski, J., & Weber, A. L.
(1999). Organization and outcomes of inpatient AIDS care.
Medical Care, 37, 760e772.
American Nurses Association (ANA). (1995). Nursing quality
indicators: Recommended definitions. Washington, DC: American
Nurses Publishing.
Anderson, F. D., Maloney, J. P., Oliver, D. L., Brown, D. L., &
Hardy, M. A. (1996). Nurse-physician communication:
Perceptions of nurses at an Army medical center. Military
Medicine, 161, 411e415.
Aydin, C. E., Bolton, L. B., Donaldson, N., Brown, D. S., Buffum, M.,
Elashoff, J. D., & Sandhu, M. (2004). Creating and analyzing a
statewide nursing quality measurement database. Journal of
Nursing Scholarship, 36, 371e378.
Bergstrom, N., Braden, B. J., Laguzza, A., & Holman, V. (1987). The
Braden scale for predicting pressure sore risk. Nursing
Research, 36(4), 205e210.
Blegen, M. A., & Vaughn, T. (1998). A multisite study of nurse
staffing and patient occurrences. Nursing Economics, 16,
196e203.
Blegen, M. A., Vaughn, T., & Goode, C. J. (2001). Nurse experience
and education: Effect on quality of care. Journal of Nursing
Administration, 31, 33e39.
Bolton, L. B., Aydin, C. E., Donaldson, N., Brown, D. S., Sandhu, M.,
Fridman, M., & Aronow, H. U. (2007). Mandated nurse staffing
ratios in California: A comparison of staffing and nursing-
sensitive outcomes pre- and post-regulation. Policy, Politics, &
Nursing Practice, 8, 238e250.
Breckenridge-Sproat, S., Freeman, J. J., Belew, D. L., Loan, L. A.,
McCarthy, M. S., & Patrician, P. A. (2015). A nursing care redesign
framework for system wide improvement: The Patient
CaringTouch System. Clinical Nursing Studies, 3(4), 72e78.
Breckenridge-Sproat, S., Johantgen, M., & Patrician, P. A. (2011).
Influence of unit-level staffing on medication errors and falls
in military hospitals. Western Journal of Nursing Research, 34,
455e474.
Brosch, L. R. (2005). Establishing a military nursing outcomes
database. Final report N02eP06). Bethesda, MD: TriService
Nursing Research Program.
Cho, S. H., Ketefian, S., Barkauskas, V. H., & Smith, D. G. (2003).
The effects of nurse staffing on adverse events, morbidity,
mortality, and medical costs. Nursing Research, 52, 71e79.
Donaldson, N. E., Brown, D. S., Aydin, C. E., & Bolton, L. B. (2001).
Nurse staffing in California hospitals 1998-2000: Findings from
the California nursing outcomes coalition database project. Policy,
Politics & Nursing Practice, 2(1), 20e29.
Dunton, N., Gajewski, B., Taunton, R. L., & Moore, J. (2004). Nurse
staffing and patient falls on acute care hospital units. Nursing
Outlook, 52, 53e59.
Ebright, P. R. (2010). The complex work of RNs: Implications for
healthy work environments. The Online Journal of Issues in
Nursing, 15(1), Manuscript 4.
Ebright, P. R., Patterson, E. S., Chalko, B. A., & Render, M. L. (2003).
Understanding the complexity of registered nurse work in
acute care settings. Journal of Nursing Administration, 33,
630e638.
Foley, B. J., Kee, C. C., Minick, P., Harvey, S. S., & Jennings, B. M.
(2002). Characteristics of nurses and hospital work
environments that foster satisfaction and clinical expertise.
Journal of Nursing Administration, 32, 273e282.
Griffith, J. R. (2015). Understanding high-reliability organizations:
Are Baldrige recipients models? Journal of Healthcare
Management, 60, 44e61.
Griffiths, P., Ball, J., Drennan, J., DallOra, C., Jones, J., Maruotti, A.,
., Simon, M. (2016). Nurse staffing and patient outcomes:
Strengths and limitations of the evidence to inform policy and
practice. A review and discussion paper based on evidence
 Nurs Outlook xxx (2017) 1e10
reviewed for the National Institute for Health and Care
Excellence Safe Staffing guideline development. International
Journal of Nursing Studies, 6, 213e225.
Horoho, P. D. (2011). Army nursing: Transforming for a new
century of caring. US Army Medical Department Journal, 4e9.
Jacox, A., Bausell, B., & Mahrenholz, D. (1997). Patient satisfaction
with nursing care in hospitals. Outcomes Management for
Nursing Practice, 1(1), 20e28.
Jennings, B. M., Loan, L. A., DePaul, D., Brosch, L. R., & Hildreth, P.
(2001). Lessons learned while collecting ANA indicator data.
Journal of Nursing Administration, 31, 121e129.
Jennings, B. M., Sandelowski, M., & Higgins, M. K. (2013). Turning
over patient turnover: An ethnographic study of admissions,
discharges, and transfers. Research in Nursing & Health, 36,
554e566.
Jennings, B. M., Sandelowski, M., & Mark, B. (2011). The nurses
medication day. Qualitative Health Research, 21, 1441e1451.
Kane, R. L., Shamliyan, T. A., Mueller, C., Duval, S., & Wilt, T. J.
(2007). The association of registered nurse staffing levels and
patient outcomes: Systematic review and meta-analysis.
Medical Care, 45, 1195e1204.
Kazanjian, A., Green, C., Wong, T., & Reid, R. (2005). Effect of the
hospital nursing environment on patient mortality: A
systematic review. Journal of Health Service Research & Policy, 10,
111e117.
Kovner, C., & Gergen, P. J. (1998). Nurse staffing levels and adverse
events following surgery in U.S. hospitals. Image: Journal of
Nursing Scholarship, 30, 315e321.
Krichbaum, K., Diemert, C., Jacox, L., Jones, A., Koenig, P.,
Mueller, C., & Disch, J. (2007). Complexity compression: Nurses
under fire. Nursing Forum, 42(2), 86e94.
Lake, E. T. (2002). Development of the practice environment scale
of the Nursing Work Index. Research in Nursing & Health, 25,
176e188.
Lake, E. T., & Cheung, R. B. (2006). Are patient falls and pressure
ulcers sensitive to nurse staffing? Western Journal of Nursing
Research, 28(6), 1e24.
Lang, T. A., Hodge, M., Olson, V., Romano, P. S., & Kravitz, R. L.
(2004). Nurse-patient ratios: A systematic review on the effects
of nurse staffing on patient, nurse employee, and hospital
outcomes. Journal of Nursing Administration, 34, 326e337.
Loan, L. A., Brosch, L. R., McCarthy, M. S., & Patrician, P. A. (2005).
Designing and implementing a national database depicting
quality of nursing care and staffing effectiveness. Army Medical
Department Journal, 50e58.
Loan, L. A., Jennings, B. M., Brosch, L. R., DePaul, D., & Hildreth, P.
(2003). Indicators of nursing care quality: Findings from a pilot
study. Outcomes Management for Nursing Practice, 7(2), 51e58.
Loan L., & Patrician, P. A. (2005). Nursing quality data: Characteristics
of the data and analytical challengesdReport from the 2004
National Nursing Quality Databases Analytical Conference. Paper
presentation at the National Nursing Quality Databases 2005
National Conference, San Francisco, CA
Loan, L. A., Patrician, P. A., & McCarthy, M. (2011). Participation in
a national nursing outcomes database: Monitoring outcomes
over time. Nursing Administration Quarterly, 35(1), 72e81.
Maloney, J. P., Anderson, F. D., Gladd, D. L., Brown, D. L., &
Hardy, M. A. (1996). Evaluation and comparison of health care
work environment scale in military settings. Military Medicine,
161, 284e289.
Mark, B. A., & Belyea, M. (2009). Nurse staffing and medication
errors: Cross-sectional or longitudinal relationships? Research
in Nursing & Health, 32, 18e30.
Mark, B. A., Hughes, L. C., Belyea, M., Bacon, C. T., Chang, Y. K., &
Jones, C. A. (2008). Exploring organizational context and
structure as predictors of medication errors and patient falls.
Journal of Patient Safety, 4, 66e77.
9
Mitchell, G. J., & Durenberger, D. (1990). Promoting value in health
care: The new Agency for Health Care Policy and Research.
Academic Medicine, 65, 204e205.
Mitchell, P. H., & Shortell, S. M. (1997). Adverse outcomes and
variations in organization of care delivery. Medical Care, 35(11
Suppl), S19eS32.
Montalvo, I. (2007). The National Database of Nursing Quality
Indicators. Online Journal of Issues on Nursing, 3(2). Retrieved
from http://www.nursingworld.org/MainMenuCategories/
ANAMarketplace/ANAPeriodicals/OJIN/TableofContents/
Volume122007/No3Sept07/NursingQualityIndicators.html
Myny, D., Van Goubergen, D., Gobert, M., Vanderwee, K., Van
Hecke, A., & Defloor, T. (2011). Non-direct patient care factors
influencing nursing workload: A review of the literature.
Journal of Advanced Nursing, 67, 2109e2129.
Myny, D., Van Hecke, A., De Bacquer, D., Verhaeghe, S., Gobert, M.,
Defloor, T., & Van Goubergen, D. (2012). Determining a set of
measurable and relevant factors affecting nursing workload in
the acute care hospital setting: A cross-sectional study.
International Journal of Nursing Studies, 49, 427e436.
National Pressure Ulcer Advisory Panel (2016). National Pressure
Advisory Panel announces a change in terminology from pressure ulcer to
pressure injury and updates the stages of pressure injury. Press release.
Retrieved from http://www.npuap.org/national-pressure-ulcer-
advisory-panel-npuap-announces-a-change-in-terminology-
from-pressure-ulcer-to-pressure-injury-and-updates-the-sta
ges-of-pressure-injury/
Needleman, J., Buerhaus, P., Meattke, S., Stewart, M., &
Zelevinsky, K. (2002). Nurse-staffing levels and the quality of
care in hospitals. The New England Journal of Medicine, 346,
1715e1722.
Needleman, J., Buerhaus, P., Pankratz, V. S., Leibson, C. L.,
Stevens, S. R., & Harris, M. (2011). Nurse staffing and inpatient
hospital mortality. The New England Journal of Medicine, 364,
1037e1045.
Page, A. (2004). In Keeping patients safe: Transforming the work
environment of nurses. Washington, DC: National Academy
Press.
Park, S. H., Dunton, N., & Blegen, M. A. (2016). Comparison of unit-
level patient turnover measures in acute care hospital
settings. Research in Nursing & Health, 39, 197e203.
Patrician, P. A., Loan, L., McCarthy, M., Brosch, L. R., & Davey, K. S.
(2010a). Towards evidence-based management: Creating an
informative database of nursing-sensitive indicators. Journal of
Nursing Scholarship, 42, 358e366.
Patrician, P. A., Loan, L., McCarthy, M., Fridman, M.,
Donaldson, N., Bingham, M., & Brosch, L. R. (2011a). The
association of shift-level nurse staffing with adverse patient
events. The Journal of Nursing Administration, 41, 64e70.
Patrician, P. A., McCarthy, M. S., Swiger, P., Raju, D., Breckenridge-
Sproat, S., Su, X., Randall, K. H., & Loan, L. A. (2017).
Association of hospital-acquired pressure injury with
temporal variations in staffing in military hospitals. Research in
Nursing & Health, 40(2), 111e119.
Patrician, P. A., Pryor, E., Fridman, M., & Loan, L. (2011b).
Needlestick injuries among nursing staff: Association with
shift level staffing. American Journal of Infection Control, 39,
477e482.
Patrician, P. A., Shang, J., & Lake, E. T. (2010b). Organizational
determinants of quality care and nurse job outcomes among
Army Medical Department registered nurses. Research in
Nursing & Health, 33, 99e110.
Spratley, E., Johnson, A., Sochalski, J., Fritz, M., & Spencer, W.
(2000). The registered nurse population: Findings from the National
Sample Survey of Registered Nurses. Washington, DC: US
Department of Health and Human Services, Health Resources
and Service Administration.
10 Nurs Outlook xxx (2017) 1e10
Suter, E., Deutschlander, S., Mickelson, G., Nurani, Z., Lait, J.,
Harrison, L., ., Grymonpre, R. (2012). Can interprofessional
collaboration provide health human resources solutions? A
knowledge synthesis. Journal of Interprofessional Care, 26, 261e268.
Swiger, P., Vance, D. E., & Patrician, P. A. (2016). Nursing workload
in the acute care setting: A concept analysis of nursing
workload. Nursing Outlook, 64, 244e254.
Tabak, R. G., Khoong, E. C., Chambers, D., & Brownson, R. C.
(2012). Bridging research & practice: Models for dissemination
and implementation research. American Journal of Preventive
Medicine, 43, 337e350.
Topaz, M., & Pruinelli, L. (2017). Big data and nursing: Implications
for the future. Studies in Health Technology and Informatics, 232,
165e171.
Twigg, D. E., Gelder, L., & Myers, H. (2015). The impact of
understaffed shifts on nurse-sensitive outcomes. Journal of
Advanced Nursing, 71, 1564e1572.
Whitman, G. R., Kim, Y., Davidson, L. J., Wolf, G. A., &
Wang, S. L. (2002). The impact of staffing on patient
outcomes across specialty units. Journal of Nursing
Administration, 32, 633e639.
Wolgast, K. A., Taylor, K., Garcia, D., & Watkins, M. (2011).
Measuring nursing essential contributions to quality patient
care outcomes. The United States Army Medical Department
Journal, 74e81.
Wunderlich, G. S., Sloan, F., & Davis, C. K. (1996). Nursing staff in
hospitals and nursing homes: Is it adequate? Washington, DC:
National Academy Press.