Professional Documents
Culture Documents
AUDIT AGENCY
AUDIT REPORT
F2009-0005-FD3000
4 May 2009
Executive Summary
i
Executive Summary
ii
Table of Contents
Page
EXECUTIVE SUMMARY i
TAB
A Nuclear Roadmap Measures 1
APPENDIX
I Audit Scope and Prior Audit Coverage 9
BACKGROUND
The nuclear roadmap assessment process was designed to identify and measure progress
made toward reinvigorating the Air Force nuclear enterprise and meet strategic objectives
described in the Roadmap. Subject matter experts (SMEs) evaluate and validate
measures such as those created to assess how well the Air Force performs critical self-
assessments and root cause analysis (RCA). Then, Air Force senior leadership assign a
weight based on importance for each level of command hierarchy. The weighted
measures produce a score for each sub-objective and an overall performance score for
each unit which indicate progress achieved on Roadmap objectives. When implemented,
it should provide the Air Force the ability to communicate its nuclear enterprise status
and trends to DoD, national leaders, and nuclear oversight bodies.
Following validation and leadership approval of the measures, appropriate offices will
collect the required data and construct an initial or baseline assessment. Many planned
measures have never been collected and evaluated across the Air Force nuclear enter-
prise. The first assessment will serve as the baseline comparison for future assessments
to evaluate the progress or regression of any particular objective. Air Force leadership
will direct periodic assessments to maintain visibility on current performance and to track
trends.
Inspection Findings. The inspection findings Roadmap measure did not identify
who was authorized to certify that RCA had been accomplished. Although SMEs
developed a measure requiring units to report the number of MAJCOM inspection
findings for which RCA was accomplished compared to the number of MAJCOM
inspection findings, the measure did not clearly state who was authorized to
certify that RCA had been accomplished. This created the potential for nuclear
units to respond differently to this measure because authority to certify RCA
completion was not assigned to a specific individual involved in the inspection
process. During the audit, SMEs adjusted this measure to specify commanders as
the sole authority to certify RCA completion. This change should provide senior
leaders with consistent measurement across all units and provide better assurance
that units are implementing Roadmap objectives.
Nuclear Infrastructure. Officials did not effectively plan to collect data needed to
compare the dollar value of Program Objective Memorandum (POM) disconnects
1
Tab A
Nuclear Roadmap Measures
against the total Air Force budget. Although Air Force SMEs developed a
measure requiring units to report the “Number of POM disconnects (unfunded
requirements) in support of nuclear infrastructure,” they did not fully consider the
detailed data needed to resolve ambiguities and to give senior leaders a better
assessment that the projects programmed and planned for a given execution year.
During the audit, management changed this measure to “Number of projects
awarded in the execution year/Number of projects programmed in the execution
year.” With this added context within the measure, senior leadership will have
information needed to make necessary budget adjustments.
Cause. This condition occurred because AF/A10 personnel did not establish a process
to systemically analyze measures and associated data requirements against Roadmap
objectives to determine if the measures were adequate. Instead, AF/A10 officials invited
broad participation by SMEs from Air Staff and MAJCOM organizations with nuclear
missions to develop Roadmap measures and relied upon the expertise of personnel
participating in the process. A systemic process to evaluate the measures helps reduce
the risk that developed measures do not adequately address Roadmap objectives.
Impact. Evaluating measures and data used to report Roadmap progress is critical to
providing an assessment of the health of the nuclear enterprise to the Secretary of the
Air Force, Chief of Staff of the Air Force, and MAJCOM commanders who make up the
Nuclear Oversight Board. These assessments will be the basis for senior leadership
decisions on additional changes to the Air Force nuclear program.
Management Comments. AF/A10 concurred with the findings in the report and stated
action taken during the course of the audit should correct the issues identified in the
report.
2
Original Measure Deficiency Identified Corrective Action*
Exhibit 1. Measures Requiring Adjustment. Strategic Objective: The Air Force will implement effective processes for uncovering, analyzing, addressing and reviewing systemic weaknesses.
Number of MAJCOM inspection findings for which RCA Measure did not identify who was Wing commanders were designated
was accomplished/Number of MAJCOM inspection authorized to report completed RCA. as sole authority to certify RCA
findings. completion.
Number of unit-identified discrepancies that include Measure did not specify who was Wing commanders were specifically
RCA/Number of unit-identified discrepancies. authorized to report RCA inclusion. identified as reporting authority.
Number root causes mitigated/Number root causes Measure did not specify who was Same corrective action as above.
identified. authorized to report mitigated root cause.
Number corrective actions closed/Number corrective Measure did not specify who was Same corrective action as above.
actions identified. authorized to close corrective action.
Number nuclear corrective actions and issues viewed in Measure did not evaluate whether units Intent of measure was clarified to
shared database such as Joint Lessons Learned Integration were addressing systemic weaknesses. users; to begin process of uncovering
System (JLLIS)/Number nuclear corrective actions and systemic weaknesses.
issues loaded into shared database such as JLLIS.
Strategic Objective: The Air Force will sufficiently invest in the nuclear deterrence mission arena.
Number of POM disconnects (unfunded requirements) in Measure did not provide a clear assessment Changed measure to compare dollar
support of nuclear infrastructure. of investment in the nuclear mission arena. value of POM disconnects to
Air Force budget.
3
Number of execution year nuclear systems and equipment Measure did not provide context/dollar Changed measure to report dollar
funding adjustments. amount for adjustments. value of adjustments.
Strategic Objective: The Air Force will implement a comprehensive process for ensuring sustained advocacy, focus, and commitment.
Number of strategic communications affirming importance Measure did not identify what could be Clarified measure to only include
of the nuclear mission delivered by Air Force General reported as a strategic communication. communication focused on nuclear
Tab A
Material shipments.
*AF/A10 officials are staffing the revised measures for approval because the coordination process identified areas of additional concern.
^This includes Nuclear Weapon Related Material as defined by the Office of the Secretary of Defense and the Air Force. The Air Force definition is broader and
includes more items.
This Page Intentionally Left Blank
4
Tab B
Internal Control Plan
BACKGROUND
Internal controls provide reasonable assurance that program objectives will be met. An
internal control plan (ICP) should provide systematic guidance on organization and
individual responsibilities, documentation requirements, data validity certification, key
word definitions, and standard data sources.
In an effort to streamline data collection, review, and reporting processes, the Air Force
planned to use a web-based continuous process improvement management tool called
the Nuclear Enterprise Management Tool to gather required data. A Roadmap ICP is
necessary to provide a consistent set of management controls designed to provide an
unbroken chain of accountability for the web-based nuclear enterprise’s assessment
Roadmap process. The goal is to ensure accuracy, completeness, and integrity of all
information capable of measuring progress made to improve the Air Force nuclear
enterprise through Roadmap action plans. Because Air Force nuclear operations are a
high visibility area, data accuracy, completeness, and integrity are imperative to meet
Roadmap objectives.
Data Validity Certification. The Air Force’s planned use of a web-based data
collection tool for data input posed a risk that unauthorized users could access
and manipulate nuclear measures data.1 An ICP would counter this risk by
limiting access to the Nuclear Enterprise Management Tool to only appoint
MAJCOM points of contact (or their alternate).2 Further, an ICP would require
these points of contact to certify that data collection is complete and was reviewed
for accuracy and completeness. This control provides a reasonable level of
confidence in the accuracy of the data collection process that would not have
existed prior to the ICP development.
1 Although the Nuclear Enterprise Management Tool is hosted on a common access card-enabled server,
the ICP provides guidance on who should be allowed a user account for this tool.
2 This control would also limit data entry capability to appointed points of contact.
5
Tab B
Internal Control Plan
Key Word Definitions. Although AF/A10 personnel attempted to use the most
specific wording possible in developing the Roadmap measures, they had not
defined key terminology. An ICP provides Air Force management a tool to
provide in-depth definitions to units to facilitate consistent data gathering efforts.
To illustrate, one Roadmap measure required units to identify the number of
squadron, group, or wing commanders with nuclear experience. The SMEs
formulating the measures intended nuclear experience to be an individual who
had worked a minimum of 2 of the past 8 years in a nuclear unit. Without an ICP,
each nuclear unit could define the experience requirement differently. This would
create major inconsistencies in data reported by units and could misinform
Air Force senior leadership on the experience level of nuclear squadron, group,
and wing commanders. This control facilitates fact-based decisions related to
future training required to strengthen the nuclear enterprise.
Cause. This condition occurred because AF/A10 personnel had not considered following
proven data collection processes, like those followed for Base Realignment and Closure
data calls.3 Further, they did not consider that an ICP would be a useful management tool
during data collection.
Impact. Sufficiently detailing measures, required data, and the measures’ intent faci-
litates a consistent and accurate picture of the health of each Air Force nuclear unit
reporting to the Secretary of the Air Force, Chief of Staff of the Air Force, and MAJCOM
commanders who make up the Nuclear Oversight Board. Without an ICP, senior leader-
ship could direct unnecessary changes to the nuclear enterprise or, conversely, inaccu-
rately conclude the nuclear enterprise or segments of the nuclear enterprise are effective
based on inaccuracies and inconsistencies in the underlying data.
Management Comments. AF/A10 concurred with the findings in the report and stated
action taken during the course of the audit should correct the issues identified in the
report.
3 During the Base Realignment and Closure Process, Air Force managers developed evaluation criteria
and made data calls to obtain information necessary to evaluate which bases should be closed. A critical
element of this process was developing an internal control plan providing guidance to Air Force units on
issues such as key word definitions, standard data sources, and documentation requirements.
6
Tab B
Internal Control Plan
7
This Page Intentionally Left Blank
8
Audit Scope and
Prior Audit Coverage
AUDIT SCOPE
Audit Coverage. We performed this audit at HQ Air Force, AF/A10 (Appendix II). We
performed the review from October 2008 to January 2009 and issued a draft report to
management in March 2009.
9 Appendix I
This Page Intentionally Left Blank
10
Locations Audited/
Reports Issued
Installation-Level
Organization/Location Reports Issued
11 Appendix II
This Page Intentionally Left Blank
12
Points of Contact
13 Appendix III
This Page Intentionally Left Blank
14
Final Report Distribution
SAF/OS ACC
SAF/US AETC
SAF/FM AFISRA
SAF/IG AFMA
SAF/LL AFMC
SAF/PA AFOSI
SAF/XC, AF/A6 AFRC
AF/CC AFSOC
AF/CV AFSPC
AF/CVA AMC
AF/A8 ANG
AF/A10 PACAF
AF/RE USAFA
USAFE
Units/Orgs Audited
AU Library
DoD Comptroller
OMB
The disclosure/denial authority prescribed in AFPD 65-3 will make all decisions relative
to the release of this report to the public.
15 Appendix IV
This Page Intentionally Left Blank
16
To request copies of this report or to suggest audit topics