This title page should include the following information, although the arrangement of the information on the title page is left to the discretion of the organiation preparing it! 1" Date of the Document 2" #dentification of $rogram%$ro&ect%'(ercise%)tudy *" Document Title a" M+) ,ame and -ersion ." Document Type /i"e", 0ccreditation $lan, -+- $lan, -+- 1eport, 0ccreditation 1eport2 and -ersion 3" #dentification of document preparer 4" Distri.ution )tatement /if re5uired2 6" 6lassification /if re5uired2
Final Draft - 18 May 2006 Document Version Version no. Date Changes Draft 1 1 Sep 01 First draft Final Draft - 18 May 2006 Table of Contents V&VPLAN EXECUTIVE SUMMA!""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "# P$%LEM STATEMENT""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "#" Inten&e& Use"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "#' M&S $(er(ie)"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "#* M&S A++lication"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "#, Accre&itation Sco+e"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 "#- V&V Sco+e"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""4 '# M&S E.UIEMENTS AND ACCEPTA%ILIT! CITEIA""""""""""""""""""""""""""""""""""""""""4 *# M&S ASSUMPTI$NS/ CAPA%ILITIES/ LIMITATI$NS/ & IS0S1IMPACTS"""""""""""6 *#" M&S Assum+tions"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""6 *#' M&S Ca+abilities"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""6 *#* M&S Limitations"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""6 *#, M&S is2s1Im+acts"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""6 ,# V&V MET3$D$L$4!""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""6 ,#" Planne& Data V&V Tas2s1Acti(ities""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""7 ,#"#" Data Verification Tas2s1Acti(ities""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""7 ,#"#' Data Vali&ation Tas2s1Acti(ities"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""7 ,#"#* e5uire& Vali&ation Data""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""7 ,#, Planne& Conce+tual Mo&el Vali&ation Tas2s1Acti(ities"""""""""""""""""""""""""""""""""""""7 ,#' Planne& Design Verification Tas2s1Acti(ities""""""""""""""""""""""""""""""""""""""""""""""""""""""7 ,#* Planne& Im+lementation Verification Tas2s1Acti(ities""""""""""""""""""""""""""""""""""""""7 ,#- Planne& esults Vali&ation Tas2s1Acti(ities""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 ,#6 Planne& V&V e+orting Tas2s1Acti(ities"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 -# V&V ISSUES""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 6# 0E! PATICIPANTS"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 6#" Accre&itation Partici+ants""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 6#' V&V Partici+ants""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 6#* $t7er Partici+ants"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 8# PLANNED V&V ES$UCES"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""8 8#" Planne& V&V Tas2ing an& 9un&ing""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""10 8#' Planne& V&V Timeline"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""10 APPENDIX A M&S DESCIPTI$N"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#" M&S $(er(ie)"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#' M&S De(elo+ment an& Structure""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#* M&S Ca+abilities an& Limitations""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#, M&S Use 3istor:"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#- Data"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#-#" In+ut Data""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#-#' $ut+ut Data"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 A#6 Configuration Management""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""11 APPENDIX % M&S E.UIEMENTS TACEA%ILIT! MATIX"""""""""""""""""""""""""""""""12 3 Final Draft - 18 May 2006 APPENDIX C %ASIS $9 C$MPAIS$N"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX D E9EENCES""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX E AC$N!MS""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX 9 4L$SSA!"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX 4 V&V P$4AMMATICS""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX 3 DISTI%UTI$N LIST""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 APPENDIX I ACCEDITATI$N PLAN""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""12 4 Final Draft - 18 May 2006 V&V PLAN EXECUTIVE SUMMA! The e(ecuti9e summary pro9ides an o9er9iew of the -+- $lan" #t should .e a synopsis, two to four pages in length, of the ma&or elements from all sections of the document, with emphasis on -+- scope, M+) re5uirements and accepta.ility criteria, -+- methodology, and -+- issues" "# P$%LEM STATEMENT This section descri.es the pro.lem the M+) is e(pected to address" The pro.lem statement ser9es as the foundation for the definition of re5uirements, accepta.ility criteria, and ultimately the accreditation assessment .y documenting! the 5uestion/s2 to .e answered and%or the particular aspects of the pro.lem that the simulation will .e used to help address the decisions that will .e made .ased on the simulation results the conse5uences resulting from erroneous simulation outputs The information included in the su.sections .elow is common to all four core documents" "#" Inten&e& Use This su.section descri.es the pro.lem to .e addressed .y the M+), including the system or process .eing represented and the role it plays in the o9erall program" "#' M&S $(er(ie) This su.section pro9ides an o9er9iew of the M+) for which this plan is written and discusses the le9el of configuration control that currently e(ists for the M+)" Detailed M+) information is pro9ided in 0ppendi( 0" "#* M&S A++lication This su.section descri.es how the M+) will .e used in the o9erall program and lists the program o.&ecti9es the M+) must meet in order to fulfill the intended use" "#, Accre&itation Sco+e This su.section descri.es the scope of the accreditation effort .ased on the assessment of the ris: of using the M+) and the a9aila.ility of resources" "#- V&V Sco+e This su.section descri.es the scope of the -+- effort .ased on the assessment of M+) re5uirements, accepta.ility criteria, and the a9aila.ility of resources" '# M&S E.UIEMENTS AND ACCEPTA%ILIT! CITEIA This section descri.es the M+) re5uirements defined for the intended use, the deri9ed accepta.ility criteria that must .e met to satisfy the re5uirements, the 5uantitati9e and 5ualitati9e metrics used to measure their success, and the order of their priority The relationship .etween the re5uirements and the accepta.ility criteria can .e shown either in te(t or in a ta.le as shown .elow" 5 Final Draft - 18 May 2006 # M&S Requirement Acceptability Criteria 1 1.1 1.2 1.n 2 2.1 2.n n n.1 *# M&S ASSUMPTI$NS/ CAPA%ILITIES/ LIMITATI$NS/ & IS0S1IMPACTS This section descri.es factors that constrain the de9elopment and%or use of the M+) or that impede the --+0 effort, including the assumptions, capa.ilities, limitations, and ris: factors affecting M+) de9elopment and ris:s associated with using the M+) for the intended use" *#" M&S Assum+tions This su.section descri.es the :nown assumptions a.out the M+), the M+) capa.ilities, the data used in support of the M+), and any constraints placed upon the M+) .y the conte(t of the pro.lem" *#' M&S Ca+abilities This su.section descri.es the :nown capa.ilities of the M+)" *#* M&S Limitations This su.section descri.es the :nown constraints and limitations under which the M+) will .e de9eloped, tested, and used, including constraints on M+) de9elopment that result in limitations in M+) capa.ility, as well as constraints on M+) testing and e9aluation that result in inade5uate information regarding M+) capa.ility /e"g", inade5uate resource, inade5uate su.&ect matter and technical :nowledge e(pertise, una9aila.le data, inade5uately defined M+) re5uirements and methodologies, inade5uate test en9ironments2" *#, M&S is2s1Im+acts This su.section descri.es the ris:s associated with de9eloping and%or using the M+) for the intended use including the ris:s resulting from identified constraints and limitations and the ris:s associated with doing and%or not doing 9arious --+0 tas:s as well as the impacts associated with those ris:s" ,# V&V MET3$D$L$4! The core of the -+- $lan lies in a step-.y-step road-mapping of how the -+- tas:s will .e performed" -+- tas:s should .e tailored according to need, 9alued added, and resources" #n this section, descri.e what -+- tas:s are planned, as well as each tas:;s o.&ecti9es, assumptions, constraints, criteria, methodology, and how they will .e measured and e9aluated" #dentify what performance metrics will .e used" 6 Final Draft - 18 May 2006 ,#" Planne& Data V&V Tas2s1Acti(ities ,#"#" Data Verification Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9erifying the data within the conte(t of how it is used in the M+)" ,#"#' Data Vali&ation Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9alidating the data within the conte(t of how it is used in the M+)" ,#"#* e5uire& Vali&ation Data This su.section descri.es%identifies the data that are needed to implement the tas:s" #t also descri.es the coordination mechanism and schedule for o.taining the needed data" ,#' Planne& Conce+tual Mo&el Vali&ation Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9alidating the conceptual model" #t should correlate specific segments of the conceptual model to the M+) re5uirements and accepta.ility criteria as well as identify which authoritati9e resources will .e used to esta.lish the 9alidity, including su.&ect matter e(perts, reference documents, and reference data" For each su.&ect matter e(pert, the following information should .e pro9ided! 1" name and contact information /e"g", address, phone num.er, email2 2" agency *" summary of rele9ant e(perience 3" education credentials 4" rele9ant pu.lications ,#* Planne& Design Verification Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9erifying the M+) design" #t should correlate specific segments of the design to the conceptual model and to the accepta.ility criteria as well as cite applica.le standards, codes, .est practices, etc" to which the design should adhere and how adherence will .e e9aluated" ,#, Planne& Im+lementation Verification Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9erifying the M+) implementation" #t should descri.e how the M+) de9elopment documentation /installation guide, user;s manual and so on2 will .e re9iewed and e9aluated as well as state how completeness, correctness, and consistency of functional re5uirements will .e measured" This su.section should include a discussion of the planned scenarios, test cases, and sample sie re5uired for the simulation tests designed to 9erify that the software code is error free and to 9erify that there is successful integration of all components into a single simulation system, system of systems, or federation" This su.section should discuss what organiation will run the tests, what organiation will analye the results, the time re5uired to do so, and the schedule for accomplishing the runs" 0n e(ample of the type of information to document follows! 7 Final Draft - 18 May 2006 #mplementation -erification Test Description! 1" #dentify the test .y name, date and time" 2" #dentify test conductor;s name, organiation, phone and email address" *" Descri.e the hardware%software architecture" 3" )tate purpose relati9e to the accepta.ility criteria" 4" $ro9ide .rief description" 6" #dentify any prere5uisite conditions that must .e esta.lished prior to performing the test case" 7" Descri.e test inputs necessary for the test case" 8" #dentify all e(pected results /pre-test predictions2 for the test case 8" Define the test procedure for the test case" 10" #dentify any assumptions made or constraints imposed in the description of the test case" 11" #dentify the 9erification techni5ue to .e used" ,#- Planne& esults Vali&ation Tas2s1Acti(ities This su.section descri.es the o9erall approach for 9alidating the M+) results" #t should correlate M+) results with accepta.ility criteria and M+) re5uirements as well as #dentify all authoritati9e resources to .e used in e9aluating the M+) results, including su.&ect matter e(perts< mathematical or statistical techni5ues< and data resources" #t should state how the resources are to .e applied and how the results are to .e e9aluated" For su.&ect matter e(perts, it should descri.e the specialied s:ills or :nowledge that is needed" For the simulation tests designed to 9alidate the results, this su.section should include a discussion of the planned scenarios, test cases, sample sie, and data re5uired" 0s well as a discussion of what organiation will run the tests, which organiation will analye the results, the time re5uired to do so, and the schedule for accomplishing the runs" 0n e(ample of the type of information to document follows! 1esults -alidation Test Description! 1" #dentify the test .y name, date and time" 2" #dentify test conductor;s name, organiation, phone and email address" *" Descri.e the hardware%software architecture" 3" )tate purpose relati9e to the accepta.ility criteria" 4" $ro9ide .rief description" 6" #dentify any prere5uisite conditions that must .e esta.lished prior to performing the test case" 7" Descri.e test inputs necessary for the test case" 8" #dentify all e(pected results /pre-test predictions2 for the test case 8" Define the test procedure for the test case" 10" #dentify any assumptions made or constraints imposed in the description of the test case" 11" #dentify the 9alidation techni5ue to .e used" 8 Final Draft - 18 May 2006 ,#6 Planne& V&V e+orting Tas2s1Acti(ities This su.section descri.e the plans for producing and deli9ering the -+- 1eport and 0ccreditation $ac:age" -# V&V ISSUES This section discusses the important unresol9ed issues rele9ant to this stage of the --+0 effort, including administration, coordination, and e(ecution" 1eport acti9ities underway to address these issues and the pro.a.ility of each acti9ity;s success" 0s the -+- effort is .oth iterati9e and dependent on the products of the M+) de9elopment process, the -+- process will most li:ely unco9er se9eral unresol9ed issues throughout the --+0 effort" 0lthough these open-ended areas of concern are common, it is important to document all issues early on and formulate what acti9ities are .eing e(ecuted, or will .e conducted, to address each issue, along with the pro.a.ility of their success" 6# 0E! PATICIPANTS This section identifies the participants in9ol9ed in the --+0 effort as well as the roles that they are assigned and their :ey responsi.ilities within that role" 1oles and :ey responsi.ilities are defined during initial planning< names and contact information of the actual participants are added when they are determined" For each person ser9ing as a )u.&ect Matter '(pert /)M'2, include a listing of the person=s 5ualifications" 6#" Accre&itation Partici+ants This su.section lists the participants in9ol9ed in the accreditation effort, including their contact information, assigned role, and the :ey responsi.ilities associated with that role" Typical accreditation roles include 0ccreditation 0uthority, 0ccreditation 0gent, 0ccreditation Team, and )u.&ect Matter '(perts /)M's2" 6#' V&V Partici+ants This su.section lists the participants in9ol9ed in the -+- effort, including their contact information, assigned role, and the :ey responsi.ilities associated with that role" Typical -+- roles include M+) $roponent, -+- 0gent, -+- Team, -alidation 0uthority, Data )ource, and )u.&ect Matter '(perts" 6#* $t7er Partici+ants This su.section identifies the mem.ers of the application program and model de9elopment effort with -+- or accreditation responsi.ilities as well as others who ha9e a role in the --+0 process" The information should include their position or role, contact information, and --+0 responsi.ilities" Typical roles include M+) $rogram Manager, 0pplication )ponsor%>ser, M+) De9eloper, Data )ource, Milestone Decision 0uthority, $rogram ?ffice, M+) De9elopment Team, >ser @roup, 6onfiguration 6ontrol Aoard, and )u.&ect Matter '(perts" 8# PLANNED V&V ES$UCES This section discusses the resources planned for the -+- effort, such as man-hours, materials, and funding" 0s with any programmatic effort, the documentation of schedule, manpower, and funding plans is not only wise, .ut re5uired" Fiscal responsi.ility for -+- dictates careful planning" Thus, the -+- $lan should include a definition of tas:s and their associated resource 9 Final Draft - 18 May 2006 re5uirements so that $rogram Managers can carefully .udget for -+- in their resource allocations" For clarity, it is .eneficial to map out the schedule, manpower, and funding re5uirements in ta.les and to identify any resource shortfalls" 8#" Planne& V&V Tas2ing an& 9un&ing This su.section identifies the resources needed to accomplish the 9erification and 9alidation planned" The information pro9ided here should include the name of the acti9ity, tas:, or e9ent, the list of re5uired resources /e"g", )M's, e5uipment, TDB funding2 needed to accomplish it, milestones, deadlines, and the $?6" The acti9ities, tas:s, and e9ents, and their associated milestones, products, and deadlines should .e consistent with information pro9ided elsewhere in this plan" 8#' Planne& V&V Timeline This su.section identifies when each :ey e9ent descri.ed in the -+- Tas:ing and Funding su.section should .e met" 10 Final Draft - 18 May 2006 APPENDIX A M&S DESCIPTI$N This appendi( contains pertinent detailed information a.out the M+) .eing assessed" A#" M&S $(er(ie) This section pro9ides a description of the M+) including the type of model /e"g", stochastic, deterministic, high resolution, low resolution, human in the loop CD#TEF, hardware in the loop CDG#EF, stand-alone, engineering, aggregated2, and what types of pro.lems it is intended to support /e"g", training, force structure analysis, command and control, e(perimentation, system analysis, analysis of alternati9es2" A#' M&S De(elo+ment an& Structure This section pro9ides information a.out how the M+) is organied and%or constructed /e"g", the M+) design2, hardware and software specifics, and technical statistics /e"g", runtime speed, capacity, .andwidth2" For M+) under de9elopment, this section includes the M+) de9elopment plan, including the de9elopment paradigm .eing followed /e"g", spiral de9elopment, model-test- model2, and .asic assumptions a.out its e(ecution" A#* M&S Ca+abilities an& Limitations This section summaries the capa.ilities and the limitations of the M+) with respect to the intended application" A#, M&S Use 3istor: This section .riefly descri.es how and when the model has .een used in the past as well as references rele9ant historical use documents" A#- Data A#-#" In+ut Data This su.section identifies the data re5uired to populate and e(ecute the simulation, including input data sets, hard-wired data /constants2, en9ironmental data, and operational data" Descripti9e metadata, metrics, and authoritati9e or appro9ed sources are pro9ided for each" A#-#' $ut+ut Data This su.section identifies the M+) output data, including a definition, the unit of measure, and the range of 9alues for each data item" A#6 Configuration Management This section includes a .rief description of the M+) configuration management program, lists the M+) artifacts and products that are under configuration management, identifies documentation and reporting re5uirements that impact the --+0 effort, and pro9ides contact information" 11 Final Draft - 18 May 2006 APPENDIX % M&S E.UIEMENTS TACEA%ILIT! MATIX This appendi( traces /e"g", in a ta.le2 the accepta.ility criteria and their measures and metrics to each M+) re5uirement" The M+) 1e5uirements Tracea.ility Matri( pro9ides a 9isual demonstration of the degree to which a relationship can .e esta.lished .etween the M+) re5uirements and associated accepta.ility criteria for accreditation /and e9entually will 9isually demonstrate the totality of e9idence collected during -+- implementation2" 0s the M+) re5uirements pro9ide the .ac:.one of accreditation and its associated -+- effort, the clearer the articulation and detailed le9el of re5uirements and accepta.ility criteria, the more accurate and rele9ant the -+-" #n this light, an M+) 1e5uirements Tracea.ility Matri( can .e an in9alua.le tool in capturing the Hone to oneI tie .etween the 0ccepta.ility 6riteria and the M+) 1e5uirements" APPENDIX C %ASIS $9 C$MPAIS$N This appendi( descri.es the .asis of comparison used for 9alidation" The .asis for comparison ser9es as the reference against which the accuracy of the M+) representations is measured" The .asis of comparison can come in many forms, such as the results of e(periments, theory de9eloped from e(periments, 9alidated results from other M+), and e(pert :nowledge o.tained through research or from )M's" APPENDIX D E9EENCES This appendi( identifies all of the references used in the de9elopment of this document" APPENDIX E AC$N!MS This appendi( identifies all acronyms used in this document" APPENDIX 9 4L$SSA! This appendi( contains definitions that aid in the understanding of this document" APPENDIX 4 V&V P$4AMMATICS This appendi( contains detailed information regarding resource allocation and funding" Planned Resource Allocations and unding V&V Acti!ity Required Resources unding Source "#$ %& % "#$ %& "#$ %& "#$ %& APPENDIX 3 DISTI%UTI$N LIST This appendi( pro9ides the distri.ution list for hardcopies of the appro9ed document" APPENDIX I ACCEDITATI$N PLAN This appendi( pro9ides a copy of or a reference to the 0ccreditation $lan for the simulation for which this document has .een prepared" 12