BOP 413.6, Analysis of Alternatives

BUSINESS OPERATING PROCEDURE Approved: 3-14-16 Analysis of Alternatives NATIONAL NUCLEAR SECURITY ADMINISTRATION Office of Cost Estimating and Program Evaluation BOP 413.6 THIS PAGE INTENTIONALLY LEFT BLANK 1 BOP 413.6 3-14-16 ANALYSIS OF ALTERNATIVES 1. PURPOSE. This Business Operating Procedure (BOP) describes the process for conducting an Analysis of Alternatives (AoA). 2. CANCELLATION. None. 3. APPLICABILITY. a. Federal. This applies to all NNSA Elements. b. Contractors. Does not apply to contractors. c. Equivalency. In accordance with the responsibilities and authorities assigned by Executive Order 12344, codified at 50 United States Code sections 2406 and 2511 and to ensure consistency through the joint Navy/Department of Energy (DOE) Naval Nuclear Propulsion Program, the Deputy Administrator for Naval Reactors (Director) will implement and oversee requirements and practices pertaining to this Business Operating Procedure (BOP) for activities under the Director's cognizance, as deemed appropriate. 4. SUMMARY OF CHANGES. Not applicable. 5. BACKGROUND. The principal customers of an AoA are the Deputy Secretary, the Administrator, the Principal Deputy Administrator, Deputy and Associate Administrators, Project Management Executives (PMEs), Federal Project Directors, and Project and Program Managers. The analysis will inform alternative selection during the acquisition process through the provision of factual and objective data. NNSA Elements are encouraged to develop their own standard operating procedures specific to their organization. At a minimum, these procedures must provide guidance on what projects, programs, or acquisition processes require an AoA, unless otherwise directed by broader NNSA or DOE Directives. These procedures will provide guidance on when the AoA will be executed in any given acquisition process. AoAs provide decision-makers with reliable and objective assessments of options for meeting mission needs gaps. An AoA is a key first step in the acquisition process intended to assess alternative solutions for addressing a validated need. The process entails identifying, analyzing, and selecting a preferred alternative to best meet the mission need by comparing the operational effectiveness, costs, and risks of potential alternatives.1 1 GAO-15-37, DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices, December 11, 2014 2 BOP 413.6 3-14-16 6. REQUIREMENTS. a. When an AoA is conducted, the mission need for the project or program must be defined and a gap analysis completed prior to developing the AoA. The team responsible for the AoA must conduct the analysis without a predetermined solution in mind. For programs and projects less than $50 million (at the upper end of the cost range), the scope and complexity of the study should be commensurate with the size and complexity of the program or project. The latter must be reflected in the Tasking Memo as well as the Study Plan. b. The Program Office and the Program Management Executive (PME) must provide the AoA Team with sufficient time and resources to ensure a robust and complete analysis. The AoA must be conducted independent of the contractor organization responsible for managing or executing the project or program. c. The AoA process must be consistent with published Government Accountability Office (GAO) best practices.2 When GAO best practices cannot be followed, any

deviations must be justified and documented in the Study Plan and the Final Report. d. The Project Owner must request the AoA action from the PME who will then assemble an AoA Team and identify a Federal AoA Lead. Guidance and oversight for the AoA process must be provided by the Steering Committee. The composition of the AoA Team and the Steering Committee must be as follows: (1) The Steering Committee Chaired by the PME and has representatives from the Project Owner, the Program Office, the Office of Cost Estimating and Program Evaluation (CEPE), the Office of Management & Budget (NA-MB), and other program and field offices as needed. For potential capital asset projects as defined in DOE Order 413.3B (or successor order), the Office of Acquisition & Project Management (NA-APM) will have representation. For potential capital asset projects greater than $100 million (at the upper end of the cost range), the Department of Energy Office of Project Management, Oversight, and Assessments (PMOA) may have a representative, as needed. (2) The AoA Team Includes members with a variety of necessary skill sets, specific knowledge, and abilities to successfully execute the AoA. For example, the AoA team includes individuals with skills and experience in the following areas: program management, federal contracting, cost 2 GAO-16-22, Amphibious Combat Vehicle: Some Acquisition Activities Demonstrate Best Practices; Attainment of Amphibious Capability to be Determined, October 28, 2015 3 BOP 413.6 3-14-16 estimating, risk management, sustainability, scheduling, operations, technology, earned value management, budget and financial analysis, and any other necessary areas of expertise. The Steering Committee may recommend members of the AoA Team. The size and composition of the AoA Team should be dictated by the size and complexity of the project. Subject matter experts may be provided by Federal staff and supplemented with contractor support. d. The following AoA deliverables must be produced (See Appendix 1, “Analysis of Alternatives Process and Deliverables”): (1) Tasking Memo: Responsible Party: Project Owner The Project Owner must draft a Tasking Memo that requests the PME to perform an AoA. The Tasking Memo must describe the mission need, the mission gaps, the assumptions, and constraints that drive the AoA, the resources available to the AoA Team, and the schedule and expected deliverables for the AoA process. (2) Study Plan: Responsible Parties: AoA Team, PME, CEPE Director The AoA Team must produce an initial Study Plan at the beginning of the AoA process. Upon its completion, it must be reviewed by the Steering Committee and approved by the PME and the Director, CEPE. The Study Plan must include the following: - Description of the mission need, program requirements, gap analysis, and assumptions and constraints that are driving the AoA, as set out by the Tasking Memo; - Description of the names, affiliations, roles, and responsibilities of the AoA Team members and Steering Committee; - Description of the oversight and review process for the AoA; - Identification of the data and resources the AoA Team will need to complete the study; - All selection and evaluation criteria that represents the mission need and program requirements; - If applicable, description of how to incorporate the National Environmental Protection Act (NEPA) in accordance with the Council on Environmental Quality guidance;

4 BOP 413.6 3-14-16 - Sufficiently detailed description of the methodology the AoA Team intends to use for each phase of analysis, and; - If applicable, description of any deviations from GAO’s best practices that cannot be met, and accompanying justifications. (3) Final Report: Responsible Parties: AoA Team, Steering Committee, PME The AoA Team must document and present the results of the AoA to the Steering Committee in a Final Report. The PME approves the Final Report. The Final Report should be a standalone document that integrates the results of the analysis, justifications, supporting documentation, and all previous project deliverables including the AoA Study Plan. The Final Report must include the following: - Names of all AoA Team members with corresponding affiliations and roles; - Description of the mission need, program requirements, gap analysis, and assumptions and constraints that are driving the AoA, as set out by the Tasking Memo; - Descriptions of all alternatives considered; - All selection criteria, justifications driving the initial screening process, and results; - All evaluation criteria and relation to mission needs and program requirements; - Complete life-cycle assessment costs of each alternative (after the initial down-select), including justifications; - Summary of risks for each alternative; - Sensitivity analysis; - The final results of the evaluation; - If applicable, description of any deviations from GAO’s best practices that cannot be met (including accompanying justifications), and; - All previous final deliverables (e.g., Study Plan). 5 BOP 413.6 3-14-16 (4) Sufficiency Memo: Responsible Party: CEPE Director After reviewing the completed AoA Final Report, the Director, CEPE must evaluate the AoA for completeness, quality, technical soundness, and adherence to established processes and policies (see Appendix 3, “Government Accountability Office 22 Best Practices for the Analysis of Alternatives (AoA) Process.”) Any dissenting opinions within the Steering Committee must be captured in the Sufficiency Memo. The Director, CEPE must submit the Sufficiency Memo to the PME and the Administrator. (5) Alternative Selection Document: Responsible Party: PME The PME must consider the results of the AoA Final Report to select a preferred alternative and document the choice in the Alternative Selection Document. 7. RESPONSIBILITIES. a. Office of Acquisition and Project Management (NA-APM). Provides subject matter and project-specific expertise to the AoA Team, to include serving on the Steering Committee for potential capital asset projects and the AoA Team, unless NA-APM determines participation is not necessary. b. Office of Management and Budget (NA-MB). Provides subject matter and project-specific expertise to the AoA Team, to include serving on the Steering Committee as a permanent member and the AoA Team, unless NA-MB determines participation is not necessary. c. Director, Office of Cost Estimating and Program Evaluation (CEPE). Develops and maintains AoA processes and procedures and serves on or delegates a representative to the Steering Committee. Along with the PME, the Director approves the Study Plan. Upon completion of the AoA, the Director submits a Sufficiency Memo to the PME and the Administrator. d. Program Office. Conducts a mission gap analysis that guides the AoA and is responsible for funding the AoA effort.

e. Steering Committee. The Steering Committee provides oversight and guidance throughout the AoA process may recommend members to serve on the AoA Team, conducts the Preliminary Results Review, and reviews the Final Report prior to its approval by the PME. 6 BOP 413.6 3-14-16 f. Analysis of Alternatives Team. Conducts and executes the AoA in accordance with this BOP. The AoA Team drafts the Study Plan, completes the AoA analysis, presents at the Preliminary Results Review, and documents all findings in a Final Report. g. Federal Analysis of Alternatives Lead. Leads the AoA Team and serves as the primary point of contact for the Steering Committee on all issues relating to the AoA Team. h. Project Management Executive (PME). Assembles the AoA Team, designates a Federal AoA Lead, and serves as the Chair of the Steering Committee. Along with the CEPE Director, the PME approves the Study Plan. Upon completion of the AoA, the PME approves the Final Report, selects an alternative, and documents the choice in the Alternative Selection Document. For some projects, the Project Owner and the PME may be the same individual. i. Project Owner. Issues the AoA Tasking Memo and leads a kick-off meeting with the AoA Team and Steering Committee. For some projects, the Project Owner and the PME may be the same individual. 8. REFERENCES. a. 77 FR 14473, Final Guidance on Improving the Process for Preparing Efficient and Timely Environmental Reviews Under the National Environmental Policy Act, Council on Environmental Quality, March 12, 2012. b. GAO-16-22, Amphibious Combat Vehicle: Some Acquisition Activities Demonstrate Best Practices; Attainment of Amphibious Capability to be Determined, October 28, 2015 c. GAO-15-37, DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices, December 11, 2014. d. DOE O 413.3B, Program and Project Management for the Acquisition of Capital Assets, November 29, 2010. e. DOE Guide 413.3-17, Mission Need Statement Guide, U.S. Department of Energy, Washington, D.C., June 20, 2008. f. DOE Secretarial Memorandum, Improving the Department’s Management of Projects, December 1, 2014. g. DOE Secretarial Memorandum, Project Management Policies and Principles, June 8, 2015. 7 BOP 413.6 3-14-16 9. DEFINITIONS. a. Alternative Selection Document-A document recording the PME's selection of an alternative upon the completion of the AoA. b. Analysis of Alternatives (AoA) - An analytical tool used by decision makers to make technical investment decisions based on many factors such as mission gaps, risk, cost, effectiveness, technology maturity, etc. Unless otherwise directed via NNSA or DOE Directive, NNSA Elements must determine what projects or programs and, at what point in the acquisition process, require an AoA. C. Federal Analysis of Alternatives Lead - Federal designee chosen by the PME that is responsible for the end-to-end execution of the AoA. d. Gap Analysis - A systematic study of assessing current or near term expected capabilities compared to those capabilities required to perfo1m a new mission. e. Life-Cycle Cost Estimate - The overall estimated cost for a particular alternative over the time period corresponding to the life of the alternative, including direct and indirect initial costs plus any periodic or continuing cost of operation and maintenance. f. Net Present Value - The net present value is the difference between the

discounted present value of benefits and the discounted present value of costs. g. Present Value - The present value of an estimate reflects the time value of money, the concept that a dollar in the future is wo11h less than a dollar today because the dollar today can be invested and earn interest. h. Project Management Executive -The individual designated to integrate and unify the management system for a program p01tfolio of projects and implement prescribed policies and practices. F01merly known as the Acquisition Executive. i. Project Owner - The mission stakeholder responsible for identifying the requirements and requesting the necessary budget to fulfill the mission need. 10. CONTACT. Director, Office of Cost Estimating and Program Evaluation (NA-1.3), 202-586-6910. BY ORDER OF THE ADMINISTRATOR: 8 BOP 413.6 3-14-16 APPENDICES. 1. Analysis of Alternatives Process and Deliverables 2. Analysis of Alternatives Analysis Steps 3. Government Accountability Office 22 Best Practices for the Analysis of Alternatives BOP 413.6 3-14-16 Appendix 1 AP1-1 APPENDIX 1: ANALYSIS OF ALTERNATIVES PROCESS AND DELIVERABLES Analysis of Alternatives Process: (1) The Project Owner initiates the AoA by issuing a Tasking Memo to the PME. The PME assembles the AoA Team and selects a Federal AoA Lead to lead the AoA Team. (2) The Project Owner or representative leads a kick-off meeting with the AoA Team and Steering Committee to communicate and clarify the contents of the Tasking Memo. (3) As the AoA process moves forward (see Appendix 2, “Analysis of Alternatives Analysis Steps”), the AoA Team documents any and all decision making and analysis throughout the process in order to validate the AoA’s objectivity. The Federal AoA Lead also keeps the Steering Committee updated throughout the process. The Steering Committee may request additional reviews throughout the process, as warranted. (4) The AoA Team reviews the mission need and defines selection criteria, which will be used to pre-screen alternatives. The selection criteria are: based on the mission need and independent of a particular capital asset or technological solution. (5) The AoA Team develops evaluation criteria based on the unique characteristics and program requirements of the assessed project and assigns weights to each criterion based on its relative importance to the mission need. At a minimum, these criteria must account for performance risks, cost, and schedule while directly reflecting mission need and program requirements. (6) The AoA Team drafts an initial Study Plan and presents it to the Steering Committee for review. The Steering Committee may provide further guidance or clarifications to the AoA Team to ensure that the process is conducted without any preconceived solutions or other evident bias. The PME and the Director, CEPE approve the Study Plan. (7) The AoA Team develops a diverse range of plausible and preliminary alternatives that could potentially meet the mission need. One alternative must represent the status quo (no alterations to current efforts) in order to provide a basis of comparison for the other alternatives. All alternatives and their descriptions must be documented. (8) The AoA Team screens the list of preliminary alternatives against the set of selection criteria. The status quo alternative should not be screened out in order to continue to provide a basis of comparison. The justification for

the results of the screening process must be documented. Appendix 1 AP1-2 BOP 413.6 3-14-16 (9) The AoA Team quantitatively analyzes and ranks the screened alternatives using the set of evaluation criteria. This analysis must be informed by a summary of quantified benefits, life-cycle cost estimates generated for each alternative, and a list of associated risks with mitigation strategies for each alternative. Alternatives should be compared using net present values, if possible. The team should use methods and techniques from industry standards and best practices as well as any applicable DOE or NNSA policies, procedures and processes. (a) The AoA Team determines each alternative’s benefits using a standardized process and documenting the rationale behind the assessment. Benefits must be quantifiable and determined over the alternative’s full life-cycle. Benefits must relate to and support the mission need. (b) The life-cycle cost estimates for each alternative should include all costs from inception of the project through design, development, deployment, operation, maintenance, and disposition. Life-cycle cost estimates should be shown in present value terms over the entire project life-cycle. Cost estimates should be expressed as a range or with a confidence interval, not solely as a point estimate. The AoA Team must document the basis, assumptions, and calculations used. (c) The AoA Team identifies a list of significant risks (programmatic, technical, and operational) and mitigation strategies for each alternative3. (10) The AoA Team conducts a sensitivity analysis that tests the sensitivity of the cost and benefit estimates and the evaluation scores and weights to changes in key assumptions and criteria4. (11) The AoA Team presents its initial findings to the Steering Committee for a Preliminary Results Review. The Steering Committee provides additional guidance to the AoA Team, as needed. (12) The AoA Team documents the results of the analysis with accompanying justifications, calculations, and supporting documentation in the AoA Final Report. The Steering Committee reviews the Final Report and the 3 Treatment of risk management techniques is outside the scope of this BOP. Each AoA should address these risks and mitigation strategies in a manner commensurate with the size of the effort and adhering to those guidelines provided in DOE G 413.3-7A “Risk Management Guide.” 4 Treatment of sensitivity analysis and decision theory techniques is outside the scope of this BOP. Each AoA should address these issues in a manner commensurate with the size of the effort. A suggested starting point is “WSRC-IM-2002-00002 Guidebook to Decision Making Methods.” BOP 413.6 3-14-16 Appendix 1 AP1-3 PME approves the document. Any dissenting opinion(s) that the Steering Committee would like on record but not documented in the Final Report must be included in the Sufficiency Memo. (13) Upon receiving the Final Report, the Director, CEPE drafts a Sufficiency Memo evaluating the completeness, quality, and technical soundness of the AoA process. The Director, CEPE submits the Sufficiency Memo to the PME and the Administrator. (14) The PME considers the results of the Final Report to select a preferred alternative and documents the choice in the Alternative Selection Document. (15) The AoA Team documents lessons learned throughout the process in a manner that best supports the Program Office.

BOP 413.6 Appendix 1 3-14-16 AP1-4 Stage AoA Initiation AoA Kick-Off AoA Planning AoA Analysis AoA Preliminary Results Review AoA Finalization AoA Review Alternative Selection Description The Project Owner requests the PME to conduct the AoA. The PME chooses an AoA Team and selects a Federal AoA Lead as the team lead. The Project Owner conducts a kick- off meeting with the AoA Team and the Steering Committee. The AoA Team describes the structure and methodology of the AoA in a Study Plan. The PME and the Director of CEPE approve the Study Plan. The AoA Team conducts the AoA analysis by screening and evaluating alternatives, conducting a sensitivity analysis, and documenting the results. The Steering Committee conducts a review of the AoA Team's initial results, providing additional guidance as necessary. The AoA Team compiles the results of the AoA analysis into one document with all supporting information. After receiving review by the Steering Committee, the Final Report is approved by the PME. The Director of CEPE reviews the Final Report and submits a Sufficiency Memo to the PME and the Administrator. The PME chooses an alternative based on the results of the AoA. Deliverable Tasking Memo Study Plan Final Report Sufficiency Memo Alternative Selection Document THIS PAGE INTENTIONALLY LEFT BLANK BOP 413.6 Appendix 2 3-14-16 AP2-1 APPENDIX 2: ANALYSIS OF ALTERNATIVES ANALYSIS STEPS THIS PAGE INTENTIONALLY LEFT BLANK BOP 413.6 3-14-16 Appendix 3 AP3-1 APPENDIX 3: GOVERNMENT ACCOUNTABILITY OFFICE 22 BEST PRACTICES FOR THE ANALYSIS OF ALTERNATIVES (AoA) PROCESS Below are the best practices as prescribed by the GAO on GAO-16-22: “Amphibious Combat Vehicle: Some Acquisition Activities Demonstrate Best Practices; Attainment of Amphibious Capability to be Determined”, Appendix I: “Best Practices for the Analysis of Alternatives Process”. This document was publicly released Oct 28, 2015. Phase I. Initialize the AOA process 1. Define mission need Definition: The customer defines the mission needs (i.e., a credible gap between current capabilities and those required to meet the goals articulated in the strategic plan) without a predetermined solution. To ensure that the AoA process does not favor one solution over another, the AoA is conducted before design and development of the required capabilities. The customer decides at which level of design completion an AoA should be performed, with the understanding that the more complete the design, the more information is available to support a robust analysis and to select a preferred alternative that best meets the mission need. Effect: Allowing mission needs to be defined in solution-specific terms creates a potential bias and could invalidate the analysis. 2. Define functional requirements Definition: The customer defines functional requirements (i.e. the general parameters that the selected alternative must have to address the mission need) based on the mission need without a predetermined solution. The customer defines the capabilities that the AoA process seeks to refine through characterized gaps between capabilities in the current environment and the capabilities required to meet the stated objectives for the future environment. These functional requirements are realistic, organized, clear, prioritized, and traceable. It is advisable that functional requirements be set early in the

AoA process and agreed upon by all stakeholders. Effect: The AoA process is tied to the identified mission needs. Setting functional requirements to a standard other than mission needs allows bias to enter the study because the requirements might then reflect arbitrary measures. Additionally, requirements not tied to mission needs make it difficult to quantify the benefits of each alternative relative to what is required and make it challenging for decision makers to assess which capability gaps will be met for each alternative. Appendix 3 AP3-2 BOP 413.6 3-14-16 3. Develop AoA time frame Definition: The customer provides the team conducting the analysis enough time to complete the AoA in order to ensure a robust and complete analysis. Since an AoA process requires a large team with many diverse resources and expertise, the process requires sufficient time to be accomplished thoroughly. A detailed schedule is developed prior to starting the AoA process. The duration of the AoA process depends on the number of viable alternatives and availability of the team members. The time frame is tailored for the type of system to be analyzed and ensures that there is adequate time to accomplish all of the AoA process steps robustly. Effect: The AoA process identifies and thoroughly analyzes a comprehensive range of alternatives. Recommending an alternative without adequate time to perform the analysis is a contributing factor to high dollar acquisitions that have significantly overrun both cost and schedule while falling short of expected performance. 4. Establish AoA team Definition: After the customer establishes the need for the AOA in steps 1 through 3, a diverse AoA team is established to develop the AoA. This team consists of members with a variety of necessary skill sets, specific knowledge, and abilities to successfully execute the study. For example, the AoA team includes individuals with skills and experience in the following areas: program management, federal contracting, cost estimating, risk management, sustainability, scheduling, operations, technology, earned value management, budget analysis, and any other necessary areas of expertise. Effect: An AoA process includes a diverse group of subject matter experts (SMEs) to perform the analysis. Since each SME brings their knowledge to the team, without the appropriate expertise on the team, errors in the results could occur and gaps in the analysis could be created, causing the AoA’s completion to be delayed as more SMEs are identified and tasked to work as part of the AoA process. 5. Define selection criteria Definition: The AoA team or the decision maker defines selection criteria based on the mission need. The defined criteria are based on mission needs and are independent of a particular capital asset or technological solution. The selection criteria are defined based on the mission need prior to starting the analysis. Effect: It is essential that the selection criteria be based on the mission needs. If there are no preset criteria based on documented requirements, bias can enter the AoA process and prevent the decision maker from forming an impartial and unbiased decision. BOP 413.6 3-14-16 Appendix 3 AP3-3 6. Weight selection criteria Definition: The AoA team or the decision maker weights the selection criteria to reflect the relative importance of each criterion. While the selection criteria are ranked in

importance, the alternatives are based on trade-offs between costs, operational effectiveness, risks, schedules, flexibility, and other factors identified by the team or the decision maker. Effect: An unjustified weighting method can oversimplify the results and potentially mask important information leading to an uninformed decision. 7. Develop AoA process plan Definition: The AoA team creates a plan to include proposed methodologies for identifying, analyzing, and selecting alternatives prior to beginning the AoA process. This plan establishes the critical questions to be explored, the selection criteria, the basis of estimates, and measures that are used to rate, rank, and decide among the alternatives. Additionally, the plan includes the criteria used to determine each alternative’s viability. A road map and standard work breakdown structure (WBS) are used to compare the alternatives with the baseline and with each other. Effect: The functional requirements and selection criteria are identified prior to the beginning of the analysis. If criteria to select the preferred alternative are established after the analysis has begun bias may influence the study’s results. Furthermore, if planned methodologies for the remaining phases of the AoA study are not established, the risk of applying poor methodologies as part of the AoA analysis increases. Phase II. Identify alternatives 8. Develop list of alternatives Definition: The AoA team identifies and considers a diverse range of alternatives to meet the mission need. To fully address the capability gaps between the current environment and the stated objectives for the future environment, market surveillance and market research is performed to develop as many alternative solutions as possible for examination. Alternatives are mutually exclusive, that is, the success of one alternative does not rely upon the success of another. Effect: An AoA process encompasses numerous alternatives in order to ensure that the study provides a broad view of the issue. If the AoA team does not perform thorough research to capture diverse alternatives, the optimal alternative could be overlooked and invalidate the AoA’s results and bias the process. Appendix 3 AP3-4 BOP 413.6 3-14-16 9. Describe alternatives Definition: The AoA team describes alternatives in sufficient detail to allow for robust analysis. All alternatives’ scope is described in terms of functional requirements. This description is detailed enough to support the viability, cost, and benefit/effectiveness analyses. Effect: Documentation is essential for validating the AoA process and defending its conclusions. Unless the AoA team adequately describes and documents the alternatives, the analysis will not provide sufficient detail to allow for valid cost-benefit estimates and will not be credible. 10. Include baseline alternative Definition: The AoA team includes one alternative to represent the status quo to provide a basis of comparison among alternatives. It is critical for the AoA team to first understand the status quo, which represents the existing capability’s baseline where no action is taken, before comparing alternatives. The baseline is well documented as an alternative in the study and is used to represent the current capabilities and also for explicit comparison later in the study. Effect: It is essential that the AoA process compare the current environment with the

possible future environment. If no status quo is examined, then there is no benchmark for comparison, allowing arbitrary comparisons between alternatives and hindering the credibility of the study. 11. Assess alternatives’ viability Definition: The AoA team screens the list of alternatives to eliminate those alternatives that are not viable, and it documents the reasons for eliminating any alternatives. All alternatives are examined using predetermined qualitative technical and operational factors to determine their viability. Only those alternatives found viable are examined fully in the AoA process. However, all assumptions regarding the alternatives’ viable and nonviable status are fully documented, including reasons that an alternative is not viable, in order to justify the recommendation. Additionally, viable alternatives that are not affordable within the projected available budget are dropped from final consideration. Effect: Not eliminating alternatives based on viability could needlessly extend the study’s duration and burden the AoA team or lead to the selection of a technically nonviable alternative. Furthermore, unless the AoA team considers affordability as part of the final recommendation, an alternative that is not feasible based on the current fiscal environment could be selected. Documenting the alternatives that are deemed nonviable BOP 413.6 3-14-16 Appendix 3 AP3-5 is important so that decision makers can clearly see why those alternatives are not considered for further analysis. Phase III. Analyze alternatives 12. Identify significant risks and mitigation strategies Definition: The AoA team identifies and documents the significant risks and mitigation strategies or each alternative. Risks are ranked in terms of significance to mission needs and functional requirements. All risks are documented for each alternative along with any overarching or alternative specific mitigation strategies. Schedule risk, cost risk, technical feasibility, risk of technical obsolescence, dependencies between a new project and other projects or systems, procurement and contract risk, and resources risks are examined. Effect: Since AoA processes typically occur early in the acquisition process, risk is inherently a part of every alternative. Not documenting the risks and related mitigation strategies for each alternative prevents decision makers from performing a meaningful trade-off analysis necessary to choose a recommended alternative. 13. Determine and quantify benefits/effectiveness Definition: The AoA team uses a standard process to document the benefits and effectiveness of each alternative. The AoA team drafts a metric framework that details the methods used to evaluate and quantify the measures of effectiveness and measures of performance for all mission needs. The AoA team quantifies the benefits and effectiveness of each alternative over the alternative’s full life-cycle, if possible. Just as costs cover the entire life-cycle for each alternative, the benefits and effectiveness measures cover each alternative’s life-cycle, if possible, in order to determine each alternative’s net present value (NPV)—the discounted value of expected benefits minus the discounted value of expected costs. In cases where the means to monetize a benefit are too vague (for example, intangibles like scientific knowledge), the AoA team treats those benefits as strategic technical benefits and uses scalability assessments to quantify

those benefits so that they are compared across all viable alternatives. In situation where benefits cannot be quantified, the AoA team explains why this is the case as part of their analysis. Effect: Determining a standard process to quantify benefits is an essential part of the AoA process. If the AoA team does not clearly establish criteria against which to measure all alternatives, bias is introduced to the study. Additionally, if the AoA team does not examine effectiveness over the entire life-cycle, decision makers cannot see the complete picture and are prevented from making an informed decision. Appendix 3 AP3-6 BOP 413.6 3-14-16 14. Tie benefits/effectiveness to mission need Definition: The AoA team explains how each measure of effectiveness supports the mission need. The AoA team shows how the measures of effectiveness describe the way the current environment is expected to evolve to meet the desired environment; the team also shows how the measures are tied to specific mission needs and functional requirements. This is the hierarchy that connects the overarching requirements to the data that are needed. Effect: Unless the AoA team thoroughly documents how the measures of effectiveness relate to specific mission needs and functional requirements, decision makers will not have proper insight into the impact of each alternative. 15. Develop life-cycle cost estimates (LCCEs) Definition: The AoA team develops a LCCE for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and disposal. The AoA team includes a cost expert who is responsible for development of a comprehensive, well-documented, accurate, and credible cost estimate for each viable alternative in the study. The LCCE for each alternative follows the GAO 12-step guide and uses a common cost element structure for all alternatives and includes all costs for each alternative. Costs that are the same across the alternatives (for example, training costs) are included so that decision makers can compare the total cost rather than just the portion of costs that varies across all viable alternatives. The AoA team expresses the LCCE in present value terms and explains why it chose the specific discount rate used. The AoA team ensures that economic changes, such as inflation and the discount rate are properly applied, realistically reflected, and documented in the LCCE for all alternatives. Furthermore, the present value of the estimate reflects the time value of money—the concept that a dollar today can be invested and earn interest. Effect: An LCCE that is incomplete (i.e. does not include all phases of an alternative’s life-cycle) does not provide an accurate and complete view of the alternatives’ costs. Without a full accounting of life-cycle costs, decision makers will not have a complete picture of the costs for each alternative and will have difficulty comparing the alternatives because comparisons may not be based on accurate information. Additionally, applying a discount rate is an important step in cost estimating because all cost data must be expressed in like terms for comparison. Unless the AoA team properly normalizes costs to a common standard, any comparison would not be accurate, and any recommendations resulting from the flawed analysis would be negated. Properly normalizing costs is particularly important if various alternatives have different life-

cycles. BOP 413.6 3-14-16 Appendix 3 AP3-7 16. Include a confidence interval or range for LCCEs Definition: The AoA team presents the LCCE for each alternative with a confidence interval or range, and not solely as a point estimate. To document the level of risk associated with the point estimate for each viable alternative, the confidence interval is included as part of the LCCE for each viable alternative (in accordance with GAO Cost Estimating Best Practice #9, risk and uncertainty analysis). Decision makers must have access to the confidence interval associated with the point estimates for all viable alternatives in order to make informed decisions. Additionally, the AoA team uses a consistent method of comparing alternatives in order to present a comparable view of the risk associated with each alternative. For example, the comparison can be based on an established dollar value across alternatives (in order to observe the confidence level for each alternative at that dollar value). Alternatively, the comparison can be based on a predetermined confidence level across alternatives (in order to observe the dollar value associated with that confidence level for each alternative). Effect: For decision makers to make an informed decision, the alternatives’ LCCEs must reflect the degree of uncertainty. Having a range of costs around a point estimate is useful because it conveys a level of confidence for each alternative to achieve a most likely cost. Without cost risk and uncertainty analysis the LCCEs for the viable alternatives are not credible. 17. Perform sensitivity analysis Definition: The AoA team tests and documents the sensitivity of the cost and benefit and effectiveness estimates for each alternative to risks and changes in key assumptions. Major outcomes and assumptions are varied in order to determine each alternative’s sensitivity to changes in key assumptions. This analysis is performed in order to rank the key drivers that could influence the cost and benefit estimates based on how they affect the final results for each alternative. Each alternative includes both a sensitivity and risk and uncertainty analysis that identifies a range of possible costs based on varying key assumptions, parameters, and data inputs. As explained in best practice #16, life-cycle cost estimates are adjusted to account for risk and sensitivity analyses. Effect: Failing to conduct a sensitivity analysis to identify the uncertainties associated with different assumptions increases the chance the AoA team will recommend an alternative without an understanding of the full impacts on life-cycle costs, which could lead to cost and schedule overruns. Appendix 3 AP3-8 BOP 413.6 3-14-16 Phase IV. Document and review the AoA process 18. Document AoA process in a single document Definition: The AoA team documents all steps taken to identify, analyze, and select alternatives in a single document. This document clearly states the preferred alternative and provides the detailed rationale for the recommendation based on analytic results. The report includes sections detailing the steps taken to initialize the AoA process, and to identify, analyze, and select alternatives. For example, one section lists the overall selection criteria and rationale for nonviable or viable ratings for alternatives, assumptions for each alternative, risk drivers and mitigation techniques, analysis of the

costs and benefits associated with each alternative, and the trade-offs between costs, benefits, and risks. Effect: Documentation is essential for validating and defending the AoA process. Without clear reports that compile all information, including standards used to rate and perform the analysis, the study’s credibility could suffer because the documentation does not explain the rationale for methodology or the calculations underlying the analysis. Having all the information related to all best practices of the AoA process in one single document also makes it easier for an independent reviewer to assess the AoA process. 19. Document assumptions and constraints Definition: The AoA team documents and justifies all assumptions and constraints used in the AoA process. Assumptions and constraints help to scope the AoA. Assumptions are explicit statements used to specify precisely the environment to which the analysis applies, while constraints are requirements or other factors that cannot be changed to achieve a more beneficial approach. Both assumptions and constraints are detailed and justified for each alternative in the AoA plan. Effect: Without documented and justified assumptions and constraints it will be difficult for decision makers to evaluate between the alternatives. 20. Ensure AoA process is impartial Definition: The AoA team conducts the analysis without a predetermined solution. The AoA process informs the decision-making process rather than reflecting the validation of a predetermined solution. The AoA process is an unbiased inquiry into the costs, benefits, and capabilities of all alternatives. Effect: An AoA process is not considered valid if it is biased. Performing a study with a predetermined solution distorts the results. The validity of the analysis is affected if bias is introduced to the inputs. BOP 413.6 3-14-16 Appendix 3 AP3-9 21. Perform independent review Definition: An entity independent of the AoA process reviews the extent to which all best practices are followed. The AoA process is completed with enough thoroughness to ensure that an independent organization outside of the project’s chain of command can review the AoA documentation and clearly understand the process and rationale that led to the selection of the recommended alternative. Part of the documentation includes approval and review from an office outside of the one that asked for or performed the AoA process. For certain projects, in addition to an independent review at the end of the AoA process, additional independent reviews are necessary at earlier stages of the process, such as reviews of the AoA process plan of the identification of viable alternatives. While early reviews are not a substitute for the independent review conducted at the end of the AoA process, they help ensure that bias is not added throughout the course of the AoA process. Effect: An independent review is one of the most reliable means to validate an AoA process. Without an independent review, the results are more likely to include organizational bias or lack the thoroughness needed to ensure that a preferred solution is chosen and not a favored solution. Phase V. Select a preferred alternative 22. Compare alternatives Definition: The AoA team or the decision maker compares the alternatives using NPV, if possible, to select a preferred alternative. NPV can be negative if discounted costs are greater than discounted benefits. NPV is the standard criteria used when deciding whether an alternative can be justified based on economic principles. In some cases, NPV cannot be used, such as when quantifying benefits is not possible. In these cases, the AoA team documents why NPV cannot be used. Furthermore, if NPV is not used to differentiate among alternatives, the AoA team should document why NPV is not used, and describe the other method that is used to differentiate, and explain why that method has been applied. Effect: Comparing items that have not been discounted (or normalized) does not allow for time series comparisons since alternatives may have different life cycles or different costs and benefits.