Applied Test and Evaluation

Test and evaluation (T&E) provide the means to ensure that a product performs in the intended way and with the intended results. Testing starts very early in a product system development, however, because test planning is dependent on good requirements. This course teaches the details of test and evaluation from product concept through operations, with a view toward application in large system development. The course is highly practical, helping students to understand not only what must be done, but why it is necessary. Students learn about a stakeholder view of test and evaluation, the use of requirements as the primary stay point for verification, planning a test program in terms of strategy, plans, and procedures, the statistical analysis of a test prior to performing the test, the differences between integration testing and final testing, and how to conduct tests at different phases of product/system development, production and support. Practice the skills on a realistic product system. Design a test program for a typical product system. Build the robotic system. Implement the test program. Analyze the results. The Robotic Test Challenge Exercise gives students the opportunity to practice the skills taught in the course from end to end during development and support of actual operating robots. The exercise happens in four segments throughout the course, encompassing a total of six hours in highly practical learning.

6 attendees from your organization as a minimum are needed to take this course. The exercises need two, 3 to 5 person work groups with one Robot kit per work group in order to have a full hands -on experience.

Follow this link for a short course overview video.

• Test Engineers
• Design Engineers (any engineering discipline)
• Systems Engineers, Project Engineers; Technical Team Leaders
• System Support Leaders
• Technical and Management Staff
• Project Managers

• The roles of T&E in development and support
• Nine basic principles of testing
• A useful life-cycle model for T&E
• How to analyze product/system requirements into test requirements that satisfy stakeholders
• T&E development: strategy, plan, procedures
• Test design and analysis; statistical methods
• System integration testing and how it differs from product/system final testing
• Test methods, test control, test results

1. Test & Evaluation Overview (1:30) – An overview of test and evaluation (T&E) principles and methods for simple products and complex systems, including T&E tasks from beginning to end of a project. Basic definitions and concepts for the course, including the differences in test, evaluation, verification, validation, developmental testing (DT&E or “alpha”), and operational testing (OT&E or “beta”). Test and evaluation relationship to other engineering and management disciplines. Cost-effective T&E, and the cost of quality. Nine basic principles of testing.
2. Test and Evaluation In the Life Cycle (1:30) – Roles of test and evaluation throughout product development and support. A useful model of typical test activity phases (requirements, planning, conduct, analysis & reporting) and their correlation with program phases. How test concepts fit into special life cycles: commercial product development, US Department of Defense acquisitions under DoD-5000, evolutionary development, iterative development, and agile development.
3. Developing Test Requirements (2:30) – Requirements as the primary method for measurement and control of product development. Where requirements come from in the systems engineering paradigm. Types of requirements, including traditional, model-based and agile requirements. Stakeholder values in the form of Measures of Effectiveness (MOE) and Measures of Performance (MOP). How to develop a requirements verification matrix (RVM); verification methods (Inspection, Analysis, Demonstration, Test). Test requirements differences for prototypes, first articles, production and support.
4. Designing a Test and Evaluation Program (2:30)– Understanding stakeholder needs for T&E information Creating a T&E strategy by evaluating the product/system development concept in context of the stakeholder needs. An effective outline of T&E strategy topics, including planning for developmental and operational testing phases. An example: the US DoD “Test and Evaluation Master Plan (TEMP)” Trade-offs involved in test decisions; maturity at different phases; level of detail. What is a “verification event”? Converting the strategy to a T&E plan defining specific verification events in terms of requirements tested, time frame, equipment/skills needed, duration, and goals. Identification of test enabling products early enough to affect the development program. Modeling and simulation for test planning. Risk management as a test planning tool.
5. Designing Tests and Evaluations (3:30) – The test procedure as a control for each verification event. Effective topics to include in a test procedure, as an outline for developing the test. Identifying the issues and goals in each verification event. Determining the requirements to include, and what not to include. “Black box” input/output analysis, choosing what to measure, and identification of observability issues. Affecting the product/system design to improve testability. Logical sequencing of the test procedure based on product/system states, input controls, and observable measurements. Review of probability and statistics principles including probability distributions. Analyzing expected variation in test data, statistical design of tests, sampling principles, selecting useful statistical methods, design of experiments, common statistical errors.
6. Integration Testing (1:00) – How to successfully manage the intricate aspects of system integration testing; level of integration planning; managing complex system integration; work-arounds. Development test concepts; five types of integration test planning; preferred order of events; component testing; conducting integration tests for complex systems; work-arounds for anomalies and failures.
7. Test Conduct (2:00) – How to perform testing; differences in testing for prototypes, first article qualification, recurring production acceptance, support; rules for test conduct. Testing for different purposes, verification vs. validation; test records; prerequisites management; test readiness certification, test constraints, test article configuration; troubleshooting and anomaly handling; measures of success and indicators of difficulty; test tools. Test failure analysis.
8. Robotic Test Challenge (6:00) – A hands-on class exercise in small groups, handled in four major segments. Part A analyzes a system concept and requirements, developing an RVM and specific test requirements,. Part B creates an effective test program and test procedures for the product system, including identifying any necessary test equipment to be developed. Part C builds the robotic systems per assembly instructions, as well as building any identified test equipment. Part D implements the test program to evaluate the final robots. 

William “Bill” Fournier is Principal Acquisition Systems Engineering with over 35 years experience. Mr. Fournier taught DoD Systems Engineering full time for over three years at DSMC/DAU as a Professor of Engineering Management. Mr. Fournier has taught Systems Engineering at least part time for more than the last 25 years. Mr. Fournier holds a MBA and BS Industrial Engineering / Operations Research and is DOORS trained. He is a certified CSEP, CSEP DoD Acquisition, LSS GB and XPMP. He is a contributor to DAU/DSMC, defense contractor internal Systems Engineering Courses and Process, and INCOSE publications. Currently, He Is working for DoD Mission Engineering / Systems Engineering office supporting Space and Missile Defense Programs. Bill has written and assessed ~20 SEP/SEMPs.