Verification & Validation of Computational Models

This training course discusses methods available to support decision-making using combinations of numerical simulations, experimental observations and expert judgments. Irrespective of the source(s) of the information, a common need is to manage uncertainty in the decision-making process. The various types of uncertainty originating in simulation models are discussed and techniques are presented to quantify their effects on the decision. Uncertainty comes in three broad categories: numerical, parametric and model-form. Methods to assess uncertainty in experimental testing and diagnose bias during expert elicitation are also overviewed.

This short-course on the Verification and Validation (V&V) of computational models teaches techniques to quantify prediction uncertainty which includes the broad classes of, first, numerical uncertainty caused by truncation effects in the discretization of partial differential equations and, second, parametric uncertainty caused by the variability of model parameters. It focuses on applications in structural mechanics and structural dynamics. The quantification includes the propagation and assessment of how much uncertainty is present in the simulation of an application of interest ( “what are the sources, how much uncertainty is present?”). It includes understanding which effects control the uncertainty (“is it predominantly the mesh discretization, parameter variability, or other phenomena?”) and what can be done to reduce the overall uncertainty (“should the mesh be refined, should small-scale experiments be performed, should model parameters be calibrated and how?”).

Technical topics addressed include:

Definitions and concepts of V&V are not discussed in detail; the short-course focuses, instead, on the implementation and applications of well-established techniques. A pre-requisite is a basic knowledge of the finite element method, or another computational technique, and familiarity with the types of uncertainty that numerical simulations introduce. The illustrations emphasize solid mechanics and structural dynamics even though the techniques presented are general-purpose and can be applied to any simulation. Applications include simulations for nonlinear vibrations, transient dynamics and wind turbine blade vibrations.

The short-course has been taught over 20 times since 2001 at private companies, government institutions, or in conjunction with technical conferences in Europe and the United States.

Course Goals

Upon completion of this course, attendees will be able to:

• Understand the objectives of code verification, model validation, uncertainty quantification
• Develop procedures for practical code verification and solution verification
• Select a particular mesh size, or time step, to discretize the equations-of-motion
• Quantify the effects of truncation error in numerical simulations
• Assess the trade-offs between more computing resources and more small-scale testing
• Describe the validation paradigm of sensitivity analysis, correlation, uncertainty analysis
• Describe the process to select and compute appropriate features from simulation outputs
• Understand techniques for global sensitivity analysis and effect screening
• Explain the role of designs-of-experiments and analysis-of-variance in model validation
• Define appropriate test-analysis correlation metrics for model revision and calibration
• Discuss when model calibration might, or not, be needed

All NAFEMS training courses are entirely code independent, meaning they are suitable for users of any software package.

Courses are available to both members and non-members of NAFEMS, although member organisations will enjoy a significant discount on all fees.

NAFEMS course tutors enjoy a world-class reputation in the engineering analysis community, and with decades of experience between them, will deliver tangible benefits to you, your analysis team, and your wider organisation.

Find out more

Course Program

The contents are presented in 12 lectures (one hour each), tentatively organized as shown. The schedules are always created to allow for ample discussion and interaction with attendees. The instructors reserve the right to modify the contents to address the audience’s needs and preferences.

Lecture 1 - Overview of Verification and Validation

• High-level comments on modeling, simulation and “predictability”
• Overview of Verification and Validation (V&V)
• Definitions, organization of V&V activities
• Which questions does V&V address? What can be learned from V&V?
• Examples of typical studies in solid mechanics and structural dynamics

Lecture 2 - Application of V&V to Wind Turbine Simulations

• Code verification of the finite element software
• Simulation of blade vibration with bounds of numerical uncertainty
• Sensitivity analysis of the numerical simulation
• Calibration of the model using statistical emulators
• Test-analysis correlation and validation assessment
  

Lecture 3 - Code Verification

• Definition of code verification, typical code verification activities
• How to define benchmark code verification problems?
• The Method of Manufactured Solutions (MMS)
• Examples of code verification studies in structural dynamics

Lecture 4 - Solution Verification

• Definition of solution verification, typical solution verification activities
• The concepts of consistency, stability and convergence
• Modified Equation Analysis (MEA) and its implication to quantify truncation effects
• Richardson’s extrapolation applied to numerical solutions
• The Grid Convergence Index (GCI) and bounds of truncation error
• Examples of solution verification in structural dynamics

Lecture 5 - Feature Extraction for Structural Dynamics

• What makes a good feature of the response analyzed?
• Features for linear, stationary dynamics Features for arbitrary time-series analysis
• Temporal moments and other features for fast, transient dynamics
• Application of Principal Component Analysis (PCA)

Lecture 6 - Testing for Structural Dynamics

• What makes a useful measurement?
• Overview of excitation, sensing, data transmission in structural dynamics
• Overview of signal processing
• The coherence function as diagnostics of measurement quality
  

Lecture 7 - Design of Computer Experiments

• Principles of the design of (physical or computer) experiments
• Full-factorial, fractional factorial designs, orthogonal arrays, central composite design
• Formulation of 2^(n-k) designs
• The concept of statistical aliasing
• Examples of designs-of-experiments applied to structural dynamics simulations
  

Lecture 8 - Sensitivity Analysis and Effect Screening

• Rationale for effect screening (“where is an observed variability coming from?”)
• Simple, linear approaches to effect screening
• Analysis-of-variance (ANOVA) using a design-of-experiments
• Main-effect and linear interaction screening
• Application to structural dynamics simulations: what is gained?
  

Lecture 9 - Development of Surrogate Models

• Surrogate modeling using a design-of-experiments
• Diagnostics of quality of an emulator
• Low-order, polynomial emulators
• Kernel regression, Gaussian process modeling
  

Lecture 10 - Sampling and Propagation of Parametric Uncertainty

• Sampling methods for the forward propagation of (parametric) uncertainty
• Monte Carlo, stratified sampling, Latin Hypercube Sampling (LHS)
• Convergence of statistical estimates
• The concept of a confidence interval
• Application of statistical sampling to simulations in solid mechanics
  

Lecture 11 - Test-analysis Correlation and Validation Metrics

• Concepts of response features and validation metrics
• Metrics for structural dynamics and general-purpose test-analysis correlation
• Metrics based on Principal Component Analysis (PCA)
• Statistical tests that account for probabilistic uncertainty
• Model calibration and inference uncertainty quantification
  

Lecture 12 - An End-to-end Example of Verification and Validation

• Engineering example of transient dynamics finite element simulations
• Verification of the finite element software Design and execution of computer experiments (predictions)
• Design of physical experiments (measurements)
• Effect screening and identification of statistically most-significant inputs
• Small-scale validation experiments: what is gained?
• Uncertainty propagation and final validation assessment
  

Lecture 13 - Concluding Remarks

• Summary of main points made during the short-course
• Aspects of V&V not covered, other sources of information
• Closing comments, discussion with attendees, exit survey