Commercial Analysis Validation

This article discusses how it might be proved that a particular numerical analysis is ‘correct’. This is called analysis validation and can take many forms. Physical testing is the most obvious and convincing means of showing that an analysis is accurate. This is often not viable or cost-effective, however. Other options are FE analyses of similar structures that have been validated through testing or independent calculations.

A Formal Procedure for Validation

The SAFESA documents mentioned in the last article (NAFEMS Documents Ref. R0039 , R0040 , R0041 ) introduce new definitions and concepts for validating numerical structural analyses. Perhaps the most important concept is distinguishing between uncertainties in the physical description of the structure, and errors in the definition of the model created to represent it. The analyst team will often have full responsibility for the latter and perhaps some of the former.

The SAFESA approach breaks down the whole analysis process into discrete steps, each one requiring validation. This takes the concept of validation to another level. For example, several sources of data all demonstrating that a mesh is suitably refined do not, on their own, represent thorough validation. Instead, validation should include information to show that errors have been considered for each modelling assumption, decision or step in the analysis process.

Validation in a Commercial Environment

Detailed model validation procedures as described in the SAFESA documents are appropriate for safety-critical structures. But in other applications, validation is at the discretion of the analysis team and devising a suitable validation strategy for each project can be difficult. Many analyses that consultancies carry out are novel, where no previous validation from similar structures is available. Despite the important requirement for it in these cases, it can be a challenge to carry out appropriate validation under commercial pressures, typically experienced by consultancy teams.

Consultancies’ clients often have a perception of analysis as being 100% accurate, a view for which they cannot be blamed. In giving the client what they want, it can be tempting to let this view prevail and short-cut the validation process since it provides no obvious improvement in the appearance of the analysis for the client. This is a risky approach however. It is the job of the analysis team to openly discuss with the client (or internal customer) the errors, uncertainties and corresponding risks associated with the full analysis process and, if a budget can be obtained for so doing, attempt to quantify them. Perseverance may be required here to portray validation as a benefit rather than an inconvenience.

A Risk-Based Approach to Analysis

An open discussion of risk can help recalcitrant clients see analysis as a powerful but not an absolute process. If the level of risk of an event occurring can be estimated, then a cost can be associated with it. For example, consider a newly designed moulded part where tooling rework is estimated to cost £50,000. We can make a rough estimate of the risk of this occurring: based on the success rate of the design team with similar past projects, let’s say that the risk of structural problems occurring in the part is 30%. With a good analysis supporting the design process, the risk of the design being structurally inadequate can be reduced to say, 10%. (That is the analysis is likely to be valid in 90% of cases.) The risk reduction value of the analysis is therefore 20% of £50,000, or £10,000. This is then a limiting cost for an analysis of this part.

(As well as discussing structural analysis, this approach could also apply to mould flow simulation, for example.) One digression that springs from this is considering the reasons for an analysis being incorrect. The risk of an analysis project drawing the wrong conclusions is different to the risk of the analysis team being at fault. Unfortunately, good professional conduct does not eliminate all risk. The distinction made previously between uncertainty and error may be useful here: errors may be considered the analysis team’s direct responsibility whereas uncertainties are more difficult to assign.

Analysis Validation to Reduce Risk

Risk assessment can also be applied to analysis validation. Continuing with the previous example, a physical test and further analyses to correlate with it could be carried out to improve confidence in the model. If this improved confidence from 90% to 95%, then the risk reduction value of the validation would be 5% of £50,000, or £2,500. So physical testing could not be justified if it cost more than around £2,500 in this example. In a risk assessment context, hand calculations, sensitivity tests and independent checking procedures, are usually cost-effective validation steps.

A difficulty with risk assessment generally is deciding on actual risk levels – a 5% risk reduction due to physical testing is not easily proven. Despite this, considering the approximate risk associated with different stages of an analysis is a useful process for an analyst. And when used merely to communicate the concept of analysis risk to a client, accurate figures are not required of course.