Code Verification v Solution Verification

A Discussion with the NAFEMS SGMWG

9-minute read

In the next part of this series, the NAFEMS Simulation Governance & Management Working Group hosted a discussion looking at code verification v solution verification. As well as reading the conversation here, you can watch the video on YouTube.

The discussion features William Oberkampf (WO) & Gregory Westwater (GW), members of the NAFEMS Simulation Governance & Management Working Group

Code Verification v Solution Verification: What's the difference?

GW: Welcome again Bill. The last time we talked, you made some statements about code verification versus solution verification that today in our conversation I'd like to understand a little bit better. Let me describe a scenario and then you can tell me where it fits with regards to those verification activities. Let’s say our commercial software provider publishes these verification problems and as part of our verification activities we run a suite of those problems covering the kinds of physics that we're using and we do that anytime we upgrade the simulation software or anytime we upgrade our operating system. So, it's something that we do only in response to these upgrade triggers. Tell me, how does that fit with your definition of verification? Also, is that probably more code verification than it is solution verification?

WO: For people that are just tuning in, let me get everybody on the same page. The two issues that we've been talking about are verification and validation. Verification has the two aspects that you mentioned; code verification and solution verification. Sometimes solution verification is called calculation verification and that term is fine too. Both code and solution verification deal entirely with estimating or finding numerical errors.

It has nothing to do with the physics assumptions in the construction of the mathematical model. It's entirely software, numerical approximations, convergence, order of accuracy, and all of those kinds of things. So, this has nothing to do with physics. Given the physics model then we have differential equations that we cannot solve analytically so we solve them approximately with numerical methods which are then implemented in software.

Now, about your question regarding code verification where the code verification is done mostly by the software companies themselves, or whatever group is developing the software. In that case that's a primary responsibility that they have, but the responsibility is also shared by organizations like yours. Once you get in, let's say, a new version of software package then you should do some basic testing on the test problems that are important in your environment. These could be code verification test problems which means on code verification you have an analytical solution and an exact solution to the mathematical model; the mathematical model is the calculus problem. So that's what we're talking about when doing code verification testing.

Solution verification testing is an entirely different activity. In solution verification it is error estimation for a given mathematical model and then we solve that discretely with finite elements, finite difference, finite volume, whatever we have to do. This means that the solution verification is done on the individual problems that we are solving for, and usually it's called the application of interest or the system of interest.

What is solution verification?

GW: So, about solution verification, can you explain a little bit more? Is that about getting things like mesh independence, discretization type error, and evaluations of those kinds of error or am I off base there?

WO: No that's exactly right. In solution verification you have a couple of elements where errors might come from. The most common ones that we're talking about here are those due to discretization from finite element, finite volume, whatever the method is. These are discrete methods in the sense that they discretize the original differential equation and then we iteratively solve those equations. Those equations are all arithmetic equations, they're not calculus problems, so they approximate the calculus problem and as is well known in numerical methods, these numerical methods have a truncation error. It's a chemical and discretization error but it's usually called a truncation error because it is an approximate solution and as the mesh is refined- whatever discrete method you're using- the solution should converge to the correct solution. The correct solution as in the same solution to the calculus problem. That’s one of the tests that you should always do.

One of the difficulties is, in relatively large-scale problems these finer and finer meshes get very expensive and unaffordable. This is especially true in three dimensional problems and of course it gets even worse in three dimensional unsteady problems like deformation of structures i.e., in in a car crash or some other dynamic environment. Those are some of the practical difficulties of trying to do solution verification.

How do we decide what a reasonable trade-off between these activities is?

GW: So what resources are available that help us decide what a reasonable trade-off between these activities is?

WO: Let me respond to the code verification part first. Code verification is entirely a testing process of the software and the numerical methods. And I say testing as opposed to an estimation problem because it is not an estimation problem. It's a testing problem of the software and the algorithm because you must have an exact solution to the mathematical problem. The common procedure is what you mentioned earlier where you test when there is a new version of the software but it should also be at the stage when you're considering a new software package. You should talk to the software vendor and ask them what their code verification data step is, ask them to show you all of the code verification problems that you have tested and then you can see how they relate to your problems of interest. If you have some exact solutions that you want to test it on you could then test the software on a trial basis.

With regards to solution verification that is typically a much more involved endeavour because you have numerical approximations, there's discretization errors there is also iterative errors which are separate from discretization. Then, if you're doing a non-deterministic simulation then you also have sampling errors. Additionally, in the solution verification if an analyst makes an error in the input data for the model that error is actually called part of the solution verification error. So that kind of thing also should be considered because it happens relatively often but that of course is a user error and the best way to check that is to check your own work and have others users also checking. So, as you can see there are a number of sources of errors that go into solution or calculation verification that people need to be aware of.

So how can a beginner figure all of this out?

GW: Are there specific resources, either standards that govern what needs to happen for various industries or applications, or publications of best practice that would help the beginner figure these steps out?

WO: That's a good practical question. One of the things that you should find out from your software vendor is the sort of error estimation tools that they have with their software package and when we say error estimation, we're talking about solution verification.

They may have tools where the emphasis is on discretization error or perhaps they have estimators on iterative errors like on a boundary value problem or an elliptic problem, or let's say a stress problem or fluid mechanics problem in all steady state. Since that's a non-linear set of discrete equations you have to iteratively solve them. And then that leaves a separate error due to discretization. So, you should ask your software vendors what tools they have to estimate the error in the system response quantities that you're interested in for your particular solution. Those are tools that you should reasonably expect from the software vendor.

Some vendors do better than others, for example in solid mechanics essentially every vendor has an error estimator based on the global error, usually referred to as the global energy norm. This is an error which in terms of strain energy, covers the entire solution domain. While that is an excellent first step, you have to realize that norm only tells you what the estimated error is over the entire domain not what the error is locally.

GW: Great. So, we have there what I think is some helpful guidance for software tools or what to expect specifically from software providers. How about NAFEMS publications or regulatory standards, is there specific guidance provided there or is this still largely up to us to define what is best practice?

WO:I think there're some weaknesses in this area. There are textbooks on this topic but there are not many publications that actually deal with it in any kind of detail when it comes to questions like what kind of numerical error estimators/ solution verification estimators might be useful. There are publications in various journals and conference papers but, in my opinion there’s not a lot. While there is some guidance that ASME has on the various aspects like code verification, solution verification validation, and so on— and NAFEMS also have some in that regard— It’s not a topic that has been, I'd say, widely explained in terms of what tools might be available, so I think some improvements are needed there.

What one thing related to verification should everyone reading do now?

GW: If you could convince everybody to do one thing related to verification, what would that be?

WO: Well Greg I’m going to cheat a little bit and give you two things! The first one is on code verification and that is; always check what the software vendor or the software developer has on the test problems they have i.e., analytical solutions or manufactured solutions, those are excellent. Check what their code verification test problems are and when I say check, I mean see if they demonstrate convergence with mesh discretization and also iterative as the mesh is continually resolved. If you only do one thing, that's the one thing you should do on the code verification side.

The second thing is on the solution verification side. You should ask your software provider what tools are available and you should use those tools if they have them. If your software provider does not have, let’s say, a local error estimator i.e., an estimate of the local stress or if it's a fluid mechanics problem, the estimated error in local pressure, temperature, shear stress or other local quantities like local heat flux then your organization should start thinking about how to build those to add on, if you can't get your software provider to build them. This is because we have found in years past that when even experienced analysts are doing solution verification, they think they have converged solutions and then when they get a bigger computer and they do more solution convergence in the sense of finer meshes they realize that the solution changes again and again. That’s something that can really burn you because you think you have the solution to the mathematical problem but you really don't.