Verification - How Much Should I Do?

A Discussion with the NAFEMS SGMWG

4-minute read

In the next part of this series, the NAFEMS Simulation Governance & Management Working Group discussed how much verification you need to do, and when to do it. As well as reading the conversation here, you can watch the video on YouTube.

The discussion features William Oberkampf (WO) & Gregory Westwater (GW), members of the NAFEMS Simulation Governance & Management Working Group

Is there an answer to the question “how much verification should we do”?

GW: When we've had auditors or customers come in, some of them have wildly different opinions on what verification is. Some of them feel that the verification activities described in our previous episode are more than adequate, while others basically want each machine verified almost every single day, just because of who- knows- what patches ran overnight, and things like that. And that's one of those situations that has had me looking for a standard and wondering, is there a definition of what is ‘good enough’ that can help answer those kinds of questions when differing opinions are brought up?

WO: The whole field of how much V&V to do can be very overwhelming which can lead to people and organizations saying, "This is just so overwhelming, I have no idea where to start, and so we'll come back and look at this at a later time," it can be like that.

There are two parts to the question you asked. One of them is: what are the most important things to do? And that's exactly what management and staff members, especially senior staff, should be asking: “Of all the myriad things there are to do, should we do code verification checks every time the operating system has a patch on it?” The answer is, that's excessive. On the other hand, if you are wondering if you should do code verification checks when you have a new version of the software package then the answer is yes, that you should do. Of course, some patches are much smaller than others, but if there's a significant upgrade to the package, then you should do those checks.

The other part of your question has to do with when either customers, regulators, or safety authorities, come in and say "What are you doing in V&V?". There is a lot of variation in terminology when it comes to answering the question ‘what is V&V?’. The systems definition of Verification & Validation is very different from what we're talking about here. So, when you deal with customers be sure that you're talking about the same Verification & Validation. This is because the systems V&V has to do with meeting requirements and ensuring that each phase correctly checks with the previous phase. Whereas what we're talking about here, which is V&V concepts, is entirely focused on solving mathematical models, usually given by differential equations. So, you have to get everybody on the same page.

And we're at a point in history where some regulators and customers are just beginning to ask questions like "What do you do in code verification and solution verification?", but we're still at an early stage on that. But as we progress forward and companies use more and more simulation in their products, particularly if they want to certify, let's say, safety or performance, then the customers or the regulators should be asking exactly, "Show me what V&V you've done." And so, I think that farsighted management will start saying; "We need to start getting more organized in this activity."

What are manufactured solutions?

GW: Here’s another quick question. You’ve previously referred to ‘manufactured solutions’, can you give a two-minute introduction on what those are?

WO: They are analytical solutions, but they're very special types. So, when we say in code verification that we must have an analytical solution some people say, "Well, we could use a very accurate numerical solution." You can, but the requirements that we want on accuracy are so demanding that they're very, very hard to find. So, the whole class of analytical solutions, the most common ones being traditional analytical solutions, are given by, typically, infinite series, and we did those mainly back in graduate school on solving differential equations.

Manufactured solutions are a special class and the reason they're so powerful is that they change the mathematical model of the physics. They move it out of the realm of a real physics problem, or approximate physics problem, into a realm where it is no longer actual physics. It's not physics that approximates any kind of reality. You might say, "Why would you want to do that?". The reason is that you can generate analytical, exact solutions to that problem more easily than you can the original physics problem. That may seem a little bit confusing, but you actually can do it. It is much easier to do it if you are changing the mathematical model, it's called changing the right-hand side of the differential equation and sometimes it's called the exultation side. When you do that, you have much more flexibility, and that's why they are such excellent problems to test in code verification.

*This conversation transcript has been edited and condensed for readability.