The Evidence Folder: How US Regulators Read a Simulation
Photo: engineering compliance documentation review meeting regulatory audit, via www.discoverengineering.org
Certification by analysis has a reputation problem in US engineering practice, and the reputation is mostly a documentation problem wearing a technical costume. Regulators do not reject simulation because the physics is wrong. They reject submissions because the model's role in the decision was never defined, because verification and validation were reported as a single undifferentiated activity, and because no document in the package lets a reviewer retrace a number from conclusion back to assumption.
The frameworks that govern this work are more similar than their domains suggest. ASME V&V 40 for medical devices, ASME V&V 10 for solid mechanics, ASME V&V 20 for computational fluid dynamics and heat transfer, NASA-STD-7009 for agency models and simulations, and the NRC's RG 1.203 for thermal-hydraulic code evaluation all converge on the same sequence: define the question, establish credibility proportionate to risk, generate evidence, and preserve the trail.
Context of use is the first document, not the last
Every credible submission starts with a sentence that is harder to write than it looks: this model will be used to make this specific decision, in this range of conditions, in place of or alongside this specific test. That statement is the context of use, and it determines everything downstream — which physics must be validated, how tight the uncertainty needs to be, and which evidence is admissible.
A model validated for static stiffness prediction is not thereby validated for fatigue initiation. A thermal model validated at steady state does not carry its credibility into a transient. Auditors are trained to look for exactly this drift, and the failure is almost never dishonest; it is a context of use that was written informally, then quietly expanded as the program's questions changed.
Verification and validation are separate line items
Verification asks whether the equations were solved correctly. Validation asks whether the right equations were solved. Submissions that blur the two invite the worst kind of review meeting, in which the team defends a mesh convergence study against a question about constitutive behavior.
Code verification is demonstrated through order-of-accuracy tests and manufactured solutions. Solution verification is demonstrated through grid or time-step convergence with a reported discretization uncertainty — the grid convergence index remains the common currency. These activities do not touch the question of physical correctness, and they should be documented as a distinct section, with its own evidence, before validation begins.
The validation hierarchy earns the top of the pyramid
Credibility accumulates from the bottom up: unit problems, then component tests, then subsystem, then full system. Each level isolates fewer phenomena and more interactions. A submission that jumps straight to a full-system comparison against a single test article is asking the reviewer to accept a single number as proof of a chain of assumptions.
In practice, the strongest packages show a ladder of comparisons with the increment of physics added at each rung, and they identify which rung the decision actually depends on. Where a rung is missing, the package says so and explains why the risk is tolerable — which is a far stronger position than a missing rung the reviewer discovers independently.
Uncertainty is a budget, not a disclaimer
Saying results are approximate is not uncertainty quantification. A usable budget separates parameter uncertainty (material properties, boundary conditions, loading), model form uncertainty (constitutive choices, turbulence treatment), and numerical uncertainty (discretization, iteration). Each line carries a magnitude and a basis: test data, literature range, or expert elicitation with recorded rationale.
The budget does more than satisfy a reviewer. It tells the design team where the next dollar of testing buys the most credibility — frequently in boundary conditions rather than in the solver, which is not where most simulation investment goes.
Traceability is the question the auditor actually asks
Given a conclusion in the report, a reviewer should be able to move to the case that produced it, to the model revision and input deck that define the case, to the verification evidence for that model class, and to the assumptions register that governs it. Four hops, all documented, all versioned.
Most failed submissions break at the second hop: the reported result does not correspond to a uniquely identified model revision, because the analysis was rerun after a geometry change and the report was not regenerated. The remedy is procedural rather than technical — every reported number carries a case identifier, and the identifier resolves to an immutable record of inputs, code version, and solver settings.
What the frameworks share
ASME V&V 40 organizes credibility around model influence and decision consequence, which together set a credibility goal; the submission then argues that the collected evidence meets it. NASA-STD-7009 decomposes credibility into graded factors including verification, validation, input pedigree, uncertainty characterization, robustness, use history, and the qualifications of the people involved — the last of which surprises engineers and is routinely under-documented. The NRC's RG 1.203 begins with a phenomena identification and ranking table that determines where validation effort must concentrate.
Read together, they describe a single discipline: rank the physics by importance, target evidence at the ranking, quantify what remains uncertain, and record who decided.
Build the folder before the auditor asks
Design reviews are the right venue for the first draft of the evidence folder, not the certification milestone. The practical sequence is unglamorous: write the context of use as a one-page controlled document; map each decision to the model and case that supports it; grade each piece of evidence against the applicable framework's factors; maintain an assumptions register with owners and dates; and regenerate the report from the case records rather than from memory.
None of that requires a new license or a faster cluster. It requires the recognition that a simulation program's output is not a contour plot. It is a defensible argument, and arguments are made of documents.
