Description
CODA is a flow solver jointly developed and used by DLR, Airbus and ONERA that must be validated continuously on very
different HPC systems. It is embedded into the FlowSimulator ecosystem, which serves as a backbone for multidisciplinary
analyses and optimizations on HPC architectures. Verifying full CFD simulations manually is tedious, slow and
error-prone, and numerical or performance regressions can easily slip through.
We present coda-e2e-testing, an end-to-end testing framework built on ReFrame for automated test execution. It can run
on HPC clusters and workstations alike, and is employed in a centralized interface for triggering tests, publishing
reports online, and archiving reports and results for future reference. The workflow is sought to minimize user
intervention and ease maintainability.
The implementation involves a thin Python layer with its own CLI around ReFrame, which provides the front end and the
scheduler abstraction (Slurm, LSF, local workstations), keeping the tests scheduler-agnostic. A single test base class
forms the backbone of all tests, and the definition of test cases and their metadata requires no knowledge of the test
framework, which keeps adding new tests and maintaining existing ones simple. Test executions are triggered from GitLab
CI via a pipeline input mask. Results are published as an interactive HTML report with interactive plots generated with
Quarto. The test suite runs unchanged on the DLR (cara/caro), Airbus (hpc5/hpc6) and ONERA (juno/sator) clusters,
configured through ReFrame system, partition and environment definitions, and supports module-based, local-development
and Apptainer-container deployments.
The test portfolio spans the full spectrum from quick simulations of simple cases to full-fledged airplanes. Sanity
checks cover key physical parameters as well as numerical ones (residuals, runtime, memory), and more detailed
comparisons are reported against a reference CODA version and external references, including experimental data and other
CFD solvers.
ReFrame was designed to compile and test code across systems and compilers; we show that it also works well as the
backbone of simulation-based end-to-end verification, with the implementation effort shifted into a reusable base test
class instead of bespoke tooling.