Description
Performance optimization of scientific and high-performance computing (HPC) applications frequently introduces subtle behavioral changes that go unnoticed until they manifest as incorrect scientific results. Detecting whether such changes alter computed state, and precisely where in the code and iteration space a divergence first occurs, remains a largely manual and time-consuming process.
Existing tools address related but distinct problems: numerical debuggers (e.g., FPChecker) quantify rounding-error propagation within a single run rather than comparing two executions against each other; record-and-replay debuggers (e.g., rr) provide deterministic replay but no notion of application-level checkpoints or domain-specific state; interactive differential debuggers such as IDD run a baseline and a regressed executable side by side under a pair of synchronized gdb/lldb sessions for the full comparison session, incurring continuous debugger-control overhead, and rely on the developer to manually drive both sessions and issue comparison commands rather than on automated, source-level state checkpoints; and offline diffing tools (e.g., h5diff) compare dumped output after the fact, without runtime or debugger integration.
In this work, we present a lightweight framework for checkpoint-based state comparison in scientific applications. Developers annotate variables of interest directly in source code with domain-meaningful, multi-dimensional context (e.g., time step, sub-iteration, MPI rank) through an explicit API. A reference run records these states; subsequent, independent runs execute at native speed and compare against the reference and, on the first detected mismatch, provide location information or halt directly into an attached debugger at the exact point of divergence - without requiring the two executions to run simultaneously, continuous dual debugger sessions, or interactive command-driven comparison.
Different methods for comparison are provided based on the type of data as well as the acceptable memory overhead, ranging from low-overhead hash-based testing up to full data set comparisons using configurable floating-point tolerances. We describe the design and implementation of our framework and demonstrate its use in localizing correctness regressions.