Description
The increasing cost of new HPC systems and a growing demand for computing resources require existing systems to be operated for longer periods and at higher utilization levels. Consequently, efficient use of available resources is becoming increasingly important. While system-side optimizations such as improved scheduling and reduced downtime contribute to higher throughput, continuous analysis and optimization of
application performance remain essential.
In this work, we present an approach that transforms continuous performance monitoring from a user-initiated activity into a transparent infrastructure service of the batch system. We integrate the lightweight Linux performance analysis tool lo2s into the Slurm Workload Manager using a SPANK plugin. This integration enables transparent monitoring
throughout the job lifecycle without requiring modifications to existing user workflows.
Due to its low overhead and support for a broad range of hardware and software metrics, lo2s is well suited for continuous monitoring in production environments. Its process-centric and system-monitoring modes enable both detailed analysis of individual applications and system-wide observation of resource utilization.
A prerequisite for this is the continuous collection of performance characteristics of production applications in the context of their system environment. However, existing performance analysis tools are typically explicitly launched by users, may have high overhead and are therefore not or only partially suited for automated, system-wide performance monitoring.
We describe the architecture and implementation of the plugin, discuss security aspects and limitations, show achievable low overhead, and demonstrate how continuous performance monitoring can support both system operators and users in understanding and optimizing HPC workloads.