Description
As these systems become larger and more complex, managing efficient utilization of resources becomes increasingly challenging.
While de-facto job schedulers such as Slurm and Flux improve resource utilization at various levels, for example by enabling job concurrency, the traditional static allocation of computational resources at job start compels users to provision for peak demand across all phases for the application, contributing to resource under-utilization.
At the same time, user workloads are increasingly dynamic, consisting of multiple distinct execution phases, each demanding different resource configurations.
Such workloads are no longer rigid, with a fixed resource allocation request, but malleable, facilitated by dynamic resource allocation.
In principle, dynamic resource allocation can boost both per-job utilization and cluster throughput; however, the gains depend on the workload and on the scheduler allocation scheduling latencies, and realizing them requires further tuning of resource change requests, e.g., which phases to resize, by how much, and when, so that each change takes effect in time to cover resource acquisition, synchronization, and data redistribution. In current practice, users rarely have this knowledge a priori.
We present a phase-aware diagnostic tool on top of FlexiAlloc, a dynamic resource management framework, to support fine-grained scheduling for better utilization via successive feedback loops.
It fuses the scheduling records with resource utilization telemetry, and correlates a job's allocation requests with observed usage (e.g., CPU, memory) on a shared timeline.
With that, the tool enables users to analyze whether their phase-targeted resource change requests were effective.
By including the tool in the scheduling workflow, it enables automatic incremental resource scaling experiments for building an evolving performance model for phase-aware allocation in later runs, ultimately assisting the scheduler to insert requests automatically via agents