Description
In the rise of machine learning, PyTorch has become a commonly used library to abstract high performance and accelerated compute kernels as accessible Python functions. Workloads using PyTorch, e.g. for LLM training or inference, are becoming prevalent consumers of HPC resources, prompting a closer look into their exact behaviours and computational efficiencies. Tracing and profiling provide the methods to deeply inspect program behaviour in any execution environment. Consequent analysis of inspection results can highlight shortcomings in code efficiency or resource utilization. The Score-P framework provides tools to capture a system-centric perspective for traditional HPC applications, and exports results in the form of OTF-2 files. Further analysis can then be conducted with programs such as Vampir or Cube. Since PyTorch exposes tracing and profiling APIs via native functions or libraries like Kineto, programmatic access to the interfaces permits us to integrate their application-centric view with Score-P. In our work, we present and demonstrate this integration by applying Score-P-based PyTorch profiling/tracing to example codes, and how the combination of views helps correlate effects seen in either the system- or the application-centric context.