PyProfQueue serves as a python package that can take in existing bash scripts, add prometheus monitoring calls, likwid performance measure calls, linaro forge map calls, and submit the script to an HPC queue system on the users' behalf.

PyProfQueue takes existing bash scripts, scrapes any queue options from them and creates two temporary files. The first file, is equivalent to the user provided bash script but with queue options removed from it. The second bash script is the file that will be submitted to the queue on the users' behalf. This second temporary script contains the queue options provided by the user as well as any additionally required commands to initialise prometheus monitoring, linaro forge map profiling and/or likwid performance measuring of the users bash script.

The user facing components of PyProfQueue are the Script class and submit function.

script = PyProfQueue.Script(queue_system: str,
                            work_script: str,
                            read_queue_system: str =None,
                            queue_options: dict = None,
                            profiling: dict = None
                            )   

change_options(queue_options: dict)

script.change_options(queue_options={'time':'24:00:00'})

The inputs to the Script class and submit function have already been described above, what has not been described in detail in the input file that users provide. Users must provide the bash script that they wish to profile and submit to a queue system. If a bash script with no queue options is provided, then the read_queue_system option can be left off, or set to None. In this case, unless both read_queue_system and queue_system are set to None, queue_options have to be provided. If queue options are available in the script then read_queue_system should be set to the name of the queue system that the original script was intended for. In order to use profiling tools, a dictionary has to be provided to the profiling option of the Script class. The keys of the dictionaries are the name of the profiling tools to be used, and the values of the dictionary is a dictionary of required and optional arguments for each profiler. As long as there is no clashes between profiling tools, multiple can be used at once. We list examples of inputs below.

profiling = {"likwid": {"requirements":["module load likwid"]}}

profiling = {"prometheus": {"requirements":["export PROMETHEUS_SOFTWARE=<path/to/prometheus>"]}}
# OR
profiling = {"prometheus": {"ip_address":["127.0.0.1:9090"]}}

profiling = {"linaro_forge": {'code_line': ['echo "Hello World"']}}

The output of the Script class, is an object that contains all the given options, file paths and other variables needed in order to create the bash scripts that can be submitted to a queue system. The outputs from submit depend on the given options. If the test and leave_scripts are set to False, then the output of submit is the same as the output of the submission command for the respective queue system being used. If test or leave_scripts are set to True then the output includes two bash scripts. The first contains the original work to be performed, the other contains all the necessary variable declarations and command calls in order to perform the profiling of the given bash script. It is the profiling script that is submitted to the queue. Additionally, if test is True, then the command line will output what command would be used in order to submit the job, but the command will not actually be called.

Where profilers are set up to return plots, the outputs are .png files. While the plots are autogenerated in most cases, it is possible to replot them in post using the functions found within the respective python scripts for a profiler.

#!/bin/bash
#SBATCH -A example_project
#SBATCH -c 16
#SBATCH -N 1
#SBATCH -o /home/queue_work/%x.%j/output.out
#SBATCH -p example_partition
#SBATCH -J TestSubmission
#SBATCH -n 1
#SBATCH -t 00:05:00

echo "The first option was:"
echo ${1}
echo "The second option was:"
echo ${2}

import PyProfQueue as ppq

ProfileScript = ppq.Script(queue_system='slurm',
                           work_script='./tmp_workfile.sh',
                           queue_options={
                             'workdir': '/home/queue_work/%x.%j',
                             'job_name': 'NewName'},
                           profiling={
                             "likwid": {'requirements': ['module load oneAPI_comp/2021.1.0',
                                                         'module load likwid/5.2.0']},
                             "prometheus": {'requirements': ['export PROMETHEUS_SOFTWARE=/home/Software']}
                           }
                           )

ppq.submit(ProfileScript, 
           tmp_work_script = './test_workfile.sh',
           tmp_profile_script = './test_profilefile.sh',
           bash_options=['"Hello "', '"World!"'],
           test=True)

The following command would be used to submit a job to the queue:
sbatch ./test_profilefile.sh

#!/bin/bash

# Any work that users may want to do on an HPC system, including environment initialisations
# For the sake of example we simply call
echo "The first option was:"
echo ${1}
echo "The second option was:"
echo ${2}

likwid-perfctr -g MEM_DP -t 300s -o ${LIKWID_RUNNING_DIR}/likwid_output.txt -O -f bash ./test_workfile.sh  "Hello " "World!"

${PROMETHEUS_SOFTWARE}
├── node_exporter
│   └── node_exporter
└── prometheus
    ├── prometheus
    └── prometheus.yml

likwid-perfctr -g MEM_DP -t 300s <output directory> <executable> <options for executable>

map --profile --no-queue -o <output file path> <user command>

As of now, no minimum Hardware requirements are known other than those forced by python 3.10, prometheus, node_exporter and likwid.

PyProfQueue
├── PyProfQueue
│   ├── batch_systems
│   │   ├── pbs.py
│   │   ├── slurm.py
│   │   └── _template_batch.txt
│   ├── profilers
│   │   ├── data
│   │   │   ├── read_prometheus.py
│   │   │   ├── likwid_commands.txt
│   │   │   ├── linaro_forge_commands.txt
│   │   │   ├── prometheus_commands.txt
│   │   │   └── _template_commands.txt
│   │   ├── likwid.py
│   │   ├── linaro_forge.py
│   │   ├── prometheus.py
│   │   └── _template_profiler.txt
│   ├── __init__.py
│   ├── plot.py
│   ├── script.py
│   └── submission.py
├── ReadMe.md
└── setup.py

The directory PyProfQueue/PyProfQueue contains the script.py and submission.py scripts which house the definition of the Script class and submission() function respectively.

The directory PyProfQueue/PyProfQueue/batch_systems contains the python files which house the dictionaries for each batch system that PyProfQueue is compatible with. PyProfQueue/PyProfQueue/profilers contains scripts of the individual profilers that PyProfQueue is compatible with including a template version that can be used to add additional profiling software compatibility to PyProfQueue.

PyProfQueue/PyProfQueue/profilers/data contains a script called read_prometheus.py which is used to scrape the prometheus database into a pandas dataframe. It also includes the text files that list the bash commands needed to initialise run and end profiling software, as well as a template version for adding more profiling software compatibility.

The base directory contains the ReadMe.md file, and the setup.py file so that the package can be installed.

In order to new batch system compatibility, a new .py file has to be created that follows the PyProfQueue/PyProfQueue/batch_systems/_template_batch.txt format. If this is added correctly, then any options that have overlap to pre-existing batch systems files will automatically be able to translate between each other.

In order to add new profiling software compatibility, a new script would need to be added to the PyProfQueue/PyProfQueue/profilers directory. This script would need to have the name of how the specific profiler should be called, as well as the following two functions:

• define_initialise(profilefile: io.TextIOWrapper, profilerdict: dict = None)
• define_end(profilefile: io.TextIOWrapper)

Both of these functions can read from a .txt file, which can be stored in PyProfQueue/PyProfQueue/profilers/data. The first define_initialise is there to add any variable declarations or code to the profiling bash script that would be needed in order to run the profiler. The second is any calls needed in order to terminate, or collect data post running the profiler.

If the profiler being added needs to be used in order to execute the user provided bash script, then the function define_run(profilefile: io.TextIOWrapper, bash_options: list, tmp_work_script: str) should be defined in the script for the specific profiler. This function should write the needed command to use the profiler to the profiling file.

• Update the package format to follow more modern conventions for installing requirements
  • e.g. add pyproject.toml
• Add Vtune profiling support

This package was initially created because of the Jira Ticket TEAL-391.

A Confluence page of what the prometheus and likwid results look like can be found on the SKAO Confluence

Developers:

• Keil, Marcus (UCL)
• Qaiser, Fawada (Durham)

Contributors:

• Morabito, Leah (Durham)
• Yates, Jeremy (UCL)