vacc: A python helper package running scripts on the VACC
A package to faciliate running python scrips on the UVM's VACC..
The package has two main modules, rsync
and submitter
. The former is meant to accelerate the process of transfering files to and from the vacc. The latter is for use on the vacc to submit vacc jobs from a python script.
Installation
To install, run pip install vacc
.
Rsync
Suppose your project directory has a subdirectory called data
which you want to sync with the vacc using rsync. This can be done with the following code:
from vacc import vacc_push
source_path = 'data' # the relative location of the folder
destination_path = 'my_project/data' #the relative location of the same folder on the vacc
vacc_push(source_path, destination_path, netid = <uvm_id>)
Naturally, the reverse process of pulling files from the vacc to your local computer is done with
from vacc import vacc_pull
source_path = 'data' # the relative location of the folder on the vacc
destination_path = 'my_project/data' #the relative location of the same folder on your local computer
vacc_pull(source_path, destination_path, netid = <uvm_id>)
If your password is not automatically configured, you will be prompted to enter your password.
For both of these functions, arbitrary rsync arguments can be passed along, such as
exclude_arg = "--exclude={'data_05-**-20','data_05-**-21'}"
vacc_push(source_path,destination_path,netid=<netid>,rsync_commands = [exclude_arg])
Finally, we can specify a dry run of the procedure before syncing the folder for real. This can be accomplished with
vacc_push(source_path,destination_path,net_id=<netid>,dry_run=True)
Submitter
To submit jobs on the vacc, we can use the following code:
from vacc import vacc_submit
run_dict = {
'module1.submodule3.func1':dict(arg1=3,arg2=False),
'module2.myfunction':dict()
}
vacc_submit(run_dict)
The above snippet is equivalent to running the following script:
from module1.submodule3 import func1
func1(arg1=3,arg2=False)
from module2 import myfunction
myfunction()
The vacc_submit
function can also take arbitrary sbatch arguments for configuring resource requirements. As an example,
vacc_submit(
run_dict,
partition='short',
mem='2gb',
time='6:00:00
)
All sbatch configuration commands can be found here.