Swarm of Functions Analysis Tools

sofa, tracing, performance, profile, swarm, visualization
pip install sofalite==0.6.3



SOFA: Swarm of Functions Analysis Authors: All the contributors of SOFA


  1. Run ./tools/ to install all the necessary packages and python packages.
  2. [OPTIONAL] Run ./tools/ $(whoami) to make network related events tracable in SOFA. After running this step, it is required to re-login to APPLY THE CHANGES!!!

Verify Permissions

Run the following simple tests to verify the permission settings of executing tcpdump under your permission.

  • tcpdump -w sofa.pcap
  • tcpdump -r sofa.pcap


  1. Simply run ./ </PATH/TO/INSTALL> to install SOFA on your system. Note that sofa will be appended to the path if the last directory is not sofa.
  2. Then, run source </PATH/TO/INSTALL>/sofa/tools/ to activate SOFA running environment. (Need to be executed on each new shell.)
  3. [ALTERNATIVE] Add source </PATH/TO/INSTALL>/sofa/tools/ in your ~/.bashrc to make this environment available on every shell.


Run bash </PATH/TO/INSTALL>/sofa/tools/ to safely remove/uninstall files of SOFA.


SOFA supports serveral different usages, like how one can use perf.

Basic Statistics

  1. Profile your program by sampling involved CPUs:
    sofa stat "wget"
  2. Profile your program by sampling all CPUs:
    sofa stat "wget" --profile_all_cpus

Performance Visualizations

  1. sofa record "wget"
  2. sofa report
  3. Open browser with one of the following links for different visualizations

Run with different plugins

  1. Run sofa record "wget" only once to record the events.
  2. Run sofa preprocess only once to process the events.
  3. Run sofa analyze with A plugin.
  4. Run sofa analyze with B plugin.
  5. Run sofa analyze with C plugin.


SOFA provides options for configurations. Some examples are shown below. Please use sofa --help to see more info.

  1. sofa --cpu_filters="idle:black,tensorflow:orange" record "python"
  2. sofa --gpu_filters="tensorflow:orange" --verbose=True record "python"

Examples of Visualization Results:

sofa record "~/cuda_samples/1_Utilities/bandwidthTest/bandwidthTest" Alt text sofa record "python --num_gpus=8 --batch_size=64 --model=resnet50 --variable_update=parameter_server --num_warmup_batches=1 --num_batches=3" Alt text sofa record "python --num_gpus=8 --batch_size=64 --model=resnet50 --variable_update=parameter_server --num_warmup_batches=1 --num_batches=3" Alt text sofa record "python --num_gpus=8 --batch_size=64 --model=alexnet --variable_update=replicated --all_reduce_spec=xring --num_warmup_batches=1 --num_batches=3" Alt text sofa record "mpirun -f hosts.txt -n 4 ./compute_pi" Alt text