statprof - statistical profiling for Python
This package provides a simple statistical profiler for Python.
Python's default profiler has been
lsprof for several years. This is
an instrumenting profiler, which means that it saves data on every
action of interest. In the case of lsprof, it runs at function entry
and exit. This has problems: it can be expensive due to frequent
sampling, and it is blind to hot spots within a function.
statprof samples the call stack periodically (by
default, 1000 times per second), and it correctly tracks line numbers
inside a function. This means that if you have a 50-line function
that contains two hot loops, statprof is likely to report them both
This package does not yet work on Windows! See the implementation and portability notes below for details.
How to get it
pip install statprof-smarkets
Warning: it uses
statprof as Python module name so this will conflict with
original statprof installation if present.
GitHub project page: https://github.com/smarkets/statprof
It's easy to get started with
import statprof statprof.start() try: my_questionable_function() finally: statprof.stop() statprof.display()
Or with a contextmanager:
import statprof with statprof.profile(): my_questionable_function()
The profiler can be invoked at more than one place inside your code and will report its findings for all of them at once at the end:
import statprof statprof.start() try: my_questionable_function() finally: statprof.stop() uninteresting_code() statprof.start() try: my_other_questionable_function() finally: statprof.stop() statprof.display()
However, when you are profiling your code by repeatedly executing it in IPython, each run will add new samples to the previously collected ones, and display results aggregated over all runs. Since you will likely just want to see the results for the last run, remember to reset the profiler first:
import statprof statprof.reset() with statprof.profile(): my_questionable_function()
Or from command line:
$ python -m statprof script.py # or $ python -m statprof -m script # or (this may depend on bash because http://www.gnu.org/software/bash/manual/bashref.html#ANSI_002dC-Quoting) $ python -m statprof -c "import hashlib"$'\n'"for i in range(10000): hashlib.md5(str(i)).hexdigest()"
For more comprehensive help, run
Because statprof uses the Unix
itimer signal facility, it does not
currently work on Windows. (Patches to improve portability would be
statprof profiler works by setting the Unix profiling signal
ITIMER_PROF to go off after the interval you define in the call to
reset(). When the signal fires, a sampling routine is run which
looks at the current procedure that's executing, and then crawls up
the stack, and for each frame encountered, increments that frame's
code object's sample count. Note that if a procedure is encountered
multiple times on a given stack, it is only counted once. After the
sampling is complete, the profiler resets profiling timer to fire
again after the appropriate interval.
Meanwhile, the profiler keeps track, via
os.times(), how much CPU
time (system and user -- which is also what
ITIMER_PROF tracks), has
elapsed while code has been executing within a
The profiler also tries (as much as possible) to avoid counting or timing its own code.
- Added support for sorting by cumulative time to the display method
- added configurable display format (displays full paths by default now)
- ability to run whole scripts under statprof from command line (thanks to Vincent Driessen and Antony Lee
- added support for
python -mstatprof -c cmdinvocation (thanks to Antony Lee)
This package was originally written and released by Andy Wingo. It was ported to modern Python by Alex Frazer, and posted to GitHub by Jeff Muizelaar. Maintained by Bryan O'Sullivan, was forked by Smarkets due to package not being maintaned anymore.
Reporting bugs, contributing patches
Please report bugs using the GitHub issue tracker.
If you'd like to contribute patches, please do - the source is on GitHub, so please just issue a pull request.
$ git clone git://github.com/smarkets/statprof