heapprof: A Logging Heap Profiler
heapprof is a logging, sampling heap profiler for Python 3.7+.
- "Logging" means that as the program runs, it steadily generates a log of memory allocation and release events. This means that you can easily look at memory usage as a function of time.
- "Sampling" means that it can record only a statistically random sample of memory events. This improves performance dramatically while writing logs, and (with the right parameters) sacrifices almost no accuracy.
It comes with a suite of visualization and analysis tools (including time plots, flame graphs, and flow graphs), as well as an API for doing your own analyses of the results.
heapprof is complementary to tracemalloc, which is a snapshotting heap profiler. The difference is that tracemalloc keeps track of live memory internally, and only writes snapshots when its snapshot() function is called; this means it has slightly lower overhead, but you have to know the moments at which you'll want a snapshot before the program starts. This makes it particularly useful for finding leaks (from the snapshot at program exit), but not as good for understanding events like memory spikes.
You can install heapprof with
pip install heapprof. heapprof is released under the
You can read all the documentation at humu.github.io/heapprof.
Navigating the Repository
If you're trying to find something in the GitHub repository, here's a brief directory (since, like most Python packages, this is a maze of twisty subdirectories, all different):
heapprofcontains the Python package itself. (The API and visualization logic)
_heapprofcontains the C/C++ package. (The core profiling logic)
docs_srccontains the sources for the documentation, mostly as
docscontains the compiled HTML version of
docs_src, created with
tools/docs.pyand checked in.
toolscontains tools useful when modifying heapprof itself.
- And then there are the configuration files for all the tools:
setup.pyis the master build configuration for the PIP package.
.pylintrcare the configuration for Python linting.
CPPLINT.cfgis the configuration for C/C++ linting.
mypy.iniis the configuration for Python type checking.
Gemfileis for setting up Jekyll for documentation hosting.
_config.ymlis the configuration for Jekyll serving.
docs/conf.pyare the configuration for building the HTML docs image via Sphinx.
.circleciis the configuration for continuous integration testing.
pyproject.tomland the root
- Additional directories which are .gitignored but which show up during use:
buildcontains C/C++ dependencies and their compiled images; it's managed by
_sitecontains the final Jekyll site which is served for documentation; it's created if you run
bundle exec jekyll serveto run the docs web server locally.