Powerful Benchmarker

Documentation

View the documentation here

Google Colab Examples

See the examples folder for notebooks that show a bit of this library's functionality.

A Metric Learning Reality Check

See supplementary material for the ECCV 2020 paper.

Benchmark results:

• 4-fold cross validation, test on 2nd-half of classes

Benefits of this library

1. Highly configurable:
   • Yaml files for organized configuration
   • A powerful command line syntax that allows you to merge, override, swap, apply, and delete config options.
2. Customizable:
   • Benchmark your own losses, miners, datasets etc. with a simple function call.
3. Easy hyperparameter optimization:
   • Append the ~BAYESIAN~ flag to the names of hyperparameters you want to optimize.
4. Extensive logging:
   • View experiment data in tensorboard, CSV and SQLite format.
5. Reproducible:
   • Config files are saved with each experiment and are easily reproduced.
6. Trackable changes:
   • Keep track of changes to an experiment's configuration.

Installation

pip install powerful-benchmarker

Citing the benchmark results or code

If you'd like to cite the benchmark results, please cite this paper:

@misc{musgrave2020metric,
    title={A Metric Learning Reality Check},
    author={Kevin Musgrave and Serge Belongie and Ser-Nam Lim},
    year={2020},
    eprint={2003.08505},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

Acknowledgements

Thank you to Ser-Nam Lim at Facebook AI, and my research advisor, Professor Serge Belongie. This project began during my internship at Facebook AI where I received valuable feedback from Ser-Nam, and his team of computer vision and machine learning engineers and research scientists.