In the middle of some calibrations...
Garrus is a python framework for better confidence estimate of deep neural networks. Modern networks are overconfident estimators, that makes themselves unreliable and therefore limits the deployment of them in safety-critical applications.
Garrus provides tools for high quality confidence estimation such as confidence calibration and ordinal ranking methods, helping networks to know correctly what they do not know.
Installation:
pip install -U garrusDocumentation:
Roadmap:
- Core:
- Calibration metrics:
- ECE
- NLL
- Brier
- Ordinal Ranking Metrics:
- AURC
- E-AURC
- AUPRE
- FPR-n%-TPR
- Visualizations:
- Reliability Diagram
- Confidence Histogram
- Garrus Profiling
- Calibration metrics:
- Confidence Calibration:
- Scaling:
- Platt
- Temperature
- Binning:
- Histogram
- Isotonic Regression
- Scaling:
- Confidence Regularization:
- Losses:
- Correctness Ranking Loss
- Focal Entropy Penalized Loss
- Language Model Beam Search
- Losses:
- Confidence Networks:
- ConfidNet
- GarrusNet
Citation:
Please use this bibtex if you want to cite this repository in your publications:
@misc{garrus,
author = {Kalashnikov, Alexander},
title = {Deep neural networks calibration framework},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/sleep3r/garrus}},
}
References:
| Papers |
|---|
| [1] Guo, Chuan, et al. "On calibration of modern neural networks." International Conference on Machine Learning. PMLR, 2017. APA |
| [2] Moon, Jooyoung, et al. "Confidence-aware learning for deep neural networks." international conference on machine learning. PMLR, 2020. |
| [3] Kumar, Ananya, Percy Liang, and Tengyu Ma. "Verified uncertainty calibration." arXiv preprint arXiv:1909.10155 (2019). |
