Tools for benchmarking and quantifying uncertainty
A lot of work has been done to quantify uncertainty and create benchmarks for uncertainty and robustness. In this section, we will cover some of the prominent ones. Please remember that the standards are still in the nascent stage, and as a result, many of these tools and GitHub repos may have certain limitations.
The Uncertainty Baselines library
Developed by researchers from the Google Brain research team, the University of Oxford, the University of Cambridge, Harvard University, and the University of Texas, the Uncertainty Baselines library contains a set of baselines that you can use to compare the performance of different deep learning methods. The baselines are implemented using high-quality methods, and they are available for a variety of tasks. You can use these baselines to get started with your own experiments. The complete work is accessible via the GitHub repo at https://github.com/google/uncertainty-baselines.
...