af-metrics

Python package for monitoring the Coffea-Casa Analysis Facility using Prometheus Monitoring tool and generating dashboards from the collected metrics using Grafana. The package deploys Prometheus locally on localhost:8003 and establishes a connection between Prometheus and the JSON Data collected from the specific Jupyter Notebook to be monitored in the Coffea-Casa Analysis Facility using the Prometheus Client Package. This JSON data serves as the custom metrics for the Prometheus server. These Prometheus metrics serve as the data for generating convenient dashboards using the Grafana tool.

The current infrastructure can be represented as:

In the existing infrastructure of Coffea Casa Analysis Facility, the individual users run their own Jupyterhub and Dask Servers at the corresponding endpoints. In our project, we try to scrape these individual endpoints and also that of the Coffea Casa Analysis Facility as a whole using useful metrics and thus monitor the facility using the Prometheus tool. Based on the scraped metrics, we can visualize the results conveniently by generating dashboards using the Grafana tool. So Prometheus connects to the Jupyterhub and Dask Servers of the existing users and the Facility to monitor the specific metrics and Grafana, in turn, connects to Prometheus and generates useful dashboards based on the results of the monitoring process. The process of dashboard generation can also be automated using grafanalib library which automatically generates a suitable json file specifying the dashboard data source as well as the appearance of the dashboard.