acrawler-prometheus
The handler working with aCrawler and Prometheus.
Export statistics for:
- Concurrent requests
- Task (Requests, Items) counts
- Queue status
Installation
$ pip install acrawler_prometheusUsage
Add Handler:
class MyCrawler(Crawler):
middleware_config = {
"acrawler_prometheus.PromExporter": 100,
}
config = {
"PROMETHEUS_INTERVAL": 5
}Avaliable Config:
PROMETHEUS_ADDR = "localhost"
PROMETHEUS_PORT = 8000
PROMETHEUS_INTERVAL = 1 # exporting interval, in second
