Celery Logger

celery-logger is a python library for logging celery events such as tasks received, tasks failed/succeeded and tasks retried, along with task args.

Features

• Simple and flexible task logs
• Multiple deployment options (docker, virtual machines)
• Integration possibilities: ELK stack and AWS cloudwatch for example

How does it work?

Celery-logger connects with your message broker (such as redis, rabbitMQ or SQS) and logs the tasks on a file or stdout.

This way you can easily have multiple celery workers and see your logs in a single place.

Sample project

We provided a sample project for playing around and seeing how it works.

Getting started:

# navigate to the sample-project folder
$ cd sample-project
# Build and start the containers:
$ docker-compose up

Querying logs

Make sure you are in the sample-project folder

Calling a few tasks

$ docker-compose exec celeryd python call_tasks.py

Tasks have been called!
Run `docker-compose logs -f celery-logger` to see the logger in action.

Taking a look in all events

$ docker-compose logs celery-logger 

Searching for failed tasks:

$ docker-compose logs celery-logger | grep task-failed

Searching for a specific task:

$ docker-compose logs celery-logger | grep "app.add"

Searching for a specific task and args:

$ docker-compose logs celery-logger | grep "app.add" | grep "(6, 1)"

Searching for a task id

$   docker-compose logs celery-logger | grep 20925a8c-03f7-4bd7-b3dd-24e2bc9e26e2

Installation

Use the package manager pip to install celery-logger.

pip install celerylogger

Usage

The executable will be available in the path as celery-logger:

# Display the help text
$ celery-logger -h
# Start logging from a redis broker
$ celery-logger --celery-broker redis://redis:6379/0

Authors

• Sergio Oliveira - Initial work - Seocam
• Thiago Ferreira - Improvements, documentation and current maintainer - thiagoferreiraw
• André Girol - Packaging and distribution - Girol