Multilog

A simple, multiprocess-safe logger for Python

Why

Python's built-in loggers are pretty handy - they're easily customized and come with useful functionality out of the box, including things like file rotation. These file handlers are thread-safe, but not process-safe, so, if you're running a webserver in a pre-forking environment, for example, you run the risk of your workers trampling over each other when writing to a common log file. File locking is a possible workaround, but that's yucky.

To avoid this, it is recommended that one uses a socket-based logger (a code sample is helpfully provided in the Logging Cookbook). However, it is just a code snippet. Multilog is a dependency-free implementation of the sample socket logger with some niceties, like fileConfig support, and parameterization.

How

Once installed, the Multilog daemon can be invoked via:

mutlilog

Usage:

usage: multilog [-h] [-s SERVER] [-p PORT] [-c CONFIG_PATH]

A simple logger for multiple Python processes.

optional arguments:
  -h, --help                    show this help message and exit
  -s SERVER, --server SERVER
                                The server hostname (default: localhost)
  -p PORT, --port PORT          The port to listen on. (default: 9020)
  -c CONFIG_PATH, --config CONFIG_PATH
                                The log configuration to load. (default: logging.ini)

By default, it will look for a logging.ini file in the current directory. If one isn't found, Multilog will yell at you. A sample configuration file for the server:

[loggers]
keys=root

[handlers]
keys=multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogServerHandler

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

and for your application:

[loggers]
keys=root

[handlers]
keys=multilogClientHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=multilogClientHandler

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

The important field is the args block in the handler_multilogClientHandler section - those parameters should correspond to the server and ports on which the multilog daemon is listening. By default, the daemon uses localhost and logging.handlers.DEFAULT_TCP_LOGGING_PORT.

For Power Users

If you want to have Multilog share your application's config, you can do the following:

[loggers]
keys=root,appName

[handlers]
keys=multilogClientHandler,multilogServerHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=NOTSET
handlers=%(root_handler)s

[logger_appName]
level=INFO
handlers=
propagate=1
qualname=appName

[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)

[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')

[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]

Then, in your application, pass the root handler name into the logging config:

import logging
logging.config.fileConfig(config_path, defaults={"root_handler": "multilogClientHandler"})

Multilog will always load the multilogServerHandler handler. If you don't want to run Multilog (if you're running a single-threaded local dev server, for example), simply change your root_handler value to multilogServerHandler to write to the handler.

Support

Multilog is compatible with Python 2.6, 2.7, and 3.3+.