S3 log parsing library
A python library to parse S3 log files.
Unit tests currently require actual S3 credentials (and a bucket with logs) and can therefore only be run manually. Mocking the relevant parts of boto.s3 is on the roadmap. Contributions are welcome :)
Download S3 logs from a bucket, and parse them.
This application does not store the log objects generated and leaves that to other applications.
From pypi, using pip:
pip install ls-s3-logs
Or just grab the sourcepackage from pypi: https://pypi.python.org/pypi/ls-s3-logs/
To get logs, simply use the Downloader class:
from itertools import chain from pprint import pprint from lss3logs.download import Downloader MY_ACCESS_KEY_ID = 'XXX' MY_KEY_SECRET = 'XXX' MY_S3_BUCKET_NAME = 'XXX' downloader = Downloader( connection=None, aws_access_key_id=MY_ACCESS_KEY_ID, aws_key_secret=MY_KEY_SECRET, ) # download 10 logs logs = downloader.download_files( MY_S3_BUCKET_NAME, prefix='logs/', max_logs=1) entries = [ log.entries for log in logs ] entries = list(chain.from_iterable(entries)) [pprint(entry.__dict__) for entry in entries]
First you need to specify the test config, which contains the AWS credentials and details of bucket tot test with. python-testconfig is used to manage the test configuration.
Copy test_config.ini.sample to test_config.ini (in the same directory) and set correct values:
To test with nose:
python setup.py nosetests
or running nosetests directly:
nosetests -s --exe
Directly and with coverage:
nosetests -s --exe --with-coverage --cover-package=lss3logs
(Note: the --exe includes python files whoch are executable, so it's optional if you don't have any.)
Checking code with pylint:
- mock boto output (see https://github.com/eykd/duo/blob/master/test_duo.py for an example)
- fix as many errors as possbile reported by pylint
- bring test coverage to 100%
- use Sphinx for docs
The regular expression for parsing the log lines is copied from a script by "kkowalczyk" located at http://code.google.com/p/kjk/source/browse/trunk/scripts/test_parse_s3_log.py