ELB Log Tools
Tools for processing logs from AWS Elastic Load Balancer
Installation
$ pip install elb-log-tools
Usage
Specifying the log bucket name
There are two options:
-
Set the
ELB_LOG_BUCKETenvironment variable, or -
Pass the bucket name to
elb-logsusing the-bor--bucketargument.$ elb-logs -b my-log-bucket
Print ELB logs to stdout
$ elb-logs | head -n 1
2019-11-06T23:00:47.799875Z production-pymcore-webserver 84.10.79.236:55801 172.16.11.238:80 0.000049 1.922708 0.000054 200 200 0 230454 "GET https://www.pymetrics.com:443/results/downloads/traits/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
Filter ELB logs
Filtering currently only outputs the PATH component of the log file.
# Only output redirects
$ elb-logs | elb-filter --include-statuses 301 302 | head -n 1
/logout/
Aggregate request counts based on pattern matching
elb-patterncounts matches URL paths against a list of regex patterns, and
outputs the counts of requests matching the patterns in CSV format.
# patterns.txt has all the URL patterns we want included in the report
$ elb-logs | elb-filter -f 404 | elb-patterncounts --patternfile patterns.txt --all
Path,Hits
/api3/apps/applications/,66762
/api3/games/games/,42192
/api/game/submission/,35418
/api2/games/list/,33487
/api2/accounts/config/,19251
/health/,18891
Examples
Get most common requests for static files that 404:
$ elb-logs | elb-filter -i 404 | grep static | head -n 50 | uniq -c | sort -bgr
![dependabot[bot]](https://avatars.githubusercontent.com/u/49699333?size=120)