Tools for processing logs from AWS Elastic Load Balancer

pip install elb-log-tools==0.3.0


ELB Log Tools

Tools for processing logs from AWS Elastic Load Balancer


$ pip install elb-log-tools


Specifying the log bucket name

There are two options:

  1. Set the ELB_LOG_BUCKET environment variable, or

  2. Pass the bucket name to elb-logs using the -b or --bucket argument.

    $ elb-logs -b my-log-bucket

Print ELB logs to stdout

$ elb-logs | head -n 1
2019-11-06T23:00:47.799875Z production-pymcore-webserver 0.000049 1.922708 0.000054 200 200 0 230454 "GET https://www.pymetrics.com:443/results/downloads/traits/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

Filter ELB logs

Filtering currently only outputs the PATH component of the log file.

# Only output redirects
$ elb-logs | elb-filter --include-statuses 301 302 | head -n 1

Aggregate request counts based on pattern matching

elb-patterncounts matches URL paths against a list of regex patterns, and outputs the counts of requests matching the patterns in CSV format.

# patterns.txt has all the URL patterns we want included in the report

$ elb-logs | elb-filter -f 404 | elb-patterncounts --patternfile patterns.txt --all


Get most common requests for static files that 404:

$ elb-logs | elb-filter -i 404 | grep static | head -n 50 | uniq -c | sort -bgr