Angle-Pipe Log

Python library for parsing webserver logs in Angle-Pipe Format

Install:

pip install anglepipelog

Log Format

DATETIME<|>STATUS<|>VHOST<|>PORT<|>REQUEST<|>BYTES<|>TIME<|>USER<|>IPADDR<|>REFERER<|>UA
2013-05-17T15:21:46+0100<|>404<|>testing.example.com<|>80<|>GET /index.html HTTP/1.0<|>391<|>0<|>-<|>127.0.0.1<|>/referer<|>firefox

Angle bracketed pipes are used as the delimiter because that arrangement is highly unlikely to occur and according to the specifications should not.

Webserver Format Strings

Apache

LogFormat "%{%Y-%m-%dT%H:%M:%S%z}t<|>%>s<|>%v<|>%p<|>%r<|>%O<|>%T<|>%u<|>%{X-Forwarded-For}i<|>%{Referer}i<|>%{User-Agent}i" anglepipe

Nginx

log_format anglepipe '$time_iso8601<|>$status<|>$server_name<|>$server_port<|>$request<|>$bytes_sent<|>$request_time<|>$remote_user<|>$remote_addr<|>$http_referer<|>$http_user_agent';

Usage

Code

import anglepipelog
log = anglepipelog.log('log_file.log')
print log.as_dicts

Output

[{'bytes': '406',
  'datetime': '2013-05-17T15:21:14+0100',
  'host': 'example.com',
  'ipaddress': '127.0.0.1',
  'method': 'GET',
  'port': '80',
  'referer': 'google.com',
  'request': '/index.html',
  'status': '200',
  'time': '0',
  'user': '-',
  'useragent': 'User Agent String\n',
  'version': 'HTTP/1.0'},
 {'bytes': '416',
  'datetime': '2013-05-17T15:21:14+0100',
  'host': 'example.com',
  'ipaddress': '127.0.0.1',
  'method': 'GET',
  'port': '80',
  'referer': 'google.com',
  'request': '/index.html',
  'status': '200',
  'time': '0',
  'user': '-',
  'useragent': 'User Agent String\n',
  'version': 'HTTP/1.0'}]

Versions

• 0.3 - Switched to Angle Bracketed Pipes as some URLs aren't meeting spec.
• 0.2 - Improved Docstrings
• 0.1 - Initial Release