Yandex-tank input/output file parser
Yandex-tank prepare phout file with statistics after load testing. tanktools module helps to parse such files and convert to DataFrame. You can use pandas module in manual mode to handle DataFrame or use build-in functions.
Generate Yandex-tank ammo from pcap or har files using pcap2ammo or har2ammo scrips. HTTP requests are extracted completely with headers and body.
So you can:
- calc quantiles
- get information about timings, latency, status codes
- extract requests by timestamp, tag and other columns
- group and analyze specific data like total/partial RPS, average request/response size
- calc statistical metrics
- convert pcap file to ammo
- convert har file to ammo
- filter out and modify requests on ammo generating
Installation
pip install tanktools
Examples
Select DataFrame by timestamp
It is possible to parse a part of staistics by time and overal count
from tanktools import phout
flags = {
'from_date': '2018-01-18 20:09:50.123',
'to_date' : '2018-01-18 20:10:00.456',
'limit': 100
}
data = phout.parse_phout('phout.log', flags)
print("Total count: %d" % phout.size(data))
Total count: 100
Print percentiles
data = phout.parse_phout('phout.log')
phout.print_quantiles(data, 'receive_time')
Percentiles for 150030 requests from 2018-01-18 20:09:42.983 to 2018-01-18 20:10:55.108: quantile (%) receive_time (mks) 10.0 9 20.0 9 30.0 10 40.0 10 50.0 10 60.0 10 70.0 11 80.0 12 90.0 13 95.0 14 98.0 16 99.0 17 100.0 716
Note
Pay attention, timings are calculated in microseconds.
Print latency median
data = phout.parse_phout('phout.log')
# Convert and print timing in milliseconds
print("\n\nLatency median: %d ms" % int(data.latency.median() / 1000))
Latency median: 30 ms
Get RPS
data = phout.parse_phout('phout.log')
rps = phout.get_rps(data)
Print HTTP response statistics
data = phout.parse_phout('phout.log')
phout.print_http_reponses(data)
HTTP code count percent (%) 500 83429 56.38 200 61558 41.60 502 2944 1.99 0 41 0.03
Select 200 OK responses and print latency median
data = phout.parse_phout('phout.log')
selected_http_responses = data[data.proto_code == 200]
print("Latency median for 200 OK: %d" %
selected_http_responses.latency.median())
Latency median for 200 OK: 3539
Print average request/response size
print("Avg. Request / Response: %d / %d bytes" % (
data.size_in.astype(float).mean(),
data.size_out.astype(float).mean()
))
Avg. Request / Response: 364 / 26697 bytes
Note
Pay attention it is required to convert data to float for correct work of mean
function
Print RPS at Nth request
print("RPS at request:")
chunk_size = int(phout.size(data) / 2)
for start in range(0, phout.size(data), chunk_size):
data_subset = phout.subset(data, start, chunk_size)
print("\t%s: %.2f" %
(start + chunk_size, phout.get_rps(data_subset)))
RPS at request: 73986: 2062.50 147972: 2530.56
pcap2ammo
Convert pcap file to Yandex-tank ammo
pcap2ammo file.pcap
73 GET https://rambler.ru/ HTTP/1.1\r\n Host: rambler.ru\r\n Content-Length: 0\r\n\r\n
Count statistics about HTTP requests
pcap2ammo -S file.pcap
Stats:
total: 1
complete: 1
incorrect: 0
incomplete: 0
Print to file
pcap2ammo -o out.ammo file.pcap
Add or delete headers
Applyed for all requests, containing specified headers
pcap2ammo --add-header 'Referer: http://domain.com' --add-header 'X-Ip: 1.1.1.1' file.pcap
pcap2ammo --delete-header 'Content-Length' file.pcap
pcap2ammo --delete-header 'Connection' --add-header 'Connection: keep-alive' file.pcap
Filter TCP/IP packets
pcap2ammo -f 'ip.src==10.10.10.10 and tcp.dport==8080' file.pcap
Filter HTTP packets
pcap2ammo -F '"rambler.ru" in http.uri' file.pcap
You can use logical expressions in filters
pcap2ammo -F '"keep-alive" in http.headers["connection"] or "Keep-alive" in http.headers["connection"]' file.pcap
String functions over HTTP headers
pcap2ammo -F '"keep-alive" in http.headers["connection"].lower()' file.pcap
Use excluding filters also
pcap2ammo -F '"rambler.ru" != http.headers["host"]' file.pcap
har2ammo
Convert pcap file to Yandex-tank ammo
har2ammo file.har
73 GET https://rambler.ru/ HTTP/1.1\r\n Host: rambler.ru\r\n Content-Length: 0\r\n\r\n
Count statistics about HTTP requests
har2ammo -S file.har
Stats:
total: 1
complete: 1
incorrect: 0
incomplete: 0
Print to file
har2ammo -o out.ammo file.har
Add or delete headers
Applyed for all requests, containing specified headers
har2ammo --add-header 'Referer: http://domain.com' --add-header 'X-Ip: 1.1.1.1' file.har
har2ammo --delete-header 'Content-Length' file.har
har2ammo --delete-header 'Connection' --add-header 'Connection: keep-alive' file.har
Filter HTTP packets
har2ammo -F '"rambler.ru" in http.uri' file.har
You can use logical expressions and python functions in filters
har2ammo -F '"keep-alive" in http.headers["connection"] or "Keep-alive" in http.headers["connection"]' file.har
har2ammo -F '"keep-alive" not in http.headers["connection"].lower()' file.har
Please, see more information about filters in pcaper package description.