TrailScraper
A command-line tool to get valuable information out of AWS CloudTrail and a general purpose toolbox for working with IAM policies
Installation
OSX
$ brew install trailscraperInstallation using pip
Requirements:
- Python >= 3.5
- pip
$ pip install trailscraperRun directly using docker
$ docker run --rm --env-file <(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latestCurrent Versions starting from 0.7.0 are found on GitHub Container Registry (ghcr.io), older versions on DockerHub
Usage
- Get CloudTrail events matching a filter from CloudTrail API
- Download some logs
- Download some logs in organisational trails
- Find CloudTrail events matching a filter in downloaded logs
- Generate Policy from some CloudTrail records
- Extend existing policy by guessing matching actions
- Find CloudTrail events and generate an IAM Policy
Get CloudTrail events matching a filter from CloudTrail API
$ trailscraper select --use-cloudtrail-api \
--filter-assumed-role-arn some-arn \
--from 'one hour ago' \
--to 'now'
{
"Records": [
{
"eventTime": "2017-12-11T15:01:51Z",
"eventSource": "autoscaling.amazonaws.com",
"eventName": "DescribeLaunchConfigurations",
...
Download some logs
$ trailscraper download --bucket some-bucket \
--account-id some-account-id \
--region some-other-region \
--region us-east-1 \
--from 'two days ago' \
--to 'now' \
Note: Include us-east-1 to download logs for global services. See below for details
Download some logs in organisational trails
$ trailscraper download --bucket some-bucket \
--account-id some-account-id \
--region us-east-1 \
--org-id o-someorgid \
--from 'two days ago' \
--to 'now'
Find CloudTrail events matching a filter in downloaded logs
$ trailscraper select --filter-assumed-role-arn some-arn \
--from 'one hour ago' \
--to 'now'
{
"Records": [
{
"eventTime": "2017-12-11T15:01:51Z",
"eventSource": "autoscaling.amazonaws.com",
"eventName": "DescribeLaunchConfigurations",
...
Generate Policy from some CloudTrail records
$ gzcat some-records.json.gz | trailscraper generate
{
"Statement": [
{
"Action": [
"ec2:DescribeInstances"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
Extend existing policy by guessing matching actions
CloudTrail logs might not always contain all relevant actions.
For example, your logs might only contain the Create actions after a terraform run when you really want the delete and
update permissions as well. TrailScraper can try to guess additional statements that might be relevant:
$ cat minimal-policy.json | trailscraper guess
{
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:ListObjects"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
$ cat minimal-policy.json | ./go trailscraper guess --only Get
{
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
Find CloudTrail events and generate an IAM Policy
$ trailscraper select | trailscraper generate
{
"Statement": [
{
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:DescribeVpcs",
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"sts:AssumeRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:iam::1111111111:role/someRole"
]
}
],
"Version": "2012-10-17"
}
FAQ
How can I generate policies in CloudFormation YAML instead of JSON?
TrailScraper doesn't provide this. But you can use cfn-flip to do it:
$ trailscraper select | trailscraper generate | cfn-flip
Statement:
- Action:
- ec2:DescribeInstances
Effect: Allow
Resource:
- '*'
How can I generate policies in Terraform HCL instead of JSON?
TrailScraper doesn't provide this. But you can use iam-policy-json-to-terraform to do it:
$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform
data "aws_iam_policy_document" "policy" {
statement {
sid = ""
effect = "Allow"
resources = ["*"]
actions = [
"ec2:DescribeInstances",
]
}
}
Why is TrailScraper missing some events?
- Make sure you have logs for the
us-east-1region. Some global AWS services (e.g. Route53, IAM, STS, CloudFront) use this region. For details, check the CloudTrail Documentation
Why are some TrailScraper-generated actions not real IAM actions?
This is totally possible. Unfortunately, there is no good, machine-readable documentation on how CloudTrail events map to IAM actions so TrailScraper is using heuristics to figure out the right actions. These heuristics likely don't cover all special cases of the AWS world.
This is where you come in: If you find a special case that's not covered by TrailScraper, please open a new issue or, even better, submit a pull request.
For more details, check out the contribution guide
Why does click think I am in an ASCII environment?
Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.
Set environment variables that describe your locale, e.g. :
export LC_ALL=de_DE.utf-8
export LANG=de_DE.utf-8
or
LC_ALL=C.UTF-8
LANG=C.UTF-8
For details, see http://click.pocoo.org/5/python3/#python-3-surrogate-handling
Development
$ ./go setup # set up venv, dependencies and tools
$ ./go test # run some tests
$ ./go check # run some style checks
$ ./go # let's see what we can do here
![dependabot[bot]](https://avatars.githubusercontent.com/u/49699333?size=120)
![github-actions[bot]](https://avatars.githubusercontent.com/u/41898282?size=120)
