github.com/Doist/s3logger
Release v1.4.0

Program s3logger collects json messages over TCP, stores them into gzip-compressed files split by time and uploads these files to AWS s3 bucket. Once started, service accepts TCP connections and expects clients to send streams of json objects over such connections. s3logger only closes connection if it encounters malformed json or single object size exceeds 4 MiB limit. s3logger only reads data from the client. s3logger accumulates received messages over predefined time window (-t flag) to a temporary log file creating new ones as needed; previous files are uploaded to s3 bucket in background and removed after successful upload. Optionally maximum size of input read can be specified in megabytes (-mb flag) to rotate file before reaching predefined time. Program only writes to a single temporary log file at a time, so json messages received from multiple concurrent connections are interleaved into a single json stream. It does its best not to lose messages, but can still drop them if they're coming faster than could be saved on disk or there's any disk write error. Stored messages are separated by new line (0xa). s3logger uploads files to a specified bucket using predefined s3 object naming scheme: It uses dt=YYYY-MM-DD "directories", object name base starting with date and time when log file was created (UTC) followed by hex-encoded 64-bit random value and .json.gz suffix. See https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables on how to configure s3 bucket access credentials. s3logger does not use TLS for its listener at the moment as it is expected to run on localhost or inside trusted network.

Homepage Go Go Documentation

License: MIT
Install: go get github.com/Doist/s3logger

Documentation

Program s3logger collects json messages over TCP, stores them into
gzip-compressed files split by time and uploads these files to AWS s3
bucket.

Once started, service accepts TCP connections and expects clients to send
streams of json objects over such connections. s3logger only closes
connection if it encounters malformed json or single object size exceeds 4
MiB limit. s3logger only reads data from the client.

s3logger accumulates received messages over predefined time window (-t flag)
to a temporary log file creating new ones as needed; previous files are
uploaded to s3 bucket in background and removed after successful upload.
Optionally maximum size of input read can be specified in megabytes (-mb
flag) to rotate file before reaching predefined time. Program only writes to
a single temporary log file at a time, so json messages received from
multiple concurrent connections are interleaved into a single json stream.
It does its best not to lose messages, but can still drop them if they're
coming faster than could be saved on disk or there's any disk write error.
Stored messages are separated by new line (0xa).

s3logger uploads files to a specified bucket using predefined s3 object
naming scheme:

    dt=2018-02-09/20180209T213803_df718a7818e53243.json.gz

It uses dt=YYYY-MM-DD "directories", object name base starting with date and
time when log file was created (UTC) followed by hex-encoded 64-bit random
value and .json.gz suffix.

See
https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
on how to configure s3 bucket access credentials.

s3logger does not use TLS for its listener at the moment as it is expected
to run on localhost or inside trusted network.

    Usage of s3logger:
      -addr string
    	address to listen (default "localhost:8080")
      -bucket string
    	s3 bucket to upload logs
      -dir string
    	directory to keep unsent files (default "/var/spool/s3logger")
      -mb int
    	megabytes of input read until file is rotated (0 to disable) (default 512)
      -prefix string
    	s3 object name prefix (directory in a bucket)
      -t duration
    	time to use single file (min 1m) (default 5m0s)

Dependencies: 19
Dependent packages: 0
Dependent repositories: 0
Total releases: 7
Latest release: Mar 27, 2022
First release: Jul 29, 2021
Stars: 1
Forks: 0
Watchers: 2
Contributors: 5
Repository size: 33.2 KB
SourceRank: 8