Kafka Avro dumping utility written in Go.
Most command-line tools that can consume Avro records from Kafka output data using Avro JSON encoding. When using Avro union types, the Avro JSON encoding includes the specific type in the field to disambiguate.
For example, for a fieldTwo
field defined as type [null, string]
in Avro:
{"fieldOne":100,"fieldTwo":{"string":"hello"}}
While the above is necessary for correctly encoding JSON data into Avro, it can become inconvenient when decoding plain JSON data from Avro and feeding it into non-Avro applications for processing.
For a more conveniently decoded JSON data, this tool uses the
goavro
library. More specifically, kadumper
uses the Standard Full JSON
codec which (from the documentation) provides full serialization/deserialization for JSON data that
meets the expectations of regular internet JSON.
For the same example fieldTwo
field shown above, kadumper
outputs the following JSON data:
{"fieldOne":100,"fieldTwo":"hello"}
As added convenience, kadumper
can also output raw textual data (non-Avro) from Kafka.
All currently available command-line arguments can be seen with -h/--help
.
Consuming Avro data from Kafka is currently only supported in Confluent Wire format with schemas automatically downloaded from a Schema Registry instance. Downloaded schemas are cached with a configurable maximum caching age. Confluent Platform and Redpanda have both been tested.
To output raw textual data (non-Avro) from Kafka, omit the --schema-registry-url
argument.
Output Kafka records are separated by a newline \n
character. For each record, output timestamps,
partitions, offsets, keys and values are separated by a TAB \t
character. All output is sent to
the standard output.
Currently, only anonymous plain-text and mutual TLS authentication connections are supported.
Note: Ready-to-use binaries are available in the releases page.
To build a snapshot locally using GoReleaser:
goreleaser build --clean --single-target --output kadumper --snapshot
To release a new version in GitHub using GoReleaser:
git tag vX.Y.Z
CGO_ENABLED=0 GITHUB_TOKEN=$(< /path/to/token) goreleaser release --clean
This project is licensed under the Apache License Version 2.0.