gozstd - go wrapper for zstd

Difference from github.com/valyala/gozstd

This fork removes the concurrency limiting channel which was a significant performance bottleneck in a highly-concurrent application running on a 64 core CPU.

Features

• Vendors upstream zstd without any modifications.

• Simple API.

• Optimized for speed. The API may be easily used in zero allocations mode.

• Compress* and Decompress* functions are optimized for high concurrency.

• Proper Writer.Flush for network apps.

• Supports the following features from upstream zstd:

  • Block / stream compression / decompression with all the supported compression levels and with dictionary support.
  • Dictionary building from a sample set. The created dictionary may be saved to persistent storage / transfered over the network.
  • Dictionary loading for compression / decompression.

  Pull requests for missing upstream zstd features are welcome.

Quick start

How to install gozstd?

go get -u github.com/ClearcodeHQ/gozstd

How to compress data?

The easiest way is just to use Compress:

    compressedData := Compress(nil, data)

There is also StreamCompress and Writer for stream compression.

How to decompress data?

The easiest way is just to use Decompress:

    data, err := Decompress(nil, compressedData)

There is also StreamDecompress and Reader for stream decompression.

How to cross-compile gozstd?

If you're cross-compiling some code that uses gozstd and you stumble upon the following error:

# github.com/ClearcodeHQ/gozstd
/go/pkg/mod/github.com/ClearcodeHQ/gozstd@v1.6.2/stream.go:31:59: undefined: CDict
/go/pkg/mod/github.com/ClearcodeHQ/gozstd@v1.6.2/stream.go:35:64: undefined: CDict
/go/pkg/mod/github.com/ClearcodeHQ/gozstd@v1.6.2/stream.go:47:20: undefined: Writer

You can easily fix it by enabling CGO and using a cross-compiler ( e.g. arm-linux-gnueabi-gcc):

env CC=arm-linux-gnueabi-gcc GOOS=linux GOARCH=arm CGO_ENABLED=1 go build ./main.go 

NOTE: Check #21 for more info.

Who uses gozstd?

• VictoriaMetrics

FAQ

• Q: Which go version is supported?
  A: go1.10 and newer. Pull requests for older go versions are accepted.

• Q: Which platforms/architectures are supported?
  A: linux/amd64, linux/arm, linux/arm64, freebsd/amd64, darwin/amd64, windows/amd64. Pull requests for other platforms/architectures are accepted.

• Q: I don't trust libzstd*.a binary files from the repo or these files dont't work on my OS/ARCH. How to rebuild them? A: Just run make clean libzstd.a if your OS/ARCH is supported.

• Q: How do I specify custom build flags when recompiling libzstd*.a? A: You can specify MOREFLAGS=... variable when running make like this: MOREFLAGS=-fPIC make clean libzstd.a.

• Q: Why the repo contains libzstd*.a binary files?
  A: This simplifies package installation to usual go get without additional steps for building the libzstd*.a