edenhill/librdkafka


The Apache Kafka C/C++ library

License: Other

Language: C

Keywords: apache-kafka, c, c-plus-plus, consumer, high-performance, kafka, kafka-consumer, kafka-producer, librdkafka


librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2019, Magnus Edenhill.

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, providing Producer, Consumer and Admin clients. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

Features

  • High-level producer
  • High-level balanced KafkaConsumer (requires broker >= 0.9)
  • Simple (legacy) consumer
  • Admin client
  • Compression: snappy, gzip, lz4, zstd
  • SSL support
  • SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM, OAUTHBEARER) support
  • Broker version support: >=0.8 (see Broker version compatibility)
  • Stable C & C++ APIs (ABI safety guaranteed for C)
  • Statistics metrics
  • Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
  • RPM package: librdkafka and librdkafka-devel
  • Gentoo package: dev-libs/librdkafka
  • Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Documentation

NOTE: The master branch is actively developed, use latest release for production use.

Installation

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:

$ yum install librdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Build from source

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer enables encoding the original size in the compression frame header, which will speed up the consumer. Use STATIC_LIB_zstd=/path/to/libzstd.a ./configure --enable-static to enable static ZStd linking. MacOSX example: STATIC_LIB_zstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Building

  ./configure
  # Or, to automatically install dependencies using the system's package manager:
  # ./configure --install-deps
  # Or, build dependencies from source:
  # ./configure --install-deps --source-deps-only

  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See the examples directory for an example producer and consumer.

Link your program with -lrdkafka (C) or -lrdkafka++ (C++).

Commercial support

Commercial support is available from Confluent Inc

Community support

Only the last official release is supported for community members.

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on the Confluent Community slack #clients channel, or irc.freenode.org #apache-kafka channel.

Language bindings

Users of librdkafka

Project Statistics

Sourcerank 13
Repository Size 14 MB
Stars 3,170
Forks 1,456
Watchers 272
Open issues 138
Dependencies 6
Contributors 144
Tags 137
Created
Last updated
Last pushed

Top Contributors See all

Magnus Edenhill Ron Dagostino Faidon Liambotis Andreas Heider Matt Howlett Eugenio Pérez Bob Halley Andreas Kohn Taylor Burmeister asharma339 Martin Rösch Andreas Smas Andreas Kohn Andreas Kohn Alexey Kurbatsky kodekarl Thomas Deutschmann Vincent Bernat Andreas Kohn Omer Katz

Packages Referencing this Repo

librdkafka.redist
The Apache Kafka C/C++ client library - redistributable
Latest release 1.1.0 - Updated - 3.17K stars
librdkafka.redist.win7-x64
The Apache Kafka C/C++ client library - redistributable
This package is no longer available on NuGet
librdkafka.redist.debian-x64
The Apache Kafka C/C++ client library - redistributable
Latest release 0.11.1-TEST4 - Updated - 3.17K stars
librdkafka.redist.win-x86
The Apache Kafka C/C++ client library - redistributable
Latest release 0.11.1-TEST4 - Published - 3.17K stars
librdkafka.redist.treziac
The Apache Kafka C/C++ client library - redistributable - TEST ONLY, use librdkafka.redist
Latest release 0.9.4-patchnupkg3 - Updated - 3.17K stars
librdkafka.redist.win7-x86
The Apache Kafka C/C++ client library - redistributable
This package is no longer available on NuGet
librdkafka.redist.win-x64
The Apache Kafka C/C++ client library - redistributable
Latest release 0.11.1-TEST4 - Published - 3.17K stars
librdkafka.testing
Fully-fledged Apache Kafka client implemented in C
Latest release 0.9.2-devel-master - Published - 3.17K stars
librdkafka.symbols
Symbols for package 'librdkafka'. This package should not likely be installed. (This is not the ...
Latest release 0.9.3-pre-wip1 - Updated - 3.17K stars
treziac.librdkafka
TEST ONLY, use librdkafka.redist
Latest release 0.9.6 - Updated - 3.17K stars
librdkafka
The Apache Kafka C/C++ client library
Latest release 0.9.3-pre-wip1 - Updated - 3.17K stars
librdkafka
The Apache Kafka C/C++ library
Latest release 0.11.3 - Published - 3.17K stars

Recent Tags See all

v1.1.0-selfstatic-test6 June 23, 2019
v1.1.0-selfstatic-test5 June 23, 2019
v1.1.0 June 23, 2019
v1.1.0-selfstatic-test4 June 23, 2019
v1.1.0-selfstatic-test3 June 23, 2019
v1.1.0-selfstatic-test2 June 20, 2019
v1.1.0-RC1 June 18, 2019
v1.0.1-selfstatic-test1 June 03, 2019
v1.0.1 May 28, 2019
v1.0.1-sslcertcb-test2 May 27, 2019
v1.0.1-sslcertcb-test1 May 24, 2019
v1.0.1-RC1 May 21, 2019
v1.0.0-RC10 March 25, 2019
v1.0.0 March 25, 2019
v1.0.0-RC9 March 21, 2019

Interesting Forks See all

redBorder/librdkafka
Apache Kafka C library (experimental C++ wrapper available)
C - BSD-2-Clause - Last pushed - 2 stars
confluentinc/librdkafka
The Apache Kafka C/C++ library
C - Updated - 2 stars - 1 forks
jcrutsinger/librdkafka
The Apache Kafka C/C++ library
C - Updated - 1 stars - 1 forks
lambdaknight/librdkafka
The Apache Kafka C/C++ library
C - Last pushed - 1 stars
jopstools/librdkafka
The Apache Kafka C/C++ library
C - Last pushed - 1 stars

Something wrong with this page? Make a suggestion

Last synced: 2019-06-23 22:18:55 UTC

Login to resync this repository