Kafka module for Puppet
Table of Contents
- Overview
- Module Description - What the module does and why it is useful
- Setup - The basics of getting started with kafka
- Usage - Configuration options and additional functionality
- Reference - An under-the-hood peek at what the module is doing and how
- Limitations - OS compatibility, etc.
- Development - Guide for contributing to the module
Overview
The Kafka module for managing the installation and configuration of Apache Kafka
Module Description
The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers.
Setup
What kafka affects
Installs the Kafka package and creates a new service.
Beginning with Kafka
To successfully install Kafka using this module you need to have Apache ZooKeeper already running at localhost:2181. You can specify another ZooKeeper host:port configuration using the config hash of the kafka:broker class.
The default configuration installs Kafka 0.11.0.3 binaries with Scala 2.11
class { 'kafka': }
If you want a Kafka broker server that connects to ZooKeeper listening on port 2181:
class { 'kafka::broker':
config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
}
Usage
You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the Apache Kafka Website
Installing Kafka version 1.1.0 with scala 2.12
We first install the binary package with:
class { 'kafka':
version => '1.1.0',
scala_version => '2.12'
}
Then we set a minimal Kafka broker configuration with:
class { 'kafka::broker':
config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
}
Classes and Defined Types
kafka
Class: One of the primary classes of the kafka module. This class will install the kafka binaries
Parameters within kafka
:
version
The version of kafka that should be installed.
scala_version
The scala version what kafka was built with.
install_dir
The directory to install kafka to.
mirror_url
The url where the kafka is downloaded from.
mirror_subpath
The url subpath where the kafka is downloaded from (default value: /kafka/{version}
).
proxy_port
The proxy port where the kafka is downloaded from.
proxy_host
The proxy host where the kafka is downloaded from.
proxy_server
The proxy server where the kafka is downloaded from (to use instead of proxy_port
and proxy_host
if you need to set an url as proxy for example).
proxy_type
The proxy type where the kafka is downloaded from (http
for example).
install_java
Install java if it's not already installed.
package_dir
The directory to install kafka.
package_name
Package name, when installing kafka from a package.
package_ensure
Package version (or 'present', 'absent', 'latest'), when installing kafka from a package.
group_id
Create kafka group with this ID
user_id
Create kafka user with this ID
user
User to install kafka as. Defaults to the kafka user.
group
Group to install kafka as. Defaults to the kafka group.
config_dir
Directory for kafka config files. Defaults to /opt/kafka/config.
log_dir
Directory for kafka log files. Defaults to /var/log/kafka.
kafka::broker
Class: One of the primary classes of the kafka module. This class will install a kafka broker.
Parameters within kafka::broker
:
version
The version of kafka that should be installed.
scala_version
The scala version what kafka was built with.
install_dir
The directory to install kafka to.
mirror_url
The url where the kafka is downloaded from.
config
A hash of the configuration options. All values are used in the
server.properties
file directly.
install_java
Install java if it's not already installed.
service_ensure
Sets the ensure state of the broker service to stopped or running.
service_install
Install the init.d service.
service_restart
Whether the configuration files should trigger a service restart
package_dir
The directory to install kafka.
kafka::consumer
Class: One of the primary classes of the kafka module. This class will install a kafka consumer.
Parameters within kafka::consumer
:
version
The version of kafka that should be installed.
scala_version
The scala version that kafka was built with.
install_dir
The directory to install kafka to.
mirror_url
The url where the kafka is downloaded from.
install_java
Install java if it's not already installed.
package_dir
The directory to install kafka.
kafka::mirror
Class: One of the primary classes of the kafka module. This class will install a kafka mirror.
Parameters within kafka::mirror
:
version
The version of kafka that should be installed.
scala_version
The scala version that kafka was built with.
install_dir
The directory to install kafka to.
mirror_url
The url where the kafka is downloaded from.
install_java
Install java if it's not already installed.
package_dir
The directory to install kafka.
kafka::producer
Class: One of the primary classes of the kafka module. This class will install a kafka producer.
Parameters within kafka::producer
:
version
The version of kafka that should be installed.
scala_version
The scala version that kafka was built with.
install_dir
The directory to install kafka to.
mirror_url
The url where the kafka is downloaded from.
install_java
Install java if it's not already installed.
package_dir
The directory to install kafka.
kafka::topic
Define: Defined type that creates Kafka topics.
Reference
Classes
Public Classes
-
kafka
: Guides the basic installation of kafka binaries -
kafka::broker
: Guides the basic installation of a kafka broker -
kafka::consumer
: Guides the basic installation of a kafka consumer -
kafka::mirror
: Guides the basic installation of a kafka mirror -
kafka::producer
: Guides the basic installation of a kafka producer
Private Classes
- [
kafka::broker::config
] Manages all the default configuration of the kafka broker application - [
kafka::broker::install
] Manages the installation of the kafka packages - [
kafka::broker::service
] Manages the kafka server service - [
kafka::consumer::config
] Manages all the default configuration of the kafka consumer application - [
kafka::consumer::install
] Manages the installation of the kafka packages - [
kafka::consumer::service
] Manages the kafka server service - [
kafka::mirror::config
] Manages all the default configuration of the kafka mirror application - [
kafka::mirror::install
] Manages the installation of the kafka packages - [
kafka::mirror::service
] Manages the kafka server service - [
kafka::producer::config
] Manages all the default configuration of the kafka producer application - [
kafka::producer::install
] Manages the installation of the kafka packages - [
kafka::producer::service
] Manages the kafka server service
Defined Types
-
kafka::topic
Create kafka topics
Limitations
This module only supports Kafka >= 0.9.0.0.
This module is tested on the following platforms:
- CentOS 5
- CentOS 6
- Ubuntu 12.04
- Ubuntu 14.04
It is tested with the OSS version of Puppet (>= 4.7.0) only.
Development
Contributing
Please read CONTRIBUTING.md for full details on contributing to this project.