Fluentd
Install, configure, and manage Fluentd data collector.
Module Description
- Installs
td-agent
package - Generates configuration file
td-agent.conf
- Generates custom configuration files and saves them to
config.d/
- Manages
td-agent
service - Installs Fluentd gem plugins
Usage Examples
Basic
Install and start the service.
class { 'fluentd': }
Routing Events To Elasticsearch
Receive logs from other Fluentd instances and via UNIX domain socket. Forward the logs to Elasticsearch.
include fluentd
fluentd::plugin { 'fluent-plugin-elasticsearch': }
fluentd::config { '500_elasticsearch.conf':
config => {
'source' => [
{
'type' => 'forward',
},
{
'type' => 'unix',
'path' => '/tmp/td-agent/td-agent.sock',
},
],
'match' => {
'tag_pattern' => '**',
'type' => 'elasticsearch',
'index_name' => 'foo',
'type_name' => 'bar',
'logstash_format' => true,
}
}
}
Forwarding Events To Fluentd Aggregator
Read logs from UNIX domain socket, then forward them to Fluentd aggregators.
include fluentd
fluentd::config { '600_forwarding.conf':
config => {
'source' => {
'type' => unix,
'path' => '/tmp/td-agent/td-agent.sock',
},
'match' => {
'tag_pattern' => '**',
'type' => forward,
'server' => [
{ 'host' => 'example1.com', 'port' => 24224 },
{ 'host' => 'example2.com', 'port' => 24224 },
]
}
}
}
Hiera Support
Defining Fluentd resources in Hiera.
fluentd::plugins:
'fluent-plugin-http':
plugin_ensure: 0.1.0
'fluent-plugin-elasticsearch':
plugin_ensure: present
fluentd::configs:
'100_fwd.conf':
config:
source:
type: forward
'200_stdout.conf':
config:
match:
tag_pattern: test
type: stdout
Config File Naming
All configs employ a numbering system in the resource's title that is used for
ordering. When titling your config, make sure you prefix the filename with a
number, for example, 999_catch_all.conf
, 500_elasticsearch.conf
(999 has
smaller priority than 500)
Reference
Classes
Public Classes
-
fluentd
: Main class, includes all other classes.
Private Classes
-
fluentd::install
: Handles the packages. -
fluentd::service
: Handles the service.
Parameters
The following parameters are available in the fluentd
class:
repo_install
Default value: true
repo_name
Default value: 'treasuredata'
repo_desc
Default value: 'TreasureData'
repo_url
Default value: 'http://packages.treasuredata.com/2/redhat/$releasever/$basearch'
repo_enabled
Default value: true
repo_gpgcheck
Default value: true
repo_gpgkey
Default value: 'https://packages.treasuredata.com/GPG-KEY-td-agent'
repo_gpgkeyid
Default value: 'C901622B5EC4AF820C38AB861093DB45A12E206F'
package_name
Default value: 'td-agent'
package_ensure
Default value: present
service_name
Default value: 'td-agent'
service_ensure
Default value: running
service_enable
Default value: true
service_manage
Default value: true
service_provider
Default value:
- when
$facts['osfamily'] == 'redhat'
: redhat - otherwise: undef
config_file
Default value: '/etc/td-agent/td-agent.conf'
config_path
Default value: '/etc/td-agent/config.d'
config_owner
Default value: 'td-agent'
config_group
Default value: 'td-agent'
configs
Default value: {}
plugins
Default value: {}
Public Defines
-
fluentd::config
: Generates custom configuration files. -
fluentd::plugin
: Installs plugins.
The following parameters are available in the fluentd::plugin
defined type:
title
Plugin name
plugin_ensure
Default value: present
plugin_source
Default value: 'https://rubygems.org'
plugin_provider
Default value: tdagent
plugin_install_options
Default value: [] see https://docs.puppetlabs.com/puppet/latest/reference/type.html#package-provider-gem, e.g.
plugin_install_options => [{'--http-proxy' => $http_proxy}]
The following parameters are available in the fluentd::config
defined type:
title
Config filename
config
Config Hash, please see usage examples.
Limitations
Tested on CentOS 6, CentOS 7, Ubuntu 14.04, Debian 7.8
Development
Bug reports and pull requests are welcome!
Running Tests
$ bundle install
$ bin/rake lint
$ bin/rake metadata_lint
$ bin/rake spec
$ bin/rspec spec/lib
$ bin/rake beaker BEAKER_set=centos-6-x64
$ bin/rake beaker BEAKER_set=centos-7-x64
$ bin/rake beaker BEAKER_set=debian-7-amd64
$ bin/rake beaker BEAKER_set=ubuntu-server-1404-x64