fluent-plugin-masking

Overview

Fluentd filter plugin to mask sensitive or privacy records with ******* in place of the original value. This data masking plugin protects data such as name, email, phonenumber, address, and any other field you would like to mask.

Requirements

Installation

Install with gem:

fluent-gem install fluent-plugin-masking

Setup

In order to setup this plugin, the parameter fieldsToMaskFilePath needs to be a valid path to a file containing a list of all the fields to mask. The file should have a unique field on each line. These fields are case-sensitive (Name != name). in case you need one (or more) of the fields to be case insensitive, use the /i suffix in your field name. see example below.

Optional configuration

• fieldsToExcludeJSONPaths - this field receives as input a comma separated string of JSON fields that should be excluded in the masking procedure. Nested JSON fields are supported by dot notation (i.e: path.to.excluded.field.in.record.nestedExcludedField) The JSON fields that are excluded are comma separated.
  This can be used for logs of registration services or audit log entries which do not need to be masked.

• handleSpecialEscapedJsonCases - a boolean value that try to fix special escaped json cases. this feature is currently on alpha stage (default: false). for more details about thoose special cases see Special Json Cases

An example with optional configuration parameters:

<filter "**">
  @type masking
  fieldsToMaskFilePath "/path/to/fields-to-mask-file"
  fieldsToExcludeJSONPaths "excludedField,exclude.path.nestedExcludedField"
  handleSpecialEscapedJsonCases true
</filter>

Example fields-to-mask-file:

name
email
phone/i # the '/i' suffix will make sure phone field will be case insensitive

Quick Guide

Configuration:

# fluent.conf
----------------------------------
<source>
  @type tail
  path /tmp/test.log
  pos_file /tmp/test.log.pos
  tag maskme
  format none
</source>

<filter "**">
  @type masking
  fieldsToMaskFilePath "/path/to/fields-to-mask-file"
  fieldsToExcludeJSONPaths "excludedField,exclude.path.nestedExcludedField"
</filter>

<match "**">
  @type stdout
</match>



# /path/to/fields-to-mask-file
----------------------------------
first_name
last_name
address
phone
password
email

Result

To run the above configuration, run the following commands:

fluentd -c fluent.conf
echo '{ :body => "{\"first_name\":\"mickey\", \"type\":\"puggle\", \"last_name\":\"the-dog\", \"password\":\"d0g43u39\"}"}' > /tmp/test.log

This sample result is created from the above configuration file fluent.conf. As expected, the following fields configured to be masked are masked with ******* in the output.

2019-09-15 16:12:50.359191000 +0300 maskme: {"message":"{ :body => \"{\\\"first_name\\\":\\\"*******\\\", \\\"type\\\":\\\"puggle\\\", \\\"last_name\\\":\\\"*******\\\", \\\"password\\\":\\\"*******\\\"}\"}"}

A sample with exclude in use:

fluentd -c fluent.conf
echo '{ :body => "{\"first_name\":\"mickey\", \"type\":\"puggle\", \"last_name\":\"the-dog\", \"password\":\"d0g43u39\"}", "excludeMaskFields"=>"first_name,last_name"}' > /tmp/test.log

2019-12-01 14:25:53.385681000 +0300 maskme: {"message":"{ :body => \"{\\\"first_name\\\":\\\"mickey\\\", \\\"type\\\":\\\"puggle\\\", \\\"last_name\\\":\\\"the-dog\\\", \\\"password\\\":\\\"*******\\\"}\"}"}

Run Unit Tests

gem install bundler
bundle install
ruby -r ./test/*.rb

Special escaped json cases handling