Copyright 2018 ABSA Group Limited
              
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
            You may obtain a copy of the License at
           http://www.apache.org/licenses/LICENSE-2.0
        
 Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
                  limitations under the License.

Enceladus

Latest Release

Build Status

master	develop

Code Quality Status

Documentation

• What is Enceladus?
  • REST API
  • Menas
  • Standardization
  • Conformance
• How to build
• How to run
• Plugins
  • Built-in Plugins
• How to contribute

What is Enceladus?

Enceladus is a Dynamic Conformance Engine which allows data from different formats to be standardized to parquet and conformed to group-accepted common reference (e.g. data for country designation which are DE in one source system and Deutschland in another, can be conformed to Germany).

The project consists of four main components:

REST API

The REST API exposes the Enceladus endpoints for creating, reading, updating and deleting the models, as well as other functionalities. The main models used are:

• Runs: Although not able to be defined by users, Runs provide important overview of Standardization & Conformance jobs that have been carried out.
• Schemas: Specifies the schema towards which the dataset will be standardized
• Datasets: Specifies where the dataset will be read from on HDFS (RAW), the conformance rules that will be applied to it, and where it will land on HDFS once it is conformed (PUBLISH)
• Mapping Tables: Specifies where tables with master reference data can be found (parquet on HDFS), which are used when applying Mapping conformance rules (e.g. the dataset uses Germany, which maps to the master reference DE in the mapping table)
• Dataset Property Definitions: Datasets may be accompanied by properties, but these are not free-form - they are bound by system-wide property definitions.

The REST API exposes a Swagger Documentation UI which documents HTTP exposed endpoints. It can be found at REST_API_HOST/swagger-ui.html
In order to switch between latest and all (latest + legacy) endpoints, use the UI definition selector (right up corner in Swagger).

Menas

This is the user-facing web client, used to specify the standardization schema, and define the steps required to conform a dataset.
The Menas web client calls and is based on the REST API to get the needed entities.

Standardization

This is a Spark job which reads an input dataset in any of the supported formats and produces a parquet dataset with the Menas-specified schema as output.

Conformance

This is a Spark job which applies the Menas-specified conformance rules to the standardized dataset.

Standardization and Conformance

This is a Spark job which executes both Standardization and Conformance together in the same job

How to build

Build requirements:

• Maven 3.5.4+
• Java 8

Each module provides configuration file templates with reasonable default values. Make a copy of the *.properties.template and *.conf.template files in each module's src/resources directory removing the .template extension. Ensure the properties there fit your environment.

Build commands:

• Without tests: mvn clean package -Dskip.unit.tests
• With unit tests: mvn clean package
• With integration tests: mvn clean package -Pintegration

Test coverage:

• Test coverage: mvn clean verify -Pcode-coverage

The coverage reports are written in each module's target directory.

How to run

REST API requirements:

• Tomcat 8.5/9.0 installation
• MongoDB 4.0 installation
• HADOOP_CONF_DIR environment variable, pointing to the location of your hadoop configuration (pointing to a hadoop installation)

Deploying REST API

Simply copy the rest-api.war file produced when building the project into Tomcat's webapps directory. Another possible method is building the Docker image based on the existing Dockerfile and deploying it as a container.

Deploying Menas

There are several ways of deploying Menas:

• Tomcat deployment: copy the menas.war file produced when building the project into Tomcat's webapps directory. The "apiUrl" value in package.json should be set either before building or after building the artifact and modifying it in place
• Docker deployment: build the Docker image based on the existing Dockerfile and deploy it as a container. The API_URL environment variable should be provided when running the container
• CDN deployment: copy the built contents in the dist directory into your preferred CDN server. The "apiUrl" value in package.json in the dist directory should be set

Speed up initial loading time of REST API

• Enable the HTTP compression
• Configure spring.resources.cache.cachecontrol.max-age in application.properties of REST API for caching of static resources

Standardization and Conformance requirements:

• Spark 2.4.4 (Scala 2.11) installation
• Hadoop 2.7 installation
• REST API running instance
• REST API Credentials File in your home directory or on HDFS (a configuration file for authenticating the Spark jobs with Menas)
  • Use with in-memory authentication e.g. ~/rest-api-credential.properties:

username=user
password=changeme

• REST API Keytab File in your home directory or on HDFS
  • Use with kerberos authentication, see link for details on creating keytab files
• Directory structure for the RAW dataset should follow the convention of <path_to_dataset_in_menas>/<year>/<month>/<day>/v<dataset_version>. This date is specified with the --report-date option when running the Standardization and Conformance jobs.
• _INFO file must be present along with the RAW data on HDFS as per the above directory structure. This is a file tracking control measures via Atum, an example can be found here.

Running Standardization

<spark home>/spark-submit \
--num-executors <num> \
--executor-memory <num>G \
--master yarn \
--deploy-mode <client/cluster> \
--driver-cores <num> \
--driver-memory <num>G \
--conf "spark.driver.extraJavaOptions=-Denceladus.rest.uri=<rest_api_uri:port> -Dstandardized.hdfs.path=<path_for_standardized_output>-{0}-{1}-{2}-{3} -Dhdp.version=<hadoop_version>" \
--class za.co.absa.enceladus.standardization.StandardizationJob \
<spark-jobs_<build_version>.jar> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version> \
--raw-format <data_format> \
--row-tag <tag>

• Here row-tag is a specific option for raw-format of type XML. For more options for different types please see our WIKI.
• In case REST API is configured for in-memory authentication (e.g. in dev environments), replace --rest-api-auth-keytab with --rest-api-credentials-file

Running Conformance

<spark home>/spark-submit \
--num-executors <num> \
--executor-memory <num>G \
--master yarn \
--deploy-mode <client/cluster> \
--driver-cores <num> \
--driver-memory <num>G \
--conf 'spark.ui.port=29000' \
--conf "spark.driver.extraJavaOptions=-Denceladus.rest.uri=<rest_api_uri:port> -Dstandardized.hdfs.path=<path_of_standardized_input>-{0}-{1}-{2}-{3} -Dconformance.mappingtable.pattern=reportDate={0}-{1}-{2} -Dhdp.version=<hadoop_version>" \
--packages za.co.absa:enceladus-parent:<version>,za.co.absa:enceladus-conformance:<version> \
--class za.co.absa.enceladus.conformance.DynamicConformanceJob \
<spark-jobs_<build_version>.jar> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version>

Running Standardization and Conformance together

<spark home>/spark-submit \
--num-executors <num> \
--executor-memory <num>G \
--master yarn \
--deploy-mode <client/cluster> \
--driver-cores <num> \
--driver-memory <num>G \
--conf "spark.driver.extraJavaOptions=-Denceladus.rest.uri=<rest_api_uri:port> -Dstandardized.hdfs.path=<path_for_standardized_output>-{0}-{1}-{2}-{3} -Dhdp.version=<hadoop_version>" \
--class za.co.absa.enceladus.standardization_conformance.StandardizationAndConformanceJob \
<spark-jobs_<build_version>.jar> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version> \
--raw-format <data_format> \
--row-tag <tag>

• In case REST API is configured for in-memory authentication (e.g. in dev environments), replace --rest-api-auth-keytab with --rest-api-credentials-file

Helper scripts for running Standardization, Conformance or both together

The Scripts in scripts folder can be used to simplify command lines for running Standardization and Conformance jobs.

Steps to configure the scripts are as follows (Linux):

• Copy all the scripts in scripts/bash directory to a location in your environment.
• Copy enceladus_env.template.sh to enceladus_env.sh.
• Change enceladus_env.sh according to your environment settings.
• Use run_standardization.sh and run_conformance.sh scripts instead of directly invoking spark-submit to run your jobs.

Similar scripts exist for Windows in directory scripts/cmd.

The syntax for running Standardization and Conformance is similar to running them using spark-submit. The only difference is that you don't have to provide environment-specific settings. Several resource options, like driver memory and driver cores also have default values and can be omitted. The number of executors is still a mandatory parameter.

The basic command to run Standardization becomes:

<path to scripts>/run_standardization.sh \
--num-executors <num> \
--deploy-mode <client/cluster> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version> \
--raw-format <data_format> \
--row-tag <tag>

The basic command to run Conformance becomes:

<path to scripts>/run_conformance.sh \
--num-executors <num> \
--deploy-mode <client/cluster> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version>

The basic command to run Standardization and Conformance combined becomes:

<path to scripts>/run_standardization_conformance.sh \
--num-executors <num> \
--deploy-mode <client/cluster> \
--rest-api-auth-keytab <path_to_keytab_file> \
--dataset-name <dataset_name> \
--dataset-version <dataset_version> \
--report-date <date> \
--report-version <data_run_version> \
--raw-format <data_format> \
--row-tag <tag>

Similarly for Windows:

<path to scripts>/run_standardization.cmd ^
--num-executors <num> ^
--deploy-mode <client/cluster> ^
--rest-api-auth-keytab <path_to_keytab_file> ^
--dataset-name <dataset_name> ^
--dataset-version <dataset_version> ^
--report-date <date> ^
--report-version <data_run_version> ^
--raw-format <data_format> ^
--row-tag <tag>

Etc...

The list of options for configuring Spark deployment mode in Yarn and resource specification:

For more information on these options see the official documentation on running Spark on Yarn: https://spark.apache.org/docs/latest/running-on-yarn.html

The list of all options for running Standardization, Conformance and the combined Standardization And Conformance jobs:

The list of additional options available for running Standardization:

Most of these options are format specific. For details see the documentation.

^* Can also be specified as a unicode value in the following ways: U+00A1, u00a1 or just the code 00A1. In case empty string option needs to be applied, the keyword none can be used.

The list of additional options available for running Conformance:

All the additional options valid for both Standardization and Conformance can also be specified when running the combined StandardizationAndConformance job

How to measure code coverage

./mvn clean verify -Pcode-coverage

If module contains measurable data the code coverage report will be generated on path:

{local-path}\enceladus\{module}\target\jacoco

Plugins

Standardization and Conformance support plugins that allow executing additional actions at certain times of the computation. To learn how plugins work, when and how their logic is executed, please refer to the documentation.

Built-in Plugins

The purpose of this module is to provide some plugins of additional but relatively elementary functionality. And also to serve as an example how plugins are written: detailed description

Examples

A module containing examples of the project usage.

How to contribute

Please see our Contribution Guidelines.

Extras

• For Menas migration, there is a useful script available in scripts/migration/migrate_menas.py (dependencies.txt provided, to install missing ones, run pip install -r scripts/migration/requirements.txt)