A collection of terascope monorepo scripts


Keywords
elasticsearch, hadoop, hdfs, json, kafka
License
MIT
Install
npm install @terascope/scripts@0.20.3

Documentation

Teraslice

Distributed computing platform for processing JSON data

Teraslice is an open source, distributed computing platform for processing JSON data. It works together with Elasticsearch and Kafka to enable highly scalable data processing pipelines.

It supports the creation of custom processor logic implemented in JavaScript and plugged into to the system to validate, transform and enrich data. Processing pipelines are scalable and easily distributable across many computers.

Build Status codecov tested with jest Known Vulnerabilities License

Getting Started

Teraslice is written in Node.js and has been tested on Linux and Mac OS X.

Dependencies

  • Node.js 8 or above
  • Yarn (development only)
  • At least one Elasticsearch 5 or above cluster

Installation

# Install teraslice globally
npm install --global teraslice
# Or with yarn, yarn global add teraslice

# A teraslice CLI client
npm install --global teraslice-cli
# Or with yarn, yarn global add teraslice-cli

# To add additional connectors, use
# npm install terafoundation_kafka_connector

Running

Create a configuration file called config.yaml:

terafoundation:
    connectors:
        elasticsearch:
            default:
                host:
                    - localhost:9200

teraslice:
    workers: 8
    master: true
    master_hostname: 127.0.0.1
    name: teraslice
    hostname: 127.0.0.1

Starting a single-node teraslice instance:

NOTE: Elasticsearch must be running first.

teraslice -c config.yaml

Deploy needed assets:

For many use cases elasticsearch is a good start.

teraslice-cli assets deploy localhost terascope/elasticsearch-assets

There are also asset bundles available for:

If you want to get a simple cluster going use, the example docker-compose file:

docker-compose up --build

Documentation

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

Apache-2.0.

Some packages in this repository are licensed under MIT.