Distributed computing platform for processing JSON data
Teraslice is an open source, distributed computing platform for processing JSON data. It works together with Elasticsearch and Kafka to enable highly scalable data processing pipelines.
Teraslice is written in Node.js and has been tested on Linux and Mac OS X.
- Node.js 8 or above
- Yarn (development only)
- At least one Elasticsearch 5 or above cluster
# Install teraslice globally npm install --global teraslice # Or with yarn, yarn global add teraslice # A teraslice CLI client npm install --global teraslice-cli # Or with yarn, yarn global add teraslice-cli # To add additional connectors, use # npm install terafoundation_kafka_connector
Create a configuration file called
terafoundation: connectors: elasticsearch: default: host: - localhost:9200 teraslice: workers: 8 master: true master_hostname: 127.0.0.1 name: teraslice hostname: 127.0.0.1
Starting a single-node teraslice instance:
NOTE: Elasticsearch must be running first.
teraslice -c config.yaml
Deploy needed assets:
For many use cases elasticsearch is a good start.
teraslice-cli assets deploy localhost terascope/elasticsearch-assets
There are also asset bundles available for:
If you want to get a simple cluster going use, the example docker-compose file:
docker-compose up --build
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
Some packages in this repository are licensed under MIT.