HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data

Introduction

HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object-based storage such as AWS S3, Azure Blob Storage, or MinIO. HSDS can be run a single machine with or without Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure).

In addition, HSDS can be run in serverless mode with AWS Lambda or h5pyd local mode.

Quick Start

Make sure you have Python 3 and Pip installed, then:

Run install: $ ./build.sh --nolint from source tree OR install from pypi: $ pip install hsds
Create a directory the server will use to store data, example: $ mkdir ~/hsds_data
Start server: $ hsds --root_dir ~/hsds_data
Run the test suite. In a separate terminal run:
- Set user_name: $ export USER_NAME=$USER
- Set user_password: $ export USER_PASSWORD=$USER
- Set admin name: $ export ADMIN_USERNAME=$USER
- Set admin password: $ $export ADMIN_PASSWORD=$USER
- Run test suite: $ python testall.py --skip_unit
(Optional) Install the h5pyd package for an h5py compatible api and tool suite: https://github.com/HDFGroup/h5pyd
(Optional) Post install setup (test data, home folders, cli tools, etc): docs/post_install.md

To shut down the server, and the server is not running in Docker, just control-C.

If using docker, run: $ ./stopall.sh

Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image. Alternatively, an external identity provider such as Azure Active Directory or KeyCloak can be used. See: docs/azure_ad_setup.md for Azure AD setup instructions or docs/keycloak_setup.md for KeyCloak.

Detailed Install Instructions

On AWS

For complete instructions to install on a single Azure VM with Docker:

See: docs/docker_install_aws.md

For complete instructions to install on AWS Kubernetes Service (EKS):

See: docs/kubernetes_install_aws.md

For complete instructions to install on AWS Lambda:

See: docs/aws_lambda_setup.md.

On Azure

For complete instructions to install on a single Azure VM with Docker:

See: docs/docker_install_azure.md

For complete instructions to install on Azure Kubernetes Service (AKS):

See: docs/kubernetes_install_azure.md

On Prem (POSIX-based storage)

For complete instructions to install on a desktop or local server:

See: docs/docker_install_posix.md

On DCOS (BETA)

For complete instructions to install on DCOS:

See: docs/docker_install_dcos.md

General Install Topics

Setting up docker:

See docs/setup_docker.md

Post install setup and testing:

See docs/post_install.md

Authorization, ACLs, and Role Based Access Control (RBAC):

See docs/authorization.md

Running serverless with h5pyd:

See https://github.com/HDFGroup/h5pyd/blob/master/README.rst

Writing Client Applications

As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).

The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.

For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest. Note: requires v1.12.0 or greater version of the HDF5 library.