Stratocumulus is the backend component of Cumulus, which aims to providing a unified backend interface for different cloud environments.
Stratocumulus does not include any Cloud SDK. You need to install the corresponding SDK separately, depending on which Cloud platform you use:
- Google Cloud: Install Google Cloud SDK.
- Amazon AWS: Install AWS CLI.
- Local machine: Works for Linux and macOS. Please make sure rsync is installed, as some commands will use it.
After that, install Stratocumulus in your Python environment by command:
pip install stratocumulus
Stratocumulus has 4 commands:
-
cp
: Copy file(s)/folder(s). -
sync
: Synchronize two folders, usually used for copying one folder. -
rm
: Remove file(s)/folder(s). -
exist
: Check if a (remote) file/folder exists.
These commands have options specific to backend. For now, Stratocumulus supports the following backends:
-
aws
: Amazon AWS. -
gcp
: Google Cloud. -
local
: Local machine. In specific, macOS or Linux.
This command copies file(s) or folder(s). See examples below:
# AWS upload
strato cp file1 folder2 s3://my-bucket/target_folder/
# AWS download
strato cp s3://my-bucket/source_folder/*.zip /target_folder/
# GCP upload
strato cp -m -r --ionice file1 folder2 gs://my-bucket/target_folder/
# GCP download
mkdir /target_folder
strato cp -m gs://my-bucket/source_folder/*.zip /target_folder/
# Local Machine
strato cp -r file1 folder2 /target_folder/
Notice that:
- For AWS backend, you must explicitly attach a trailing slash for source folder.
- For Google Cloud download, you'll have to explicitly create the target folder, and then run
strato cp
command. - Wildcards are acceptable.
-
-r
option is not needed for AWS, and copying folders is always recursive. -
-m
and--ionice
options only work for Google Cloud.
For details on the options, try command strato cp -h
.
This command synchronizes two folders. Notice that this synchronization will delete content in the target folder not existing in the source folder.
See examples below:
# AWS
strato sync source_folder s3://my-bucket/target_folder
# GCP
strato sync -m --ionice source_folder gs://my-bucket/target_folder
# Local Machine
strato sync source_folder target_folder
Notice that:
-
-m
and--ionice
options only work for Google Cloud.
For details on the options, try command strato sync -h
.
This command deletes file(s) or folder(s). See examples below:
# AWS
strato rm s3://my-bucket/file1 s3://my-bucket/folder2/
# GCP
strato rm -m gs://my-bucket/file1 gs://my-bucket/folder2 gs://my-bucket/folder3/*.zip
# Local Machine
strato rm file1 folder2
Notice that:
-
-m
option only works for Google Cloud. - For AWS backend, wildcards are not yet accepted.
For details on the options, try command strato rm -h
.
This command checks if a (remote) file or folder exists. If it exists, the command terminates without any output. Otherwise, the command will break with Python subprocess.CalledProcessError
exception.
See examples below:
# AWS
strato exists s3://my-bucket/file1
# GCP
strato exists gs://my-bucket/folder2/
# Local Machine
strato exists folder2/
Notice that this command works for both file and folder, regardless of backend.
For details on the options, try command strato exists -h
.
Type strato -h
or strato --help
to check available Stratocumulus commands.
Type strato -v
or strato --version
to check the version of Stratocumulus currently installed on your machine.