dlow
A library for downloading an S3 folder recursively and unzipping its contents. Extensible to other sources and post-processes.
Installation
This library is available via pip from PyPI
pip install dlowExample use
import logging
from dlow import ResourceDownloadOrchestrator
from dlow.s3 import S3FolderDownloader, S3FolderResourceDescriptor
from dlow.post_processing import FileUnzipper
s3_bucket_name = 'SOME_BUCKET_NAME'
s3_folder = '/some_folder'
download_orchestrator = ResourceDownloadOrchestrator(dest_dir='/resources',
resource_downloader=S3FolderDownloader(s3_bucket_name, s3_folder),
resource_descriptor=S3FolderResourceDescriptor(s3_bucket_name, s3_folder),
post_download_processors=[FileUnzipper(delete_archive=True)],
clear_dest_dir=True)
download_orchestrator.ensure_resources_ready(logging.getLogger('someLogger'))AWS authentication for S3
This library uses the boto3 library to access S3. This reads the same configuration files on your host filesystem as the AWS CLI. See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
