Convenient Python Interface for Integrating with DataStream API
Collection of Python helper utilities useful for making API requests to DataStream. Key functionality include data package(s) submission and status monitoring. Built-in conveniences include automated access token refresh, multiple package handling, and HTTP connection pooling.
INSTALLATION
type "[sudo] python setup.py install" to install for all users, or look at easy_install documentation to install in an alternate location
make sure you set environment variables like so
export ING_CLIENT_SECRET=[your client secret]
export ING_CLIENT_ID=[your client id]
USAGE: UPLOAD
usage: Simple Script to Upload a zip file [-h] [--server SERVER]
[--client-secret SECRET]
[--client-id CLIENT_ID]
[--logging-level LOG_LEVEL]
file1 [file1 ...]
python ds_upload.py t08.a.mdasnoactivate_CPCancerType_AcuteMyeloidLeukemia.zip
Starting to upload t08.a.mdasnoactivate_CPCancerType_AcuteMyeloidLeukemia.zip
{u'status': u'PROCESSING', u'pipeline-name': u'Variant Analysis Cancer Pipeline', u'users': [u'mdas@ingenuity.com'], u'analysis-name': u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.', u'title': u'MyTestID-01', u'status-url': u'https://api-stable.ingenuity.com/v1/datapackages/DP_186674283961627935264', u'warnings': [{u'message': u'r_001.txt: The field Investigation File Name is missing. We strongly recommend including this field to improve the relevance of the analysis. This field may be required in the future.', u'code': u'isatab.recommended.field', u'details': {u'fieldname': u'Investigation File Name', u'fieldtype': u'field', u'filename': u'r_001.txt'}}], u'results-url': u'https://api-stable.ingenuity.com/datastream/analysisStatus.jsp?packageId=DP_186674283961627935264', u'percentage-complete': 40, u'creator': u'mdas@ingenuity.com', u'method': u'partner integration', u'stage': u'Loading samples'}
##USAGE: STATUS usage: Simple Script to check status of package [-h] [--server SERVER] [--status_url STATUS_URL] [--dp_id DP_ID] [--client-secret SECRET] [--client-id CLIENT_ID] [--logging-level LOG_LEVEL]
optional arguments:
-h, --help show this help message and exit
--server SERVER url of upload endpoint (default: https://api-
stable.ingenuity.com/datastream/api/v1/)
--status_url STATUS_URL
status_url_of_package (default: None)
--dp_id DP_ID DP_ID of package (default: None)
--client-secret SECRET
supply client secret on the command line, or set an
environment variable named ING_CLIENT_SECRET (default:
ENV['ING_CLIENT_SECRET'])
--client-id CLIENT_ID
supply client id on the command, or set an environment
variable named ING_CLIENT_ID (default:
ENV['ING_CLIENT_ID'])
--logging-level LOG_LEVEL
supplying DEBUG will also start file logging for
convenience (default: WARNING)
python ds_status.py --dp_id=DP_186674283961627935264
{u'status': u'PROCESSING', u'pipeline-name': u'Variant Analysis Cancer Pipeline', u'users': [u'mdas@ingenuity.com'], u'analysis-name': u'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.', u'title': u'MyTestID-01', u'status-url': u'https://api-stable.ingenuity.com/v1/datapackages/DP_186674283961627935264', u'warnings': [{u'message': u'r_001.txt: The field Investigation File Name is missing. We strongly recommend including this field to improve the relevance of the analysis. This field may be required in the future.', u'code': u'isatab.recommended.field', u'details': {u'fieldname': u'Investigation File Name', u'fieldtype': u'field', u'filename': u'r_001.txt'}}], u'results-url': u'https://api-stable.ingenuity.com/datastream/analysisStatus.jsp?packageId=DP_186674283961627935264', u'percentage-complete': 40, u'creator': u'mdas@ingenuity.com', u'method': u'partner integration', u'stage': u'Loading samples'}
##USAGE: FTP UPLOAD ds_ftp_upload.py -h usage: Simple Script to Upload files via FTP [-h] [--server SERVER] [--logging-level LOG_LEVEL] [--ftp-dir FTP_DIR] [--finish] [--username USER] [--password PASSWD] file1 [file1 ...]
positional arguments:
file1 a file to upload
optional arguments:
-h, --help show this help message and exit
--server SERVER url of FTP server to construct URIs with (default:
ftps2.ingenuity.com)
--logging-level LOG_LEVEL
supplying debug will also start file logging for
aconvenience (default: INFO)
--ftp-dir FTP_DIR directory top upload to (should be in emailed
instructions (default: None)
--finish send 'package is done' signal at end of transfer
(default: False)
--username USER ingenuity username (email address you registered with
(default: None)
--password PASSWD password for automated applications (default: None)
ds_ftp_upload.py --ftp-dir=for_johndoe@ingenuity.com_FTP_Test_ds2N93amCwTYz4p2bfcjYa1klSra test_data/a_ldt_92344.txt test_data/i_ldt_92344.txt test_data/r_001.txt test_data/s_001.txt test_data/hiseq_v3.1_run943-2_4025.vcf
['test_data/a_ldt_92344.txt', 'test_data/i_ldt_92344.txt', 'test_data/r_001.txt', 'test_data/s_001.txt', 'test_data/hiseq_v3.1_run943-2_4025.vcf']
2015-02-09 07:21:33,139 INFO FTP server is: ftps2.ingenuity.com, dir is: for_johndoe@ingenuity.com_FTP_Test_ds2N93amCwTYz4p2bfcjYa1klSra
Enter your Ingenuity/FTP username:johndoe@ingenuity.com
Enter password for johndoe@ingenuity.com:
2015-02-09 07:21:41,437 INFO Connected to FTP...changing dir to for_johndoe@ingenuity.com_FTP_Test_ds2N93amCwTYz4p2bfcjYa1klSra
2015-02-09 07:21:41,457 INFO Uploading test_data/a_ldt_92344.txt
2015-02-09 07:21:41,785 INFO Uploading test_data/i_ldt_92344.txt
2015-02-09 07:21:42,139 INFO Uploading test_data/r_001.txt
2015-02-09 07:21:42,456 INFO Uploading test_data/s_001.txt
2015-02-09 07:21:42,768 INFO Uploading test_data/hiseq_v3.1_run943-2_4025.vcf