- Import scientific data from various sources through one easy Python API.
- Use iterator patterns for each data source (configurable data generators + functions to get next data chunk).
- Skip parser programming and file format handling.
- Enjoy a common namespace for all data and unleash the power of data fusion.
- Handle data distribution in different modes: (1) local download, (2) caching of accessed data, or (3) online stream access
- Easily pull data on cloud servers through Python scripts and facilitate large-scale parallel processing.
- Build on an extensible plattform: Adding access to a new data source only requires addition of its "DataFetcher.py".
- Open source (MIT License)
Supported data sets:
Namespace |
Preview |
Description & Data Source |
|
Astronomy | |||
astro.kepler |
Light curves for stars imaged by the NASA Kepler Space Telescope Source: https://keplerscience.arc.nasa.gov |
||
astro.spectra |
Spectra from the Sloan Digital Sky Survey Source: https://www.sdss.org/dr14/spectro/ |
||
astro.tess.data |
Light curves from TESS Data Alerts Source: https://archive.stsci.edu/prepds/tess-data-alerts/ |
||
astro.tess.simulated |
Simulated light curves from TESS End-to-End 6 Source: https://archive.stsci.edu/prepds/tess-data-alerts/ |
||
astro.voyager |
Data from the Voyager mission. Source: https://spdf.gsfc.nasa.gov/ |
||
Engineering | |||
engineering.la.traffic_counts |
Traffic Count data in Los Angeles. Source: https://data.lacity.org/A-Livable-and-Sustainable-City/LADOT-Traffic-Counts-Summary/94wu-3ps3 |
||
engineering.webcam.mit_sailing |
Images from webcams located at the MIT Sailing Pavilion Source: http://sailing.mit.edu/webcam.php |
||
Finance | |||
finance.timeseries |
Financial time series data retrieved using Alpha Vantage API. Source: https://www.alphavantage.co/ |
||
Geoscience | |||
geo.era_interim |
Era-Interim data at different pressure values from the European Centre for Medium-Range Weather Forecasts accessed through the University Corporation for Atmospheric Research. Source: https://rda.ucar.edu/datasets/ds627.0/ |
||
geo.gldas |
Land hydrology model produced by NASA. This version of the data is generated to match the GRACE temporal and spatial characteristics and is available as a complementary data product. Source: https://grace.jpl.nasa.gov/data/get-data/land-water-content |
||
geo.grace |
NASA GRACE Tellus Monthly Mass Grids. 30-day measurements of changes in Earth’s gravity field to quantify equivalent water thickness. Source: https://grace.jpl.nasa.gov/data/get-data/monthly-mass-grids-land |
||
geo.grace.mascon | NASA GRACE Tellus Monthly Mass Grids - Global Mascons. 30-day measurements of changes in Earth’s gravity field to quantify equivalent water thickness. Source: https://grace.jpl.nasa.gov/data/get-data/jpl_global_mascons | ||
geo.groundwater | United States groundwater monitoring wells measuring the depth to water level. Source: https://waterservices.usgs.gov | ||
geo.magnetometer | Data collected at magnetic observatories operated by the U.S. Geological Survey. Source: https://geomag.usgs.gov | ||
geo.mahali.rinex |
Rinex files from the MIT led NSF project studying the Earth’s ionosphere with GPS. Web: http://mahali.mit.edu |
||
geo.mahali.tec |
Total Electron Content from the MIT led NSF project studying the Earth’s ionosphere with GPS. Web:http://mahali.mit.edu |
||
geo.mahali.temperature |
Temperature data from the MIT led NSF project studying the Earth’s ionosphere with GPS. Web: http://mahali.mit.edu |
||
geo.modis |
Spectroradiometer aboard the NASA Terra and Aqua image satellites. Generates approximately daily images of the Earth’s surface. Source:https://modis.gsfc.nasa.gov |
||
geo.pbo |
EarthScope - Plate Boundary Observatory (PBO): Daily GPS displacement time series measurements throughout the United States. Source: http://www.unavco.org/projects/major-projects/pbo/pbo.html |
||
geo.sentinel_1 |
Sentinel-1 TOPSAR data from the European Space Agency retrieved from the Alaska Satellite Facility. Source:https://www.asf.alaska.edu/ |
||
geo.srtm |
Elevation data at a one arc second resolution from the Shuttle Radar Topography Mission (SRTMGL1). Source: https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003 |
||
geo.uavsar |
UAVSAR SLC data from JPL. Source: https://uavsar.jpl.nasa.gov/ |
||
geo.wyoming_sounding |
Sounding data from the University of Wyoming. Source: http://weather.uwyo.edu/upperair/sounding.html |
||
Planetary Science | |||
planetary.ode |
Mars planetary data from PDS Geosciences Node's Orbital Data Explorer. Source: http://pds-geosciences.wustl.edu/default.htm |
||
Solar Science | |||
solar.sdo |
Images from the Solar Dynamics Observatory. Source: https://sdo.gsfc.nasa.gov/ |
Install
pip install scikit-dataaccess
Documentation
- User Manual: /docs/skdaccess_manual.pdf
- Code documentation (Doxygen): /docs/skdaccess_doxygen.pdf
- Code visualization (treemap): /docs/skdaccess_treemap.png
- Code class diagrams: /docs/class_diagrams
Contributors
Project lead: Victor Pankratius (MIT)
Contributors: Cody M. Rude, Justin D. Li, David M. Blair, Michael G. Gowanlock, Guillaume Rongier, Victor Pankratius
New contributors welcome! Contact to contribute and add interface code for your own datasets
Acknowledgements
We acknowledge support from NASA AIST14-NNX15AG84G, NASA AIST16-80NSSC17K0125, NSF ACI-1442997, and NSF AGS-1343967.
Examples
Code examples (Jupyter notebooks) for all datasets listed above are available at: /skdaccess/examples