ckanext-timeseries
Timeseries data is streams of data that is indexed by timestamp, such as sensing data. Current default CKAN Datastore plugin only index data based on an auto-incremental integer with no support for timeseries data. Perceived that timeseries data is an important capability when working with sensor network, a new plugin that supports operations based on timestamp natively would be necessary. Thus, the purpose of this repository.
This plugin is fully compatible with current Datastore API <http://docs.ckan.org/en/latest/maintaining/datastore.html#the-datastore-api>, the endpoint is changed from 'datastore_*' to 'datastore_ts_*'. For example, for search query: send request to 'datastore_ts_search' endpoint. Original datastore plugin should still be enabled so that you can view the resources on CKAN web interface.
Changes
- If you are coming from version < v0.1.0, the schema has changed. A command has been created to upgrade the schema. Please run the following command:
`
paster --plugin=ckan timeseries -c <path to ini configuration file> upgrade-schema autogen_timestamp _autogen_timestamp
`
- From v1.0.0 the plugin name has changed from "ckanext-datastore_ts" to "ckanext-timeseries", please install ckanext-timeseries as bellow.
Requirements
Being developed under CKAN 2.6
Installation
To install ckanext-timeseries:
-
Activate your CKAN virtual environment, for example:
. /usr/lib/ckan/default/bin/activate
-
Install the ckanext-timeseries Python package into your virtual environment:
pip install ckanext-timeseries
-
Add
timeseries
to theckan.plugins
setting in your CKAN config file (by default the config file is located at/etc/ckan/default/production.ini
). -
Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:
sudo service apache2 reload
Config Settings
CKAN Timeseries uses configurations from Datastore plugin so make sure those are set. In brief:
sqlalchemy.url = postgresql://ckan_default:pass@localhost/ckan_default
ckan.datastore.write_url = postgresql://ckan_default:password@localhost/datastore_default
ckan.datastore.read_url = postgresql://datastore_default:password@localhost/datastore_default
CKAN Timeseries introduces a new configuration to set the maximum size of a resource table (as we are dealing with real time data). When a resource reaches this limit, it's table will be cleaned, the default 30% of the oldest data will be deleted. This percentage can be customized by user when creating a CKAN Timeseries resource. Please look at the wiki page for more detail.
ckan.timeseries.max_resource_size = 500
The number is observed in MB
Development Installation
To install ckanext-timeseries for development, activate your CKAN virtualenv and do:
git clone https://github.com/namgk/ckan-timeseries.git cd ckanext-timeseries python setup.py develop
Running the Tests
To run the tests, do:
nosetests --nologcapture --ckan --with-pylons=test-core.ini ckanext/timeseries/tests/test.. .py
To run the tests and produce a coverage report, first make sure you have
coverage installed in your virtualenv (pip install coverage
) then run:
nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.timeseries --cover-inclusive --cover-erase --cover-tests
Registering ckanext-timeseries on PyPI
ckanext-timeseries should be availabe on PyPI as https://pypi.python.org/pypi/ckanext-timeseries. If that link doesn't work, then you can register the project on PyPI for the first time by following these steps:
-
(First time only) Create a source distribution of the project:
python setup.py sdist
-
(First time only) Register the project:
python setup.py register
-
Upload the source distribution to PyPI:
python setup.py sdist upload
-
Tag the first release of the project on GitHub with the version number from the
setup.py
file. For example if the version number insetup.py
is 0.0.1 then do:git tag 0.0.1 git push --tags