Python and Jupyter Client Extensions for the IncQuery Server

Requires IncQuery Server (IQS) to operate.

Builds

PyPI

Name	Package
incqueryserver-api-python-client
incqueryserver-jupyter

Conda

Name	Downloads	Version	Platforms	Build

Try it first

Click the binder shield above to spin up a deployment publicly hosted on MyBinder, with wired-in guest access to a public IQS demo instance and the OpenMBEE public MMS model repository.

Warning: The resulting notebook will not be persistent, it cannot be shared and it can be deleted anytime! This usage is recommended only for one-time, short-lived experiments!

Getting started

Assuming that you have cloned this repository to your computer at path ${this-git-repo}, the following instructions will help you get the Jupyter extensions for IQS running. As a main dependency, this involves the installation of the (automatically generated) Python-based IQS API client.

Prerequisites

For first-time users, we generally recommend Miniconda3/Anaconda and conda (see below), but it is possible to do without.

Installing Jupyter using Conda

1. Make sure Miniconda3/Anaconda is installed, along with Python 3.7+ and the conda package manager

2. Make sure Jupyter is installed:

    conda install jupyter
   

   See details at: jupyter.org

Installing Jupyter using pip

If you already have Python 3.7+, you can alternatively use the pip package manager to acquire Jupyter and install the rest of the packages.

    pip install jupyter

Installing the Python-based IQS API client

You must install the generated Python client package by executing ONE of the following options.

Release version from public repository

Builds are available on PyPI and on Conda forge.

The Conda-forge package is maintained in the incqueryserver-api-python-client feedstock repository

• Install from PyPI: pip install incqueryserver-api-python-client
• Install from Conda forge: conda install -c conda-forge incqueryserver-api-python-client

Note that these publicly available builds have been generated to support basic authentication. If your IQS instance uses another authentication method, you need to install the package build shipped with your IQS instance.

Version supplied with IQS instance

Locate the Python client library hosted at your IQS installation; e.g. the library accompanying the public IQS demo server is available at https://openmbee.incquery.io/client/ for installation; copy the link to the actual package archive (sdist) file. Install this sdist using pip. If you are using Anaconda, make sure to issue the below command line from your Anaconda console.

    pip install ${address-of-python-client-sdist}

Roll your own: generate Python client for your own IQS instance

1. Make sure you have OpenAPI Generator. We have verified that OpenAPI 3.3.4 works; OpenAPI 4+ has some issues. Just download the .jar from the Mvn Repository.

2. Locate the OpenAPI definition file (.yaml) shipped with your IQS instance. You will find the link in the "Swagger UI" tab of the IncQuery Server Web Console; e.g. the API definition of the public demo instance is available here.

3. Within the OpenAPI definition file, find a line that looks like version: 0.16.0; it will be found near the top, below info:. This version number is the API version of your IQS installation.

4. At the end of the OpenAPI definition file, append a few lines (continuing the section components: and then adding a new section security:) to specify the authentication method of your IQS instance. Be mindful of proper indentation. For instance, in case of basic authentication, you would add these lines:

     securitySchemes:
       basicAuth:
         type: http
         scheme: basic       
   
   security:
     - basicAuth: []
   

5. Generate a Python API into ${this-git-repo}/source-gen/incqueryserver-api-python-client using the following command line:

        java -jar ${path-to-OpenAPI/openapi-generator-cli-3.3.4.jar} generate -i ${path-to-API-definition-yaml} -g python -o ${this-git-repo}/source-gen/incqueryserver-api-python-client -DpackageVersion="${API-version-of-your-IQS-instance}" -DpackageUrl="https://incquery.io" -DpackageName=iqs_client -DprojectName=incqueryserver-api-python-client  
   

6. Ensure that the generated Python client implementation is available to notebooks by executing ONE of the following options:

   • For end users:

        pip install ${this-git-repo}/source/incqueryserver-jupyter
     

   • For IQS developers who repeatedly edit the .yaml definition file and regenerate the Python code: issue one of the following commands to perform a "developer" install directly from the generated source project; changes to the source will be reflected immediately to newly (re)started Python interpreters / Jupyter kernels (previously started kernels that already executed the import will keep the old content until restarted). Select the appropriate command line depending whether you use pip or conda.

        pip install -e ${this-git-repo}/source-gen/incqueryserver-api-python-client
        conda develop ${this-git-repo}/source-gen/incqueryserver-api-python-client
     

Note: the generated client will only include the Acquisition API in versions 0.16 and above.

Installing the Jupyter-specific client extensions

You must install the Jupyter support itself by executing ONE of the following options.

Release version from public repository

Builds are available on PyPI and on Conda-forge

The Conda-forge package is maintained in the incqueryserver-jupyter feedstock repository

• Install from PyPI: pip install incqueryserver-jupyter
• Install from Conda-forge: conda install -c conda-forge incqueryserver-jupyter

Install from source using Conda

Ensure that the non-generated Jupyter-specific client extensions are available to notebooks by executing ONE of the following options:

• For end users (in the Anaconda console environment):

  • Clone the feedstock repository where the package is maintained
  • Build a conda package from the client extensions sources, then install it:

    git clone https://github.com/conda-forge/incqueryserver-jupyter-feedstock.git
    conda build -c conda-forge ${feedstock-git-repo}/recipe
    conda install -c conda-forge --use-local incqueryserver-jupyter
    

  • Plase note that this will install the release version from PyPI

• For developers: issue the following command to perform a "developer" install directly from the source project; changes to the source will be reflected immediately to newly (re)started Python interpreters / Jupyter kernels (previously started kernels that already executed the import will keep the old content until restarted).

     conda develop ${this-git-repo}/source/incqueryserver-jupyter
  

Install from source using pip

Ensure that the non-generated Jupyter-specific client extensions are available to notebooks by executing ONE of the following options:

• For end users:

pip install ${this-git-repo}/source/incqueryserver-jupyter

• For developers: issue the following command to perform a "developer" install directly from the source project; changes to the source will be reflected immediately to newly (re)started Python interpreters / Jupyter kernels (previously started kernels that already executed the import will keep the old content until restarted).

pip install -e ${this-git-repo}/source/incqueryserver-jupyter

Note that since we rely on conda packaging by default, the dependencies may have to be installed manually in case of using pip.

Final steps

Whether you installed from source or from the public repository, it is a good idea to verify your installation. Also, you might need a few additional components if you want to run our demo notebooks.

Smoke test

Run test_iqs_client.py and test_iqs_jupyter.py as a quick smoke test to verify correct installation of the generated client package and the Jupyter extensions package, respectively.

    python ${this-git-repo}/test-scripts/test_iqs_client.py
    python ${this-git-repo}/test-scripts/test_iqs_jupyter.py

Additional dependencies

The demo notebook uses plotly and cufflinks to demonstrate possible applications of the client extensions package. It is not recommended to install cufflinks-py using conda, as conda-forge seems to host an obsolete version not compatible with the demo; simply issue pip install cufflinks from the Anaconda console instead.

The direct connection to the MMS server additionally requires the installation of mms-python-client; you might have missed this dependency if you installed the Jupyter extensions in development mode.

• CAUTION pin to mms-python-client==3.4.2.1 until Open-MBEE/mms-alfresco#346 is resolved, see #35

Running the notebook

Optional: set up environment variables first

Several functions and classes defined in client extension take optional arguments whose default values can be injected via environment variables. This allows the notebook itself to be much simpler, by omitting connection data etc.

Here is an example start-jupyter.cmd file you may wish to place into your notebook home, and use it to start jupyter with the right default values:

@echo off

set IQS_JUPYTER_default_IQS_address=127.0.0.1:47700/api
set IQS_JUPYTER_default_IQS_username=...
set IQS_JUPYTER_default_IQS_password=...

set IQS_JUPYTER_default_twc_workspace=4d6ce495-1273-452c-a548-36fcd922184e
set IQS_JUPYTER_default_twc_resource=34cc77c8-d3ef-40a6-9b91-65786117fe67
set IQS_JUPYTER_default_twc_branch=bd03a239-7836-4d4c-9bcb-eba73b001b1e
set IQS_JUPYTER_default_twc_revision=1

set IQS_JUPYTER_default_mms_org=6384a103-766c-46e0-830d-8a3b1f479479
set IQS_JUPYTER_default_mms_project=PROJECT-bef4f459-5d90-41fb-bc86-4f6d4ebd2dfd
set IQS_JUPYTER_default_mms_ref=master
set IQS_JUPYTER_default_mms_commit=560d3959-3912-434a-a914-8d039d3c9a06

set IQS_JUPYTER_default_twc_osmc_address=https://twc.demo.iqs.beta.incquerylabs.com:8111/osmc
set IQS_JUPYTER_default_twc_osmc_username=...
set IQS_JUPYTER_default_twc_osmc_password=...

jupyter notebook

Caution: beware of whitespace, make sure there is none before/after the =. A similar shell script can be used in case of *nix systems; a docker file might be another good way to provide environment variables.

Run the notebook

1. Designate a location as the notebook home, where notebook files will be stored. For example, use ${this-git-repo}/example-notebook-home.
2. (in the Anaconda console environment) start a Jupyter server from ${path-to-notebook-home}: jupyter notebook
3. Interact with Jupyter either via the newly opened browser window, or using the URL or token printed by the Jupyter server to its stdout. Create a new notebook or open an existing one within the notebook folder; see an example in ${this-git-repo}/example-notebook-home/iqs-demo-twc.ipynb (if the parent folder is designated as notebook home, you should already see this notebook as a starting point).
4. Within the notebook, get started by:

   1. Import the Jupyter-specific client library

        import iqs_jupyter
      

   2. Create a widget to specify an access point

        connector = iqs_jupyter.IQSConnectorWidget()
      

   3. Fill out the text field of the widget to specify the address to a running IQS server instance, then create the main client object iqs:

        iqs = connector.connect()
      

      • To skip manual form filling, it is possible to specify the initial content of the widget fields using a number of ways. First, IQSConnectorWidget has the optional parameters initial_address, initial_user, initial_password. Second, if such parameters are not given, results may be automatically filled from environment variables (see below). If such default values are known, it is possible to skip this step and the previous one by not displaying a widget at all:

            iqs = iqs_jupyter.connect()
        

   4. You can now interact with the IQS instance. For example, if your IQS installation is connected to MMS, browse for an MMS commit using

        commit_selector = iqs.jupyter_tools.mms_commit_selector_widget()
      

   5. Access the full JSON/RPC API of IQS in the form of (you may use TAB completion):

        iqs.${api-category-label}.${api-call}