RemoteJob

Run python3 functions in remote containers.

This is usefull to work locally but distribute part of the work in docker clusters.

pip install remotejob

Example

import remotejob

import mymodule
from mymodule import myfunction

async def myfunction_in_docker():
    with remotejob.RemoteJob() as runner:
        try:
            await runner.pip_install(['numpy', 'sklearn'])
            await runner.send_module_recursive([mymodule])

            res = await runner.run(myfunction, *args, **options)

remotejob.RemoteJob connect itself to the local docker deamon by default but you can specify a remote cluster.

For example: remotejob.RemoteJob('user@mycluster.com', 'mycluster.com')

 API

class RemoteJob

__init__(base_url='unix://var/run/docker.sock', host_name='0.0.0.0', port=None, **clientArgs)

Connect to a remote docker server and create a slave container. You should use it with the with statement or ensure RemoteJob.close is called to free the resources on the docker server.

Args:

• base_url: str URL to the Docker server. see docker.DockerClient.
• host_name: str Adress from which the container will be accessible.
• port: str An unused and accessible port of the host. By default affect a port by RemoteJob instance from 8100.
• **clientArgs: Options passed to docker.DockerClient.

async RemoteJob.pip_install(*args)

Util which run python -m pip install on the remote container

Args:

• *args: Arguments passed to 'python -m pip install'

async RemoteJob.send_module_recursive(*args)

Util which copy a module sources on the remote container.

Args:

• *args: List of python module to copy in the runner

async RemoteJob.run(fn, *args, **arg_dict)

Run a function in the remote container and return the result. The function source must be available on the remote container (native, pip installed, or in a module passed to send_module_recursive). All arguments and the returned value must be serializable by dill a pickle extension.

Args:

• fn: callable Function to be executed remotely
• *args: Passed to fn
• **arg_dict: Passed fo fn

Build

Docker image

docker build . -t remotejob