DS2Sim

A simple Space Sim to experiment with machine learning.

It wraps Horde3D with Cython and uses it to render scenes like these.

You can use it to create your own training data, but it is easier to use the data set from the DS2Data project. Not only does that project have a representative training set of the relevant objects, it also comes with a pre-rendered flight path to test your model.

Once you have your model you can plug it into the viewer application (see View The Space Simulation section for details), and fly through the scene while it identifies the objects. You may also replace the manual controls with another AI to make it fly autonomously.

Installation

First, this will only work with an NVidia GPU, because the project uses headless rendering. If you do not have an NVidia card, you may still train and test ML models with the data from the DS2Data set.

Next, since this is a Cython wrapper for Horde3D, you will need - you guessed it - Cython and Horde3D. To compile and install Horde3D, activate the virtual environment of your choice, install Cython, and then clone/compile/install Horde3D like so:

git clone https://github.com/olitheolix/Horde3D
mkdir -p Horde3D/build
cd Horde3D/build
git checkout ds2sim
cmake .. -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=`python -c 'import sys; print(sys.prefix)'`
make install

Afterwards, install DS2Sim via PIP:

pip install ds2sim

View the Space Simulation

This consists of two parts: a web server to supply the rendered images, and a Qt application to display them.

Start the web server and load the default scene:

ds2server --default-scene

Then put the following code into a file. Note that almost the entire file is boilerplate for Qt.

import sys
import ds2sim.viewer
import numpy as np

import PyQt5.QtGui as QtGui
import PyQt5.QtCore as QtCore
import PyQt5.QtWidgets as QtWidgets


# For convenience.
QPen, QColor, QRectF = QtGui.QPen, QtGui.QColor, QtCore.QRectF
DS2Text = ds2sim.viewer.DS2Text

class MyClassifier(ds2sim.viewer.ClassifierCamera):
    def classifyImage(self, img):
        pass

# Qt boilerplate to start the application.
app = QtWidgets.QApplication(sys.argv)
widget = MyClassifier('Camera', host='127.0.0.1', port=9095)
widget.show()
app.exec_()

The only part here that is not boilerplate is MyClassifier, and even that does nothing right now.

When you run this program you should see the scene. Click in it, and use the ESDF keys, as well as the mouse, to fly through the scene.

Plug Your ML Model Into The Simulation

The real fun is, of course, to use ML to find and identify all the cubes while you fly around. To do so, overload the classifyImage method in the previous demo like so:

# For convenience.
QPen, QColor, QRectF = QtGui.QPen, QtGui.QColor, QtCore.QRectF
DS2Text = ds2server.viewer.DS2Text

class MyClassifier(ds2sim.viewer.ClassifierCamera):
    def classifyImage(self, img):
        # `img` is always a <height, width, 3> NumPy image.
        assert img.dtype == np.uint8

        # Pass the image to your ML model.
        # myAwesomeClassifier(img)

        # Define a red bounding box.
        x, y, width, height = 0.3, 0.4, 0.3, 0.3
        bbox = [QPen(QColor(255, 0, 0)), QRectF(x, y, width, height)]

        # Define a green text label.
        x, y = 0.3, 0.4
        text = [QPen(QColor(100, 200, 0)), DS2Text(x, y, 'Found Something')]

        # Install the overlays.
        self.setMLOverlays([bbox, text])

The classifyImage method will be called for each frame. It always receives one RGB image as a NumPy array. Pass that image to your classifier to identify the objects in the scene.

When you have found out which objects are where, you can add overlays to highlight them. Every overlay is a 2-tuple: a QPen to define the colour, and a primitive to draw. Currently, QRect and DS2Textures are the only supported primitives. Pass all overlays to setMLOverlays and it will show up in the scene.

The example code from above will produce an output like this.

Single Frame	Spaceflight