YoshiX

YoshiX (Yoshi EXperiment) is a library for collecting and processing experimental data such as benchmarks results. This is particularly suited for experiment who require to run an algorithm several time with different parameters combinations.

YoshiX is directly inspired by Unit Testing but provide also functionality to store, process and export heterogeneous data coming from algorithms and experiments.

Features

The slowly increasing list of features includes:

• Runs experiments in a replicable way.
• Modular experiment specification! Each experiment is a python file, YoshiX will automatically find every experiment in a folder and will run each one of them!
• Automatically iterate over the parameter space according your own generators (any generator can be used as a source, from the range function to a custom generator!).
• Collects the experiment results in a synthetic way (the YoshiX Egg).
• Export the eggs into different formats (CSV, JSON and many other to come).

Usage

Collecting Data

This is the basic structure for a Yoshi experiment.

class SmallExample(yoshix.YoshiExperiment):

    def setup(self):
        # Setup the "egg" and the experiment.
        self.setup_egg(("Data1", "Data2"))

    def single_run(self, params):
        # Run the actual experiment.

        print("Single Run {}".format(self.run_counter))
        # ... Do some computation...
        self.partial_egg["Data1"] = random.randint(0, 100)
        self.partial_egg["Data2"] = random.randint(0, 100)
        # ... Do some other thing...

    def after_run(self):
        # Clean up...

The setup method is used to initialize the experiment and, in particular the "egg" that will be used to collect the data.

The single_run method is where the actual experiment is computed. This method will be called several times with different parameters (see generators) collecting each run results in a table.

The after_run method is where the experiment can perform some kind of global clean up (e.g., deleting files) or processing and exporting the collected data.

Generators and Fixed Parameters

Yoshi's Generators are exactly like standard Python generators. They are linked to particulars parameters and are used by Yoshi in order to generate the input parameters. Any Python generator can be used as a generator.

In the following example we can see how to assign a fixed parameter and a variable parameters using a generator. In the example the Min parameter will be 0 for every experiment run. Otherwise the Max parameters use the range builtin generator in order to variate the Max parameter from 1 to 10.

def setup(self):
    self.setup_egg(("Min", "Max", "A", "B"))
    self.assign_fixed_parameter("Min", 0)
    self.assign_generators("Max", range(1, 10))

Parameter Transformer

If some parameter contains complex object it is possible to assign a transformer function to the parameter in order to output a string representation of the object. You can imagine this as an external __str__() version for the parameter data.

def setup(self):
    self.setup_egg(("Map", "Start", "End", "ExpandedNodes"))
    self.assign_generators("Map", benchmark.maps_loader())
    self.assign_transformer("Map", lambda x: x.get_filename())
    ...

In the example, maps_loader returns a list of parsed representations of some experimental maps. This is handy, because we can use the params field of the single_run to access the already parsed map. However, in the output egg we want a simpler representation of the map, e.g., the filename. So we attach a transformer function to the Map parameter to extract the filename from the map before it is put in the result egg.

Private Generators

Sometimes we may want to use generators but we do not want to put them in the egg. We call this private generators. To use a private generator we can simply use the private flag in the generator setup.

self.assign_generators("Private Data", range(1,10), private=True)

Private generators key are not specified in the egg setup (because they will not be written on the egg). Private generators can be used as standard params in the single_run body.

print("Accessing Private Data with Value {}".format(params["Private Data"]))

Exporting

Once the experiment is completed and the egg is ready we can decide to export the egg in some other format. For instance we can decide to export the egg in CSV.

def after_run(self):
    YoshiEggCSVExporter(self.egg, "test.csv").export()
    print("Bye Bye")

In this example we use the YoshiEggCSVExporter exporter in order to export the egg into a csv file.

Command line usage

Once you have a set of experiments in a folder (for instance, "examples") you can use

python -m yoshix.run_yoshi .\examples

in order to run all the experiments in the folder.