A small library to reproduce the scores on numer.ai diagnistics dashboard.

Installation

pip install numereval

Structure

Numerai main tournament evaluation metrics

numereval.numereval.evaluate

A generic function to calculate basic per-era correlation stats with optional feature exposure and plotting.

Useful for evaluating custom validation split from training data without MMC metrics and correlation with example predictions.

from numereval.numereval import evaluate

evaluate(training_data, plot=True, feature_exposure=False)

Correlations plot	Returned metrics

numereval.numereval.diagnostics

To reproduce the scores on diagnostics dashboard locally with optional plotting of per-era correlations.

from numereval.numereval import diagnostics

validation_data = tournament_data[tournament_data.data_type == "validation"]

diagnostics(
    validation_data,
    plot=True,
    example_preds_loc="numerai_dataset_244\example_predictions.csv",
)

Validation plot	Returned metrics

Specific validation eras

specify a list of eras in the format eras = ["era121", "era122", "era209"]

validation_data = tournament_data[tournament_data.data_type == "validation"]

eras = validation_data.era.unique()[11:-2]

numereval.numereval.diagnostics(
    validation_data,
    plot=True,
    example_preds_loc="numerai_dataset_244\example_predictions.csv",
    eras=eras,
)

Validation plot	Returned metrics

Docs will be updated soon!