Synthetic Data - Quality Assurance 🔎

Documentation | Sample Reports | Technical White Paper

Assess the fidelity and novelty of synthetic samples with respect to original samples:

calculate a rich set of accuracy, similarity and distance metrics
visualize statistics for easy comparison to training and holdout samples
generate a standalone, easy-to-share, easy-to-read HTML summary report

...all with a few lines of Python code 💥.

Installation

The latest release of mostlyai-qa can be installed via pip:

pip install -U mostlyai-qa

The latest development version can be installed directly from GitHub:

pip install -U git+https://github.com/mostly-ai/mostlyai-qa.git@main

Quick Start

import pandas as pd
import webbrowser
from mostlyai import qa

# fetch original + synthetic data
base_url = "https://github.com/mostly-ai/mostlyai-qa/raw/refs/heads/main/examples/quick-start"
syn = pd.read_csv(f"{base_url}/census2k-syn_mostly.csv.gz")
# syn = pd.read_csv(f'{base_url}/census2k-syn_flip30.csv.gz') # a 30% perturbation of trn
trn = pd.read_csv(f"{base_url}/census2k-trn.csv.gz")
hol = pd.read_csv(f"{base_url}/census2k-hol.csv.gz")

# runs for ~30secs
report_path, metrics = qa.report(
    syn_tgt_data=syn,
    trn_tgt_data=trn,
    hol_tgt_data=hol,
)

# pretty print metrics
print(metrics.model_dump_json(indent=4))

# open up HTML report in new browser window
webbrowser.open(f"file://{report_path.absolute()}")

Basic Usage

from mostlyai import qa

# analyze single-table data
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
)

# analyze sequential data
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
    tgt_context_key = "user_id",
)

# analyze sequential data with context
report_path, metrics = qa.report(
    syn_tgt_data = synthetic_df,
    trn_tgt_data = training_df,
    hol_tgt_data = holdout_df,  # optional
    syn_ctx_data = synthetic_context_df,
    trn_ctx_data = training_context_df,
    hol_ctx_data = holdout_context_df,  # optional
    ctx_primary_key = "id",
    tgt_context_key = "user_id",
)

Sample Reports

Baseball Players (Flat Data)
Baseball Seasons (Sequential Data)

Citation

Please consider citing our project if you find it useful:

@software{mostlyai-qa,
    author = {{MOSTLY AI}},
    title = {{MOSTLY AI - Quality Assurance}},
    url = {https://github.com/mostly-ai/mostlyai-qa},
    year = {2024}
}