Installation · Example of Usage · Development

ViREx-Bench is a framework for inference-time scaling on Vietnamese reasoning tasks. The project is designed to benchmark different prompting strategies under consistent evaluation settings and measure how much each strategy improves over a direct-answer baseline.

You can install ViREx-Bench using uv:

uv add virex-bench

or using pip:

pip install virex-bench

Verify the installation by running:

uv run virex-bench --version

# or simply
uv run vb --version

ViREx-Bench evaluates a model on a Vietnamese reasoning task using a prompting strategy and a decoding method. Models are served behind any OpenAI-compatible endpoint (e.g. vLLM, SGLang).

The example below evaluates a model on vietnamese-logical-reasoning, which is scored with an LLM-as-a-judge. The default judge model is deepseek/deepseek-v4-flash (you can override this via setting VIREX_BENCH_JUDGE_MODEL, e.g. export VIREX_BENCH_JUDGE_MODEL=deepseek/deepseek-v4-pro). To use the judge, export VIREX_BENCH_JUDGE_API_KEY with a valid API key.

# Evaluate Qwen3.5-4B with Tree-of-Thought (with Beam search) on the vietnamese-logical-reasoning task
uv run vb run \
  --model Qwen/Qwen3.5-4B \
  --task vietnamese-logical-reasoning \
  --strategy tot-beam \
  --backend hosted_vllm \
  --api-base https://your-endpoint/v1 \
  --api-key "$OPENAI_API_KEY" \
  --num-threads 32 \
  --output-dir results

import virex_bench as vb

# Init the model
lm = vb.get_model(
    "Qwen/Qwen3.5-4B",
    backend="hosted_vllm",
    api_base="https://your-endpoint/v1",
    api_key="<empty>",
)

# Get the task
task = vb.get_task("vietnamese-logical-reasoning")

# Select a strategy and decoding method
# Here we will use Tree-of-Thought with Beam search (tot-beam) as the strategy and single-pass decoding
strategy = task.get_strategy("tot-beam")
decoding = vb.get_decoding(name="single-pass", strategy=strategy)

report = vb.evaluate(
    task=task,
    lm=lm,
    strategy=strategy,
    decoding=decoding,
    num_threads=32,
    # max_examples=10,  # Uncomment this line for quick testing with a small number of examples
)
vb.save_report(report, "./results")

print(f"{report.metric}={report.score:.4f} over {report.num_evaluated_examples} examples")

Clone the repository and sync the environment with uv:

git clone https://github.com/minhnguyent546/ViREx-Bench.git
cd ViREx-Bench
uv sync  # creates the venv and installs the package with dev dependencies

Quality checks — prefix commands with uv run so they use the project environment:

uv run --no-sync ruff check             # lint
uv run --no-sync ruff format            # format
uv run --no-sync pytest                 # test suite
uv run --no-sync ruff check path/to/file.py   # scope lint/format to a single file
uv run --no-sync ruff format path/to/file.py

ViREx-Bench talks to models through any OpenAI-compatible endpoint. To serve a model locally with vLLM or SGLang, install the corresponding dependency group and use the provided launchers — see scripts/serving/README.md for hardware requirements and full instructions.

uv sync --group vllm-v0-23        # or: uv sync --group sglang-v0-5-14

This repository's source code and associated datasets are licensed under the GNU General Public License v3.0. See LICENSE for details.

If you find ViREx-Bench useful in your research, please consider citing:

@misc{nguyen2026virexbench,
    author={Minh-Thien Nguyen},
    title={{ViREx-Bench}: A Framework for Inference-Time Scaling on {Vietnamese} Reasoning Tasks},
    year={2026},
    howpublished={\url{https://github.com/minhnguyent546/ViREx-Bench}}
}