This repository is the code and supplementary material for the paper:
Automated Standardization of Legacy Biomedical Metadata Using an Ontology-Constrained LLM Agent. Josef Hardi, Martin J. O'Connor, Marcos Martínez-Romero, Jean G. Rosario, Stephen A. Fisher, Mark A. Musen. arXiv: https://arxiv.org/abs/2604.08552
ARMS is an LLM agent that standardizes legacy biomedical metadata records into the CEDAR template format. Instead of treating ontology constraints as static text in a prompt, the agent calls external services at inference time — fetching the live CEDAR template and querying BioPortal for canonical ontology terms — through Model Context Protocol (MCP) tools. This repository contains the agent, the evaluation framework, and the data used to produce every number and figure in the paper.
The agent is a standalone package under arms-agent/, published to PyPI as arms-agent and installable on its own. The repository root is the evaluation harness that measures it. See its README.md to use the agent outside this repository.
| Component | Location |
|---|---|
| Agent graph (ReAct, LangGraph) | arms-agent/src/arms_agent/agent.py |
The three MCP tools (get_cedar_template, term_search_from_ontology, term_search_from_branch) |
arms-agent/src/arms_agent/tools.py |
| ARMS system prompt | arms-agent/src/arms_agent/prompts.py |
| Baseline system prompts | evaluation/conditions/prompt_only/prompts/baseline.py |
| ARMS agent | evaluation/conditions/agent_tool/arms.py |
| Baseline agent | evaluation/conditions/prompt_only/baseline.py |
| Component | Location |
|---|---|
| Expert-curated gold standard | data/<assay>/gold/ |
| Legacy input records | data/<assay>/input/ |
| Baseline output | data/<assay>/output/<model>/baseline/ |
| ARMS output | data/<assay>/output/<model>/arms-agent/ |
| CEDAR template specifications (one per assay) | data/schemas/<assay>.json |
| Sampling function (stratified, per-assay random sample) | data/sampling.py |
The evaluation set is 839 records across 12 assay types, sampled independently within each assay (up to 100 per assay; assays with fewer curated records included in full). See data/sampling.py for the exact procedure.
| What it produces | Location |
|---|---|
| Exact-match accuracy metrics; per-field results | evaluation/analysis/metrics/ |
| Per-assay and overall accuracy tables | evaluation/analysis/data_analysis/ |
| Confidence intervals and statistical tests | evaluation/analysis/significance/ |
| Result plots | evaluation/plots/ |
| End-to-end analysis notebook | experiment.ipynb |
All analysis runs on the prediction files already in the data folder. No LLM API calls are needed to reproduce the accuracy numbers, confidence intervals, significance tests, or error breakdowns.
Analysis code is available in the experiment.ipynb notebook.
To regenerate predictions (this calls the OpenAI, CEDAR, and BioPortal APIs), create a .env file with OPENAI_API_KEY, CEDAR_API_KEY, and BIOPORTAL_API_KEY, then:
uv run python -m evaluation \
--input data/atacseq/input \
--target-schema https://repo.metadatacenter.org/templates/dd5e8653-81cf-470b-b71b-15cab421bb84 \
--output data/atacseq/output/gpt5mini \
--model gpt-5-mini --concurrent 8 --agent-tool arms-agentRuns can be traced to Langfuse to inspect each LLM call, MCP tool call, and agent step. Add LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_HOST to .env; tracing activates only when both keys are set and is otherwise a no-op. Set LANGFUSE_TRACING_ENVIRONMENT (or pass --langfuse-environment to the evaluation CLI) to separate sweeps from each other within a project. See .env.example.
This is a uv workspace: the root holds the evaluation harness, and arms-agent/ holds the published package. One uv sync --all-extras sets up both, and the agent is installed in editable mode, so edits under arms-agent/src/ take effect at once.
uv run python -m pytest # tests
uv run ruff check arms-agent/ tests/ evaluation/ # lint
uv run ruff format arms-agent/ tests/ evaluation/ # formatBSD 2-Clause License.
