A low-RAM LLM gateway for coding agents. One static binary (~6.7 MiB, ~9 MiB RAM idle) that gives every OpenAI/Anthropic-compatible CLI — opencode, Claude Code, Codex, Cursor, … — automatic provider failover, RTK token compression (≥90% on tool-heavy traffic), response caching, and a per-agent token/cost ledger.

npm install -g routre-cli     # one command, no Go toolchain needed
routre-cli setup              # wizard: provider URLs + API keys
routre-cli serve              # gateway on 127.0.0.1:20128
routre-cli start              # start the daemon (systemd/launchd or detached)
routre-cli list               # connected providers + token/cost ledger

Point any agent at http://127.0.0.1:20128 via OPENAI_BASE_URL / ANTHROPIC_BASE_URL — failover, compression, and caching come for free.

• Why this exists
• Install
• Quick start
• How it works
  • Automatic failover
  • RTK token compression
  • Response cache
  • Cross-kind streaming translation (OpenAI ↔ Anthropic)
  • Token & cost ledger
  • Always-on daemon
• Configuration
• Commands
• Benchmarks
• Changelog (separate file: CHANGELOG.md)
• Project layout
• Known gaps
• License

Built as a decision-driven spike (see SPEC.md for the full decision record and roadmap):

• Not 9router — right feature set, but Node/Next.js with ~80 MB idle RAM and a documented unbounded leak (~4.8 GB in 3 days), plus unbenchmarked 20–65% savings claims.
• Not LiteLLM/Portkey self-host — Python/Postgres/Redis stacks sized in gigabytes.
• Single static Go binary, stdlib only — proven low-RAM pattern (Go ~5K QPS at ~11 ms proxy overhead), cross-compiled for 6 platforms (7 npm packages, incl. scoped win32).
• Honest metrics — the 90% claim is defined, gated, and reproducible: routre-cli bench fails the build if it regresses.

npm install -g routre-cli
routre-cli version

The npm package ships one static Go binary per platform (via optional dependencies) plus a tiny Node launcher. No Go toolchain, no runtime dependencies.

The Windows packages are scoped (@mariobgsp/…) because npm's spam detection rejects the unscoped routre-cli-win32-* names.

make build          # needs Go ≥ 1.22
./routre-cli version

make dist-npm       # cross-compiles all 6 platforms → npm/dist/*.tgz (7 packages)
NPM_TOKEN=<token> bash ./npm/publish.sh   # optional: manual publish to the registry

There is no CI publish workflow — npm releases are tag-driven only: cut a version, bump npm/routre-cli/package.json, tag vX.Y.Z. The tarballs in npm/dist/ are built locally (make dist-npm). Publish order matters if you do push to the registry: platform packages first (Unix, then scoped Windows), launcher last — the launcher's optionalDependencies reference the platform packages.

routre-cli setup        # interactive: listen addr, providers, URLs, keys, prices
routre-cli check        # validate config + which API keys are set
routre-cli serve        # gateway on 127.0.0.1:20128
routre-cli start --autostart  # start daemon + enable boot/login auto-start
routre-cli stop         # stop the daemon
routre-cli list         # providers + token/cost ledger

setup writes two files next to config.json:

• config.json — providers, tiers, base URLs, models (no secrets)
• routre-cli.env — API keys, 0600 permissions, auto-loaded by serve / check / list (no shell exports needed)

export ANTHROPIC_BASE_URL=http://127.0.0.1:20128   # Claude Code
export OPENAI_BASE_URL=http://127.0.0.1:20128      # Codex / opencode / etc.

Endpoints: POST /v1/chat/completions, POST /v1/responses, POST /v1/messages, GET /v1/models, GET /v1/status, GET /v1/usage, GET /healthz, GET /metrics (Prometheus).

/v1/responses speaks the OpenAI Responses API (what opencode's built-in openai provider uses) and is translated to /v1/chat/completions for the upstream providers, then wrapped back into the Responses envelope for the client. It works with OPENAI_BASE_URL out of the box.

The repo ships config.all.json — a ready config exposing 506 models through one endpoint, verified live:

cp config.all.json config.json
# routre-cli.env:
#   OPENCODE_GO_API_KEY=<from ~/.local/share/opencode/auth.json>
#   OPENROUTER_API_KEY=<your key>
routre-cli serve
curl http://127.0.0.1:20128/v1/models          # 506 models
# use any model as <provider>/<model>, e.g.:
#   opencode-zen/claude-fable-5  opencode-go/hy3  openrouter/deepseek/deepseek-chat

Provider-qualified model names. The <provider>/ prefix is a client-side routing label only — it tells the gateway which configured provider to route to — and is stripped before the request is sent upstream. The upstream always receives the bare listed model name: opencode-go/gpt-5.6-luna → gpt-5.6-luna upstream, openrouter/openai/gpt-5.6-luna → openai/gpt-5.6-luna (multi-slash IDs are kept intact — only the first segment is removed). This matches how opencode itself resolves provider/model (it splits the reference at the first /).

Tier order: opencode-go → opencode-zen (subscription), openrouter (fallback). If a model is missing or a provider 5xx/401s, the gateway fails over automatically.

./cmd/mock-upstream/mock-upstream -addr 127.0.0.1:19999   # mock provider
# config with base_url "http://127.0.0.1:19999/v1"
MOCK_KEY=x ./routre-cli serve -config config-mock.json
opencode run --model <provider>/<model> "hello"

• Providers are configured in tiers (subscription → cheap → free) and tried in order; within a tier, providers are tried in order.
• Failures (5xx, 429, 401/403, network errors) fail over to the next provider; the failed one enters an exponential cooldown (2 s base → 30 min cap). Success resets. Cooldowns are per provider — one failing provider never cools down the others.
• Transient blips are retried first: a network error or 5xx is retried once on the same provider (500 ms delay) before failover — an hour-long upstream 503 no longer burns every fallback in the same window.
• Auth rotation is recovered: on a 401/403 the gateway re-reads the routre-cli.env key file and, if the API key changed, retries the same provider once with the fresh key before failing over.
• Upstream Retry-After is honored: a 429/5xx carrying a Retry-After header sets that provider's cooldown to at least the mandated delay (it acts as a floor, never shortening the default backoff).
• Streaming requests fail over too: an upstream 5xx/429 answered before the first stream byte is treated like a non-streaming failure; after the first byte, a stream abort stops the request (no duplicated output).
• Client-caused errors (400/404/422, e.g. context-length) are surfaced, not retried.
• Honest error identity: model_not_found (503) only when no configured provider (and no fallback) can serve the model; when every provider that could serve it is cooling down, the gateway returns providers_unavailable (503) with a Retry-After header instead — the remedy is waiting, not editing the config.
• Zero-config model handling (forward_unknown: true, default): a model absent from every provider's models whitelist is forwarded verbatim to available providers in tier order. A provider that does not carry the model rejects it (400/404); that rejection is treated as "try the next provider" (with no pointless same-provider retry), so a model carried by any configured provider works with no config edit. If every provider rejects it, the last rejection is surfaced. Set forward_unknown: false to restore strict whitelist behavior (unknown models return model_not_found).
• The gateway holds the provider API keys (from api_key_env / routre-cli.env) and injects them upstream — a client's Authorization header is a placeholder and is never forwarded.

Heuristic compression of tool_result content — no local LM, no network calls:

Safety contract: fail-open (malformed JSON passes through), never grows a payload, 500 B–10 MiB window, per-request safe. The bench command measures reduction on 5 realistic tool-heavy payloads and gates both the aggregate (91.5%) and the worst per-payload (90.3%) at ≥90%.

• Exact-match LRU keyed by SHA-256 of the processed body (post-RTK). Defaults: 512 entries / 1 h TTL / 8 MiB max entry; the shipped config.all.json uses 4096 entries / 24 h TTL / 64 MiB budget.
• Non-streaming responses only; streamed responses are never cached.
• Optional prefix_order moves system messages first for stable keys and stable upstream prompt-cache prefixes.
• Optional prompt_cache (Anthropic outbound only) injects cache_control {type:"ephemeral"} breakpoints on the system prefix and last message, so repeat agentic prefixes are billed at the cache-read rate. Off by default; strictly additive (an existing cache_control is never overwritten).
• Cache hits record their token savings in the ledger — credited with the upstream-reported prompt token count stored on the cached response, so the ledger matches the provider's billing numbers instead of length-based estimates.

An agent pinned to the OpenAI dialect can stream from an Anthropic provider and vice versa: when the client and upstream speak different API dialects, the gateway rewrites the event stream in flight.

• In-flight SSE state machine — never buffers the whole response. Frames are translated as they arrive and flushed immediately (no tail-latency cost), keeping memory flat for long streams.
• Tool-call fidelity: tool_use_id ↔ tool_call_id round-trips unchanged (no gateway-generated ids), so agent tool loops work across dialects. Partial JSON tool arguments (input_json_delta / tool_calls[].arguments) pass through verbatim; the client accumulates them.
• Honest termination: Anthropic message_delta/error → OpenAI finish_reason (max_tokens→length, end_turn→stop, error→content_filter) and the reverse; every stream ends with [DONE].
• Failover contract preserved: strictly retryable before the first byte reaches the client; after the first byte the stream can't fail over (no duplicated output) — same rule as same-kind streaming.
• Usage tokens are captured from the stream for both dialects, so the ledger records real completion counts instead of zeros.

Non-streaming cross-kind requests fall back to lossy translation (flat tool output, no tool_call_id link) for cheap-tier fallback — fine for one-shot prompts, but streaming is the preferred path for cross-kind tool loops.

Per coding agent (by User-Agent): requests, tokens in/out, RTK savings, cache savings, and estimated cost.

• Persisted to ~/.routre-cli/usage.json — survives restarts, autosaved every 60 s and on SIGHUP, so a crash loses at most one minute of ledger. Works offline from the persisted file when the gateway is down.
• Costs come from provider-reported usage (OpenRouter reports real usage.cost) or from price_in / price_out in the config (USD per 1M tokens).
• Tail per-request detail with routre-cli logs, see the ledger with routre-cli list.

GET /metrics serves Prometheus exposition text — useful for dashboards and uptime checks. It reports: uptime seconds, request totals by client/provider/model/outcome class, upstream failover totals by provider/class, cache hits/misses and the hit ratio, RTK compression applied count and saved tokens, and provider-reported prompt-cache read tokens. The per-request JSONL log (request_log in config, tailed with routre-cli logs) and the /v1/status + /v1/usage JSON endpoints cover the structured detail.

• deploy/routre-cli.service + deploy/routre-cli.socket (systemd; socket activation → ~0 MB idle) and deploy/dev.routrecli.daemon.plist (launchd for macOS). MemoryMax guard included.
• SIGHUP reloads config + env without dropping connections (SIGINT / SIGTERM = graceful shutdown, ledger saved first).
• routre-cli start [--autostart], stop [--autostart], and restart manage the daemon through systemd (system or --user scope) or launchd; without an installed service they fall back to a detached background process logging to ~/.routre-cli/daemon.log.

The gateway binds 127.0.0.1 by default, so it is only reachable from the local machine — but any local process could still send requests through it and burn your provider keys. For shared machines or extra hardening you can enable a shared secret:

"auth": { "secret_env": "ROUTRE_SECRET", "header": "X-Routre-Key" }

With auth.secret_env set, every /v1/* request must carry the matching secret in the configured header (or Authorization: Bearer <secret>); mismatches get a 401 invalid_api_key with no upstream call. /healthz and /metrics stay open for probes/scrapers. The secret lives in routre-cli.env (0600), never in the config.

routre-cli setup offers to enable this and generates a random secret. When enabled, routre-cli serve also mints a one-time process token (~/.routre-cli/auth.tok, 0600, regenerated each start) so the local list/check/logs commands keep working without you pasting the secret into flags.

{
  "listen": "127.0.0.1:20128",
  "rtk":   { "enabled": true, "min_bytes": 500, "max_bytes": 10485760 },
  "cache": { "enabled": true, "max_entries": 512, "ttl_seconds": 3600, "prefix_order": false },
  "tiers": [
    { "name": "subscription", "providers": [
      { "name": "openrouter", "kind": "openai",
        "base_url": "https://openrouter.ai/api/v1",
        "api_key_env": "OPENROUTER_API_KEY",
        "models": ["tencent/hy3"],
        "price_in": 0, "price_out": 0 }   // USD per 1M tokens; 0 = unknown
    ]}
  ]
}

A full reference config with 506 models lives in config.all.json; a minimal template is config.example.json.

== configured providers ==
  [subscription] openrouter  openai  key ok  models=tencent/hy3,... cost n/a

== live gateway ==
  openrouter     up

== token & cost ledger ==
  source: live

  codex
    requests: 2
    consumed: 58 tokens (18 in + 40 out)
    saved:    3351 tokens (rtk 2308 + cache 1043)
    cost:     n/a (no prices configured)   saved: n/a
    by provider/model:
      codex/tencent/hy3  2 req  58 tok  saved 3351

  opencode
    requests: 4
    consumed: 112 tokens (62 in + 50 out)
    saved:    28 tokens (rtk 0 + cache 28)
    cost:     $0.000021   saved: $0.000000

  TOTAL
    requests: 6
    consumed: 170 tokens   saved: 3379 tokens (95.2%)
    cost:     $0.000021   saved: $0.000000

Measured on this machine, 2026-08-15:

Reproduce:

make build test bench        # bench gates 90% (fails on regression)
./scripts/measure-ram.sh ./routre-cli ./config.example.json 30

main.go                  CLI (setup/serve/check/start/stop/restart/list/bench/version)
bench.go                 RTK benchmark + 90% gate
setup.go                 interactive setup wizard
start.go                 daemon start/restart (systemd/launchd/detached spawn)
stop.go                  daemon stop (systemd/launchd/port scan + SIGTERM)
list.go                  providers + per-agent token/cost ledger
internal/config/         JSON config + routre-cli.env + SIGHUP reload
internal/router/         tiers, failover, cooldowns (exponential backoff)
internal/rtk/            token compression (12 filters + autodetect)
internal/cache/          exact-match LRU + prefix ordering
internal/proxy/          HTTP gateway, SSE relay, key injection, translation
internal/usage/          token/cost ledger (persisted to ~/.routre-cli/)
internal/tokenize/       token estimator (benchmark instrument)
internal/mock/           mock upstream (tests + keyless e2e)
benchdata/               tool-heavy request bodies for the bench gate
scripts/measure-ram.sh   RSS/peak/growth measurement
deploy/                  systemd unit+socket, launchd plist
npm/                     npm distribution (7 packages: 4 Unix + 2 scoped win32 + launcher)

• Gemini is a streaming dialect for OpenAI↔Gemini (request + non-streaming response + in-flight SSE translation with [DONE] termination); the Anthropic↔Gemini pair is not yet implemented (a gemini-kind provider served to an Anthropic-dialect client is rejected, not mis-answered).
• Token estimates are an approximation (≈4 bytes/token) — a benchmark instrument, not billing-grade (tiktoken integration is planned).
• 90% is measured on tool-result tokens; output tokens are never compressed, so real-session savings depend on the tool-traffic mix (this is exactly what routre-cli list shows you).
• 401/403 token refresh is implemented (re-reads the env key file and retries once on rotation); cooldown + failover still apply when the key is unchanged or still rejected.

See CHANGELOG.md for the release history (all versions, including unreleased changes).

MIT — see LICENSE. (The RTK filter approach is a clean-room reimplementation of the MIT-licensed 9router open-sse/rtk ideas.)

See SPEC.md for the full decision record, metric definitions, failover policy table, and validation roadmap.