Failover

OpenCode plugin for automatic API-key failover and rotation across multiple provider keys.

opencode-failover is a OpenCode plugin that automatically rotates API keys across multiple provider credentials. When a key hits a rate limit, the plugin quarantines it and switches to the next available key -- zero downtime, zero manual intervention.

• Multiple keys per provider with weighted round-robin distribution
• Automatic quarantine on rate-limit (429) with exponential backoff
• Permanent disable on auth failure (401/403)
• Works with NVIDIA NIM, OpenRouter, Anthropic, OpenAI, and any OpenCode-compatible provider

Quick Start

opencode plugin opencode-failover

Set your API keys. Ask the LLM in the TUI (natural language):

Add these NVIDIA API keys for failover rotation: nvapi-xxx, nvapi-yyy, nvapi-zzz

The plugin saves keys to .env and activates automatically.

Or create .env manually:

NVIDIA_API_KEYS="nvapi-xxx,nvapi-yyy,nvapi-zzz"

Restart OpenCode. The plugin activates automatically and begins rotating keys.

Quick Prompt (copy & paste)

Open the opencode TUI and send this to the LLM:

Add these NVIDIA API keys for failover rotation: nvapi-key1, nvapi-key2, nvapi-key3

The plugin saves them to .env. Works immediately in the current session.

To check key status later, ask:

Show me the keychain status

To remove keys:

Remove all NVIDIA API keys from the keychain

Or remove a specific key:

Remove nvapi-key1 from NVIDIA

Most LLM providers enforce per-key rate limits. When you hit the limit, requests fail and you are stuck waiting.

opencode-failover solves this by:

• Rotating to the next available key on rate-limit (429)
• Quarantining exhausted keys with exponential backoff (60s to 300s)
• Disabling permanently on auth failure (401/403)
• Recovering quarantined keys automatically when their timer expires

One provider, multiple keys, zero downtime.

Features

• Weighted round-robin key rotation
• Exponential backoff quarantine (60s, 120s, 240s, 300s cap)
• retry-after header respect (milliseconds, seconds, HTTP-date formats)
• Permanent disable on auth errors (401/403) and billing errors (402)
• Temporary quarantine on server errors (5xx)
• Rate-limit pattern detection (Anthropic, OpenAI, and generic patterns)
• keychain-status tool for real-time key monitoring
• /keychain slash command for live key status in the TUI
• Debug logging via OPENCODE_FAILOVER_DEBUG=1
• Works with any OpenCode-compatible provider

Installation

Option 1: OpenCode CLI (recommended)

opencode plugin opencode-failover

This installs the package and updates your opencode.json automatically.

Option 2: npm

npm install opencode-failover

Then add to your opencode.json:

{
  "plugin": ["opencode-failover"]
}

Option 3: Local development

git clone https://github.com/bulutmuf/opencode-failover.git
cd opencode-failover
bun install

Then configure OpenCode to load it from the source directory. Add to your opencode.json:

{
  "plugin": ["/path/to/opencode-failover/src/index.ts"]
}

Configuration

Environment variables

Single provider (comma-separated keys):

export NVIDIA_API_KEYS="nvapi-key1,nvapi-key2,nvapi-key3"

Multiple providers (JSON):

export OPENCODE_FAILOVER_PROVIDERS='{
  "nvidia": {
    "keys": ["nvapi-key1", "nvapi-key2", "nvapi-key3"],
    "scheme": "Bearer"
  },
  "openrouter": {
    "keys": ["sk-or-key1", "sk-or-key2"],
    "header": "Authorization"
  }
}'

opencode.json options

{
  "plugin": [
    ["opencode-failover", {
      "providers": {
        "nvidia": {
          "keys": ["nvapi-key1", "nvapi-key2", "nvapi-key3"],
          "scheme": "Bearer"
        },
        "openrouter": {
          "keys": ["sk-or-key1", "sk-or-key2"],
          "header": "Authorization"
        }
      }
    }]
  ]
}

Config options

Option	Type	Default	Description
keys	string[]	required	API keys for this provider
header	string	"Authorization"	HTTP header to inject
scheme	string	"Bearer"	Header value prefix
weight	Record<string, number>	{}	Per-key rotation weights

Precedence

1. OPENCODE_FAILOVER_PROVIDERS env (full JSON) overrides everything
2. OPENCODE_FAILOVER_KEYS env (JSON key map) overrides individual env vars
3. <PROVIDER>_API_KEYS env + opencode.json options (merged)
4. opencode.json options only

How It Works

Request    -->  Plugin intercepts via global fetch patch
             -->  Identifies provider by existing Authorization header
             -->  Picks next key (weighted round-robin)
             -->  Sets Authorization header
             -->  OpenCode makes LLM call
             -->  Success? Done.
             -->  Error?   Plugin classifies HTTP response:
                    429 / rate-limit  -->  Quarantine key (exponential backoff)
                    401 / 403 / 402  -->  Disable key permanently
                    5xx              -->  Quarantine key temporarily
                    ResourceExhausted -->  Overload action (2s backoff)
             -->  Next request picks a different key
             -->  Quarantined keys auto-release when timer expires

Error classification

Error	Action	Behavior
429	Quarantine	Exponential backoff: 60s, 120s, 240s, 300s cap
401 / 403	Disable	Permanent, requires manual re-enable
402	Disable	Billing error, permanent
5xx	Quarantine	Temporary, same backoff as 429
Rate-limit pattern	Quarantine	Detected in body/message text
Overload pattern	Quarantine	2s backoff (ResourceExhausted)
Other	Ignore	No action taken

Quarantine schedule

Consecutive errors	Quarantine duration
1	60 seconds
2	120 seconds
3	240 seconds
4+	300 seconds (cap)

If the provider returns a retry-after header, that value overrides the schedule.

Tools (LLM natural language)

These are tools the LLM can call. Say them in natural language:

Tool	What to say	Description
keychain-setup	"Add these API keys for nvidia: key1, key2"	Save API keys for a provider to .env (appends to existing)
keychain-remove	"Remove all nvidia API keys" or "Remove key1 from nvidia"	Remove all or specific API keys for a provider from .env
keychain-status	"Show me the keychain status"	Show all configured keys, their status, weights, and retry timers
keychain-reset	"Reset the keychain"	Reset all quarantined keys back to active state immediately

Example output:

## nvidia
  nvapi...abc [w=2] -- active
  nvapi...def -- QUARANTINED until 2026-07-04T12:35:00.000Z
  nvapi...ghi -- active
  [2 active, 1 quarantined, 0 disabled]

TUI Dashboard

Type /keychain in the prompt (or /failover, /opencode-failover) to open the keychain dashboard.

The dashboard displays live key counts and allows you to instantly search and select models from all your configured providers, automatically switching the current session.

The dashboard is read-only — use LLM tools (keychain-setup/keychain-remove) to manage keys.

The TUI plugin auto-registers on first start. A ~/.config/opencode/tui.json backup is created before any changes.

Supported Providers

Works with any provider that uses API key authentication:

Provider	Env var	Default scheme	Aliases
NVIDIA NIM	NVIDIA_API_KEYS	Bearer	nim
OpenRouter	OPENROUTER_API_KEYS	Bearer
Anthropic	ANTHROPIC_API_KEYS	Bearer
OpenAI	OPENAI_API_KEYS	Bearer
Cloudflare Workers AI	CLOUDFLARE_WORKERS_AI_API_KEYS	Bearer	cf, cloudflare
Any custom	<PROVIDER>_API_KEYS	Bearer

Note: For Cloudflare Workers AI, you must provide your Account ID as well: Add this cloudflare key: xxx with account_id: yyy.

The provider ID must match the providerID used in your OpenCode model configuration.

Development

git clone https://github.com/bulutmuf/opencode-failover.git
cd opencode-failover
bun install
bun test

Debug mode

OPENCODE_FAILOVER_DEBUG=1 opencode

Logs key injection, quarantine decisions, and provider pool initialization.

Project structure

src/
  index.ts          Plugin factory: hooks wiring + tools
  lib/
    config.ts         Env + options parser
    state.ts          KeyPool: rotation, quarantine, backoff
    classify.ts       Error classifier: status/body -> action
    auth.ts           Native key backup/restore from auth.json
    fetch-patch.ts    Global fetch monkey-patch interceptor
    shared.ts         Masked state file (~/.opencode/failover-state.json)
    version-check.ts  npm registry poll for new versions
    tui.tsx           TUI dashboard: /keychain slash command
  state.test.ts     7 tests
  classify.test.ts  19 tests
  config.test.ts    7 tests
  auth.test.ts      6 tests
documents/
  00-architecture.md
  01-error-patterns.md
  02-quarantine-strategy.md
  03-decisions.md
  04-provider-guides.md
  05-troubleshooting.md
  06-security.md
  07-changelog.md
  08-contributing.md
  09-faq.md

Architecture

See documents/ for detailed Architecture Decision Records:

Document	Description
Architecture Overview	Hook surface, module layout, OpenCode integration
Error Classification	Decision table, pattern matching, retry-after parsing
Quarantine Strategy	Exponential backoff, cap, recovery semantics
Design Decisions	Naming, scope, config precedence, distribution
Provider Guides	Per-provider setup (NVIDIA, OpenRouter, Anthropic, OpenAI, custom)
Troubleshooting	Common issues, debug mode, key status inspection
Security	Key masking, env safety, production risks
Changelog	v0.1.0 release notes
Contributing	Dev setup, commit format, test rules, PR flow
FAQ	Provider compatibility, edge cases, debugging

Contributing

Contributions are welcome. Please open an issue or submit a pull request.

License

This project is licensed under the MIT License.

Copyright (c) 2026 bulutmuf

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

---

Built for OpenCode