Help & docs

Three steps to a working agent session — most of the platform makes sense once you have one running.

1. Add a host — Open the Hosts page and click Add host. Enter the address, SSH user, and paste a private key (or password). LobsterFarm probes the host immediately so a typo or wrong credential fails right there, not at session-start time. Hosts →
2. Add an AI channel — On the Channels page, pick a provider and paste an API key (OpenAI / Anthropic) or an OAuth refresh token (Gemini CLI / Codex). Lobsterfarm probes the credential up front so a typo or revoked key fails immediately, not at session time. Channels →
3. Run a session — Open the Sessions page and start one. Each session attaches to a tmux pane on the host you picked; the agent runtime (currently Claude Code) is wired to your channels through the gateway. The Live / Ended filter narrows the list once you accumulate history. Sessions →

Tip: hit ⌘ K anywhere to fuzzy-jump between pages, switch theme (light / dark / solarized-dark / dracula / nord / tokyo-night), or sign out — no mouse needed.

Once the first session is running, here is what each top-level page is good for. None of these surfaces are stubs — they are wired to real backend endpoints.

• Dashboard — Aggregate health at a glance plus a "recent hosts" panel — the box you SSHed into last is one click away. Dashboard →
• Hosts — Inventory with full-text search across name / address / notes, sort=recent ordering, last-seen timestamps, and a TOFU SHA-256 fingerprint pinned at first connect (also included in CSV export). Drill into a host for the SFTP browser and port-forward manager. Hosts →
• Sessions — Live and historical agent sessions with a Live / Ended filter, automatic reconnect on a dropped SSH pipe, and asciinema recordings on S3. Sessions →
• Deploy — Templates that fan-out to multiple hosts. Run detail aggregates host status, lets you filter by success / failed / running / pending, asks for confirmation before cancel, and surfaces a "retry failed" CTA once the run is terminal. Deploy →
• Proxies — HTTP / SOCKS5 egress proxies fronting outbound LLM calls. Each row shows reference counts so you cannot accidentally delete one still wired to a channel. Proxies →
• Snippets — Reusable command snippets with tag filtering, free-text search, and one-click copy — useful for the same `journalctl -u …` you keep typing. Snippets →

  ┌──────────────┐       ┌────────────────┐       ┌──────────────┐
  │  your agent  │ ────▶ │ lobsterfarm gateway  │ ────▶ │  provider    │
  │ (openai SDK) │       │  router + pool │       │  (Anthropic) │
  └──────────────┘       └────────────────┘       └──────────────┘
                            │      │
                       picks channel by
                       priority + health

A channel is one credential + one upstream model. Lobsterfarm probes it at create-time, then every few minutes while it’s in an active pool. Each channel carries its own health flag, cooldown timer, and quota-remaining counter so routing decisions don’t need a sidecar cache.

When the same provider has multiple accounts (e.g. Google-corp + Google-personal), create a channel per account and give each a descriptive accountLabel.

A pool is a named list of interchangeable channels. The router picks one at request time:

1. Filter out anyone whose cooldown_until is in the future.
2. Sort by priority (lowest first), then by the pool’s selection strategy.
3. Try the first; on 429 / 401 / 5xx, cool them down and try the next.
4. Log a failover row in gateway_events for every hop.

Tip: keep pool sizes small (2–5 members). Deep pools increase tail latency and make failures harder to reason about. If you need more headroom, scale horizontally by creating per-model pools.

Pick API key when the provider issues one (OpenAI platform, Anthropic console, Groq, OpenRouter, …). The key is encrypted at rest and decrypted only in the gateway process when it signs an upstream request.

Pick OAuth when the account is a human login (Gemini CLI flow, ChatGPT plan via Codex CLI). You paste the long-lived refresh token once; LobsterFarm mints short-lived access tokens on demand and refreshes them a minute before they expire. If a refresh fails (revoked consent, password change, MFA re-challenge) you'll see an oauth.refresh-failed event and the channel drops to unhealthy until you rotate credentials.

All channels in the pool are cooling down

The router returned 503 because every member’s cooldown_until is still in the future. Usually a provider-wide incident. Mitigation: add a fallback pool for a different provider of the same model family, or lower defaultCooldownSec if you think the outage is transient.

oauth.refresh-failed event, channel went unhealthy

The refresh token was revoked. Re-run the CLI auth flow (Gemini / Codex), put the new refresh-token in a secret, and point the channel at it via Rotate credential. No router restart needed.

probe.fail with http 404

Almost always a model-name typo. Confirm the spelling against the provider docs — gpt-4o, claude-sonnet-4.7, gemini-2.5-pro.

Quota shows 0 but account works in the CLI

Some providers (Gemini free tier) only expose quota after the first successful call. Wait for one real request (or hit Rotate credential to re-probe) and the counter will populate.

Can’t find the right requestId in my app logs

The gateway propagates X-Request-Id. If your client doesn’t set one, the gateway mints a req_* id and puts it on both gateway_events.request_id and usage_events.request_id.

Sessions — Sessions list lives at /sessions. Each session attaches to a tmux pane on a host and needs an environment (project on host) + an agent profile (model + system prompt). The list filters Live / Ended and survives reconnects; recordings are persisted to S3 as asciinema cast. Sessions →

Routes — Model routes (/model-routes) map a virtual model name like `prod-sonnet` to one or more channels / pools. Your agent uses the virtual name; you re-point the underlying channel without redeploying agents. Routes →

Rotate credential — Rotate a credential without restarting anything: open /channels, hit the Rotate button on a card, paste the new key/token. The old secret is overwritten in place; the next request picks up the new one immediately.