PR #20: LocalLLM / vLLM Provider Support

The vLLM choice is technically mature. An OpenAI-compatible endpoint keeps the adapter layer minimal — the getClient() override in backend/src/lib/llm/openai.ts is clean and does exactly what it needs to. vLLM's continuous batching gives better GPU utilisation than naive per-request invocations, which matters once you have eight attorneys hitting the same server during morning prep.

Server-side env for VLLM_BASE_URL is the right primitive. The cloud-API-key dance is a liability in legal: keys rotate, rate limits spike during depositions, and you are dependent on third-party uptime during a filing deadline. Shared inference server, no per-user auth complexity, no key in every browser session — that is the correct shape.

The dispatch table in backend/src/lib/llm/models.ts is clean. providerForModel() is a single lookup function with three branches and a hard throw on unknown — that is the correct shape for a routing primitive that will grow over time.

For lawyers actually working under data residency requirements, this PR closes a real, recurring problem. White-collar criminal defence under active DoJ investigation. Russian clients in Zurich post-sanctions. M&A diligence with confidentiality undertakings that explicitly preclude third-party cloud processing. In every one of those scenarios, the local-inference path is not a nice-to-have — it is the only legal way to use AI assistance at all.

The cloud-LLM rate-limit problem is its own quiet failure mode. Most lawyers learn it the hard way the first time their LLM stops responding mid-deposition prep. A self-hosted vLLM endpoint takes the rate-limit failure mode off the critical path entirely.

Three open questions — each grounded in a specific file in the diff. None block merge; all matter for legal-grade deployment.

1. Model version in the audit chain

VLLM_MAIN_MODEL=BredaAI is stored as "localllm-main" in the chat record. In eighteen months, if an attorney needs to reproduce what the model said in a matter, "localllm-main" does not answer the question: which version, which quantisation, which checkpoint. A running vLLM server returns its loaded model name on /v1/models. Capturing that at session open — one GET, cache for the session lifetime — and binding it to the chat record costs one additional DB column and closes the reproducibility question. The difference between "localllm-main" and "BredaAI-v3-Q5_K_M-2026-04-15" is the difference between a note and an audit trail.

2. Tool-argument parse failure and the silent empty-object path

From backend/src/lib/chatTools.ts:

let args: Record<string, unknown> = {};
try {
    args = JSON.parse(tc.function.arguments || "{}");
} catch {
    /* ignore */
}

The empty-object fallback prevents a hard crash on malformed arguments — fine. The failure mode, though, is invisible. A malformed tool call becomes a call with no arguments. Downstream, read_document receives doc_id: undefined, which either fails at the label-resolution step or reads the wrong document silently. The model then sees a tool result (possibly an error, possibly wrong content) with no signal that its argument generation was malformed. It cannot retry because it does not know it failed. A failure that is invisible is, sub silentio, the worst kind. Treating parse failure as a tool error — returning a structured error content to the model — lets the model self-correct in the next turn.

3. Supabase + R2 as hard dependencies against the local-first thesis

This PR makes the model inference layer local — the right call. But the README still lists Supabase Auth, Supabase Postgres, and Cloudflare R2 as required services. For a firm operating in an air-gapped environment or under data residency requirements that preclude shared cloud infrastructure, the deployment story is now: local inference, cloud persistence. The security boundary is still outside the firm's perimeter. A SQLite + local filesystem backend behind the same storage interface would complete the local-first thesis. Not a quick fix — but worth naming explicitly in the deployment docs.

@willchen96 — Mike was the catalyst. The 2,481 stars and 702 forks in eight days are the proof. You opened a category that the closed-source incumbents had quietly priced out of reach for solo practitioners and sub-50-lawyer firms. That trajectory is the proof the category was under-served.

@jpbreda — clean diff, tested against your own vLLM endpoint at bredaai.com, submitted with the merge conflicts that are a function of Mike's velocity rather than your work. That is the contribution shape Mike's main needs more of.