The Age of Self-Hosted SLMs on Infra You Already Have

Enterprises spent a decade buying GPUs they barely used, Kubernetes they over-provisioned, and on-prem racks they were told would be obsolete once "the model" lived in someone else's region. That story is ending. The useful work (classify this clause, extract this field, draft this notice against this playbook) now fits in a small language model you can run on the same boxes. The age of self-hosted SLMs is not a new data center. It is the infra that was already there, finally doing the job.

The job was never "run GPT"

Most internal AI requests we see are not open-ended chat. They are narrow, repeatable, and radioactive to send off-site: contract clause labels, KYC document fields, ticket routing, policy Q&A over a private wiki. A 7B–8B model, or a domain BERT with LoRA, is enough if the workflow is structured. A frontier API is a convenience. It is not a requirement, and for a lot of firms it is a non-starter.

We proved the pattern on our own contract risk classification research: Legal-BERT with LoRA, then a calibrated reasoner, running as a normal service, not as a call to a public LLM. Macro-F1 moved from a lexical baseline of 0.719 to 0.866 without shipping a single contract to a third-party model host. That is the SLM thesis in one line: the model is a component on your network, not a destination on the internet.

Sit on the infra that already exists

The expensive mistake is treating "AI" as a new island: new VPC, new GPU SKU, new observability stack, new vendor security questionnaire. The cheaper, more durable version is boring. Inference is another process next to the app that already has identity, logging, backups, and a change window.

• Same Kubernetes (or systemd) as the API that already serves the workflow
• Same private network, same IAM, same SIEM, tokens do not leave the tenant
• Same Postgres you already backup, embeddings and spans live beside the documents
• Same GPU nodes that sat idle after overnight batch, or the CPU path if latency allows
• Same air-gap story the CISO already signed off for the rest of the stack

If the model cannot run where the document already lives, you do not have a self-hosted SLM. You have a second cloud with extra steps.

Why SLMs fit the rack and the policy

Small models win here for reasons that have nothing to do with leaderboard screenshots.

• They fine-tune. LoRA on a domain checkpoint beats prompting a giant general model when the labels are yours (41 CUAD categories, your playbook, your ticket taxonomy).
• They fit. 7B quantized sits on a single workstation GPU or a share of an existing inference node. You do not reserve an H100 island for clause tagging.
• They fail locally. Timeout, OOM, and fallback are your on-call problem, which is what you wanted, because the alternative is a silent vendor outage during close.
• They keep bytes in-region. Legal, bank, and health teams already have this as a board constraint. The architecture should match the memo.

We treat a hosted frontier model as an optional upstream, not as the system of record. LexVault can call Claude for generation, but the citation check is ours, and Ollama is the fallback when the document must not leave the instance. The multi-agent debate stack is the same idea in the open: BYOK plus local Ollama, keys encrypted at rest, no telemetry. Self-host is a product requirement, not a slide.

What we actually put on the existing cluster

A production SLM service is a small set of pieces you already know how to run:

1. A runtime you can pin: vLLM, llama.cpp, or Ollama behind an internal HTTP contract, not a notebook
2. A model artifact in your registry, with a SHA, a license you can defend, and a rollback
3. A LoRA or classifier head trained on your labels, stored next to the base weights
4. A queue in front so a burst of PDFs does not take down the same node as payroll
5. Eval fixtures that run on deploy, twenty to fifty real documents, not a vibe check
6. A documented fallback: smaller model, rules, or human, never a fabricated citation

None of that is new infra. It is a Deployment, a PVC, a Service, and the same CI that already ships the rest of the product. We wrote about the same discipline for graphs in LangGraph in production: typed state, checkpoints, human gates. Swap "graph" for "weights" and the on-call story is identical.

What we kill in the first workshop

• A new "AI VPC" whose only purpose is to talk to a SaaS model
• Sending full contracts to a public API "just for the pilot"
• Fine-tuning a 70B when an 8B plus a classical head already beats the baseline
• GPU shopping before you have a golden set and a latency SLO
• Chat as the homepage of an internal tool whose real job is extract, classify, or route

That last one is the same argument as why legal AI needs citations, not chat. Self-hosting does not excuse a box that speaks fluently without a span. It makes the span cheaper to verify, because the weights and the document share a filesystem policy.

Cost is not the only reason. It is the honest one.

Per-token bills look small until a batch of 10,000 agreements hits a summarizer. Self-hosted SLMs turn that into electricity and depreciation you already pay. More important: the unit of cost becomes a job you can estimate (documents per hour on this node) instead of a surprise invoice when a team discovers the prompt. Finance understands capacity. They do not understand "we used the model a lot this sprint."

When a frontier API still belongs

We are not religious. Use a hosted model when the task is open-ended, rare, or genuinely needs a generalist: a one-off research memo, a messy email that is not PII-heavy, a UI copy pass. Keep it off the path that carries the corpus. Hybrid is the default: SLM on the private job, API on the generic one, server-side checks on anything that will be shown as fact.

How to start on Monday

1. Name one workflow whose documents must not leave the building
2. List the infra it already sits on, cluster, GPU, identity, backup
3. Pick a small open weight or a domain encoder; do not start with a 70B
4. Train or LoRA on fifty labeled examples you already have
5. Serve it on the existing node with a health check and a rollback SHA
6. Gate the first production traffic with a human and a golden eval

If that list feels like application engineering, good. That is the point. The age of self-hosted SLMs is the age of putting models where the work already runs. See the classification research, how we scope, or start an inquiry with the workflow and the constraint: same VPC, no new island.