Download PDF ↓
On-Prem AI Fleet · Agentic-OS

Model Cards

Intended use, limitations, and governance notes for the language, embedding, and image models running in production on this sandbox — written in the spirit of AWS's AI Service Cards, applied to a fully self-hosted stack. Every model below runs on-prem — no inference call in this fleet leaves the building.

Compute
2× RTX 3090, 24GB VRAM each
Serving
Ollama → LiteLLM (OpenAI-compatible)
Tracing
Langfuse, every LiteLLM call
Secrets
Vault-issued, never in source
qwen3-coder:30b
Coding agent backend
Agentic
ClassMixture-of-experts code model, quantized (GGUF, Ollama registry default)
Served viathe inference proxy (LiteLLM) → the model host
Consumersopenclaw-agent (CLI coding agent)
Pinned depslitellm[proxy], fastapi==0.115.6, langfuse==2.60.9

Intended use

Autonomous and semi-autonomous code editing, refactors, and shell-level tasks issued through openclaw-agent. Runs alongside a native Ollama provider path that was left untouched — this is an additive LiteLLM route, not a replacement.

Limitations

  • Quantized weights — no benchmark run against the full-precision checkpoint on this fleet; treat generated code as unreviewed until tested, same as any LLM output.
  • No sandboxed execution boundary beyond openclaw-agent's own allowlist — the model does not independently rate-limit or scope its own tool calls.

Deployment notes

Version pins exist because litellm==1.95.0 breaks on newer FastAPI (get_flat_dependant import error) and langfuse 4.x drops the attribute LiteLLM's callback expects. Upgrade either only after re-verifying both constraints.

qwen3:8b
Router / classification model
Router
ClassSmall, low-latency general model — designated router/classifier
Served viathe model host, direct and via LiteLLM
ConsumersFlowise chatflows (email classification), Sandy chat routing

Intended use

Fast triage and classification ahead of a heavier model or a human decision — e.g. sorting inbound email into Work / Personal / Tour / Other / Spam and producing a structured {category, needs_reply, draft_reply} object for review.

Responsible AI design choices

  • Draft-only, human-in-the-loop: the email classifier chatflow stops at a JSON draft. No send/label action is wired to it — a person reviews and sends, by design, not by omission.
  • Local-only by policy: an earlier OpenAI-backed variant of this same flow was retired in favor of qwen3:8b specifically to keep the pipeline at zero cloud calls, rather than carry a design that broke that guarantee.

Limitations

Selected in part because the flow's original router (qwen2.5:7b) aged out of the registry — re-validate classification accuracy after any future model swap rather than assuming continuity.

nomic-embed-text
Embedding model
Retrieval
ClassText embedding model
Vector storeQdrant (the vector store)
ConsumersAll Flowise RAG chatflows, incl. Sandy's own knowledge base

Intended use

Backs every retrieval-augmented chatflow in the fleet — documents and notes are embedded once at ingest and queried at chat time against Qdrant, rather than re-embedded per request.

Limitations

  • Retrieval quality has not been measured with a held-out eval set — there is no golden-set or regression suite behind these chatflows yet, so silent retrieval drift after a document-set change would currently go unnoticed.
  • Single embedding model across all collections — no per-domain fine-tuning, so domain-specific jargon retrieves at whatever quality the base model provides.
gpt-oss:20b
General-purpose chat model
Chat
ClassOpen-weight general chat/reasoning model
Served viathe inference proxy → Open WebUI
Alternatives on fleetdeepseek-coder-v2:16b, qwen3.6:27b, gemma4:31b-it-q8_0

Intended use

Default conversational model surfaced to Open WebUI users — general Q&A, drafting, and reasoning where no specialized router has already selected a narrower model.

Limitations

No model-selection telemetry exists yet to confirm this is actually the model most sessions land on — Open WebUI allows per-session override, and usage hasn't been audited against Langfuse traces to date. Treat this card's "default" framing as intent, not a measured fact, until that's checked.

FLUX.1 schnell / Kontext
Image generation & editing
Multimodal
ClassText-to-image (schnell) and instruction-based image editing (Kontext)
RuntimeComfyUI on the model host, port 8188
ConsumersOpen WebUI image tools, standalone Photo Editor app

Intended use

Text-to-image generation and instruction-driven edits, including multi-image reference chaining (e.g. transferring a garment from one reference image onto a subject in another).

Limitations

  • No content-safety filter layered in front of this endpoint at the application level — anything ComfyUI itself doesn't refuse will render.
  • Multi-image compositing (garment transfer, subject blending) has an unreviewed likeness/consent surface — treat outputs involving real people's photos with the same care as any other image edit of a real person, on or off this fleet.
Resume Pipeline · Specialist Agents

Six single-purpose models behind the Resume Pipeline chatflow (separate Azure DevOps repo). All six fine-tune the same Qwen3 8.2B base via dedicated Modelfiles — narrow prompts, tight decoding limits, and in three cases a code-level guardrail that checks the model's own output rather than trusting the prompt alone.

grammar-agent
Grammar & mechanics review
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.15 · num_ctx 8192 · num_predict 800 · repeat_penalty 1.05
Toolreview_grammar, memoized per session+candidate
Observed latencyp50 103.0s · p90 961.4s (~16 min)

Intended use

Grammar, spelling, and mechanics review of a saved or pasted resume — feedback only, does not rewrite structure or content.

Responsible AI design choices

Decoding limits above exist because of a real incident: the original config (repeat_penalty 1 — no penalty — with no num_predict cap and Qwen3's thinking mode never disabled) let one request run past 50,000 tokens without stopping. Fixed with the caps shown here plus /no_think appended to the outgoing prompt.

Limitations

Session+candidate-keyed caching means a substantive resume edit followed by an immediate re-ask can replay a stale cached result rather than regenerate — an accepted trade-off against the alternative (redundant-call storms that stalled real requests).

formatting-agent
Structural reformatting
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.15 · num_ctx 16384 · num_predict 7000 · repeat_penalty 1.05
Toolreformat_document — highest-volume tool, 250 traced calls
Observed latencyp50 40.8s · p90 1144.5s (~19 min)

Intended use

Restructures and cleans up a resume's formatting. Explicitly forbidden from cutting content — shortening is condense-agent's job, kept as a separate model so "reformat" can never silently lose material.

Responsible AI design choices

Caught fabricating content in production — invented an Executive Summary section, fake percentages, and invented months/cities for date ranges the source never specified. Fixed with explicit verbatim-grounding rules in the Modelfile, verified clean against real resumes afterward. Also affected by a silent-truncation mode on long resumes (12KB+): the tool now fetches a saved candidate's text directly from docgen instead of asking the model to re-type it as a JSON argument.

Limitations

The p90 here is a known Ollama daemon-hang signature (multiple requests completing at an identical timestamp), not generic slowness — mitigated by a watchdog service since 2026-08-11, whose real catch-and-recover behavior against a live hang is still unverified in production.

condense-agent
Length reduction
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.15 · num_ctx 16384 · num_predict 4000 · repeat_penalty 1.05
Toolcondense_resume, keyed by session+candidate+target length
Observed latencyp50 19.7s · p90 28.1s — tightest of the six

Intended use

Shortens a resume to a target length by cutting and tightening the weakest/oldest material first. Every output ends with a CUT SUMMARY: line naming everything removed, so nothing is dropped silently.

Responsible AI design choices

Prompt-only "don't invent sections" rules didn't hold: a live test produced a fabricated "Technical Skills" section listing "Linux" — a skill absent from the actual source resume. Fixed with a code-level check that strips any output heading whose exact text doesn't appear in the source, with follow-up fixes for whitespace artifacts (e.g. a PDF-extracted no-break space) and legitimate heading merges (e.g. "Education" + "Professional Development" → "Education & Certifications") so the guardrail doesn't flag real edits as fabrication.

Limitations

Early version cached by content fingerprint, but the orchestrator re-serializes long text slightly differently each call — so a single turn could trigger 7+ live model calls before a cache hit ever landed. Fixed by dropping the fingerprint and keying on session+candidate+target only.

resume-critic-agent
Content & narrative critique
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.2 · num_ctx 8192 · num_predict 1800 · repeat_penalty 1.05
Toolreview_resume_content, memoized per session+candidate
Observed latencyp50 17.6s · p90 734.6s (~12 min)

Intended use

Qualitative feedback on length, quantification, targeting, and narrative strength — critique only, produces no rewritten output of its own.

Limitations

Same Ollama-hang latency signature as the other specialist agents; shares GPU scheduling with the fleet's general chat model, so real-world response time depends on what else is loaded at request time, not just this model's own cost.

cover-letter-agent
Cover letter drafting
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.3 · num_ctx 10240 · num_predict 1200 · repeat_penalty 1.05
Toolbuild_cover_letter — not memoized (multi-posting use is legitimate)

Intended use

Drafts a cover letter from a candidate's saved resume against one specific job posting.

Responsible AI design choices

Code-level guardrail strips any generated sentence that claims a technology or term from the job posting which the candidate's own resume never mentions — a fabrication check applied to the actual output, not left to prompt instructions alone.

job-fit-agent
Fit scoring
Resume Pipeline
ClassQwen3 8.2B base, dedicated system prompt
Decodingtemp 0.2 · num_ctx 12288 · num_predict 1800 · repeat_penalty 1.05
Toolassess_job_fit — not memoized (multi-posting use is legitimate)

Intended use

Produces a justified 0–100 fit score against one specific job posting, with named strengths, gaps, and a self-check of its own reasoning.

Limitations

The score is model-justified, not validated against actual hiring outcomes — treat it as a structured opinion to inform a decision, not a calibrated probability.