AI Governance & MLOps Lifecycle

Independent R&D · Reference build · single-operator scale

Problem

A 7-agent LLM system (a resume-processing pipeline: job-fit scoring, formatting, grammar, keyword extraction, condensing, cover-letter drafting, and critique) had no gate between “someone edited a model’s system prompt” and “that model is live.” An edit could silently regress a model’s behavior — including in ways the author’s own testing wouldn’t catch, because the author is the one who introduced the mental model the regression fits.

The lifecycle built to close that gap needed to answer four questions before any promotion: does the candidate actually perform better than what’s live, is what’s live still what was approved, can a bad promotion be undone without guesswork, and is the evaluation itself trustworthy — not just fast.

Architecture

• Model registry & staged promotion (MLflow, Postgres-backed) — a candidate is built to a staged tag (prm-<sha>), never overwriting :latest directly.
• Champion/challenger A/B gate — the staged candidate is evaluated head-to-head against the live champion on the same testset before any cutover is allowed.
• Preflight + rollback — promotion requires a prior @validated pass; the pre-promotion model is preserved as a pre-<digest> snapshot, and a rollback path restores it bit-for-bit on failure.
• Automated drift detection — a weekly cron compares each live model’s Modelfile against its registered @production version.
• LLM-as-judge evaluation — a registered judge model scores each agent’s output on substance (accuracy, completeness, absence of fabrication), not surface grammar; testsets are logged as MLflow GenAI datasets so a scoring change is auditable against a fixed rubric.
• Data lineage (Marquez/OpenLineage) across the MLflow ↔ Ollama pipeline, so every model version, eval run, and promotion decision traces back to its inputs.

Implementation notes — where the design met reality

Two hardening passes came directly from the lifecycle catching problems in itself, not just in the agents it was gating:

• A silent-no-op footgun. The underlying model tool (ollama create) can return success without actually replacing a model’s live system prompt. The promotion script now diffs the live digest before and after every promotion and aborts cleanly — leaving the staged tag and snapshot recoverable — if nothing actually changed.
• A false-positive in the drift checker itself. The first version flagged three agents as drifted because it compared raw Modelfile text, which is sensitive to harmless parameter reordering. It was rewritten to canonicalize parameter order before comparing, validated against both a true-negative (reordering only) and a true-positive (a real prompt change) case. A governance system that produces false alarms trains people to ignore it — this fix mattered as much as the gate itself.

Measured result

A real promotion cycle, not a simulated one, run against all 7 agents against a 0.70 pass threshold:

The gate correctly withheld job-fit-agent: it was scoring a mismatched candidate/role pairing 95/100 while its own generated reasoning said the candidate didn’t meet the role’s requirements — an arithmetic self-contradiction inflating its own output score. Root-caused to the scoring prompt computing a fabricated total instead of reporting sub-scores; rebuilt to emit Accuracy/Completeness/Clarity independently and compute the average downstream. Re-evaluated at a mean of 0.967 — PASS — and promoted through the same staged/snapshot path as any other change.

Evidence

A separate rollback was independently exercised and verified bit-identical against its pre-rollback digest, confirming the recovery path works, not just that it exists on paper.

Limitations

This is a single-operator, single-host reference build, not a production system with concurrent users or a multi-environment promotion path — the honest comparison is to a well-instrumented personal MLOps pipeline, not an enterprise platform team’s. The lifecycle has been exercised through one real full gate-fail-fix-promote cycle described above; it has not yet been measured at volume (aggregate pass rates, mean time to detect drift across many cycles), and that aggregate data would need to come from the live tracking system directly rather than from this writeup.