A single reference for how this self-hosted AI platform is actually built, audited live rather than from the original plans: the Incus fleet layout, storage and GPU topology, the MLOps governance lifecycle, the security stack, and a personal identity-lab practice environment.
~180 containers/VMs split across 7 projects by workload type.
Champion/challenger gate, drift detection, staged rollback.
Five domains around the same single-operator core.
Compute, security, data, observability, automation, inference.
A practice AD/Entra domain with its own subordinate CA.
Five segmented bridges, one proxied egress path.
Two ZFS tiers by latency need, verified scrub, offsite backup.
Tracked openly rather than hidden — see the closing section.
prm-<sha> build:latestThe job-fit agent scored a mismatched candidate 95/100 while its own reasoning said otherwise — an arithmetic self-contradiction. The gate held it back at 0.667, it was root-caused and re-prompted, and it re-evaluated to 0.967 before promotion. The pre-fix version is still visible in the live model registry today.
| Domain | Built & authored | Configured on top of defaults | Upstream / vendor as-is |
|---|---|---|---|
| BUILD | Deployment runbooks · promotion scripts | Incus profiles · GPU passthrough config | OCI base images · package repos |
| SECURE | Vault policies · AppRole delivery | Semgrep / Gitleaks scan rules | DefectDojo · SonarQube defaults |
| PROTECT | sandbox-backup.sh · retention policy | ZFS snapshots · pre-migration safety snaps | QNAP offsite target |
| OBSERVE | Token-metrics proxy · Grafana dashboards | Prometheus · Loki · Pyroscope · CheckMK | Alert notification channels |
| AUTOMATE | Ansible playbooks · Python / shell tooling | Semaphore · git repositories | Provider APIs |
| GOVERN | MLflow gate logic · drift-check canonicalization | Marquez lineage · GenAI datasets | Personal AWS Organization (separate project) |
Domain controller — AD DS, DNS, Group Policy
Subordinate Certificate Authority — PKI issuance
Domain-joined Windows 11 client
Secondary workstation for join/GPO testing
SaaS-integrated box — Zoho identity federation
Remote monitoring & patch management agent
Practice AD forest
Own project, own network
Complements real M&A identity-integration work across acquisitions.
A subordinate CA most personal labs skip entirely.
TacticalRMM mirrors real endpoint-management tooling.
Its own Incus project — no path into the AI fleet's data.
| Bridge | Subnet | Role | Internet |
|---|---|---|---|
| agent-net | 10.0.10.0/24 | Primary fleet fabric ("agent-net") — ~98 instances | No — proxy-only egress via tinyproxy |
| lan | 10.0.1.0/24 | Real LAN, LAN VLAN — human + internet access | Yes |
| gpu-net | 10.0.30.0/24 | Isolated GPU network — Ollama + its proxies only | No, by design |
| default-br | 10.0.40.0/24 | Default Incus bridge — Windows VMs, UniFi | Yes |
| k8s-net | 10.0.50.0/24 | k3s cluster — 3 masters, 3 workers | Yes |
| mgmt-br | — | Unmanaged, unused, undocumented — flagged for review | Unknown |
Same dd fsync test run on each pool. The HDD mirror was recently reseated and independently verified: all four drives SMART-passed and a full scrub completed with zero errors before this deck was refreshed.
DB persistence. Several DB-backed containers still sit on plain container-root disk rather than decoupled named volumes — a rebuild would take the data with it.
GPU contention grew. Three consumers (Ollama, SwarmUI, a k3s Triton node) now share two physical GPUs, up from two — no load policy has been exercised yet.
mgmt-br is unexplained. An unmanaged, unused bridge with no documentation trail — flagged rather than ignored.
Flowise sits empty. The real agent-orchestration effort moved to a separate project instead — worth a deliberate decision on Flowise's future.
The AppSec stack is new. Semgrep/Trivy/Gitleaks/SonarQube/DefectDojo are deployed but haven't produced a real finding yet — this section will update once they have.
A reference build with no listed gaps is less believable than one with a maintained, dated punch list.
This matches the portfolio's own scope statement: reference-implementation evidence, not scale/production.
This list is generated from the project's own sandbox-overview.md and updated as items close.
From the Incus fleet layout to model governance, security scanning, and a dedicated identity-practice lab — one operator, one box, tracked openly rather than oversold.