Inside a Production MSP Systems Lab
How an enterprise Managed Service Provider (MSP) engineers a resilient on-premises testbed combining hardware clustering, isolated iSCSI storage fabrics, and zero-leak local AI agents.
iSCSI MPIO SAN
Layer-3 Microsegmentation
Local GPU LLM Engine
Active Directory Split-DNS
Core Architectural Principle: Zero Assumptions & Dual-Source Verification
Every service, IP address, and storage path in this environment is cross-verified across at least two independent sources of truth (live kernel/process inspection + authoritative documentation/directory state). This design enables rapid prototyping of client solutions with predictable, production-grade fidelity.
1. Compute & Virtualization Topology
The virtualization layer balances high-throughput bare-metal efficiency with flexible workload portability:
- Primary Bare-Metal Hypervisor: Runs an open-source enterprise hypervisor (XCP-ng Dom0) featuring dual-NIC management bonding (
balance-slb) providing active-active 2 Gbps aggregated network throughput. - Secondary Deployment & Imaging Host: Operates Windows Server 2022 hosting multi-VLAN PXE services (iVentoy Mode 2 + WDS) to deliver zero-touch OS deployments across the entire bench.
- Consolidated Orchestration: Core appliances (enterprise monitoring NOC, microservice staging nodes, and smart infrastructure controllers) run as isolated virtual machines on the cluster.
2. Centralized Storage: Dual-Path iSCSI Multipathing (MPIO)
Storage architecture distinguishes an amateur lab from an enterprise facility. Rather than relying on file-level network shares, virtual disk images (VDIs) leverage a dedicated enterprise iSCSI SAN array:
True MPIO Multipathing
Discrete, unbonded physical NICs connect directly into storage switches with active round-robin striping across all block devices.
Hardware Flow Control
End-to-end 802.3x Ethernet pause frames eliminate packet drops under sustained sequential block reads (~110+ MB/s per link).
Strict Storage Isolation
Dedicated non-routable VLANs with DHCP explicitly disabled, completely segregated from all workstation and server subnets.
3. The Private AI Engine: Local GPU Acceleration
To power automation workflows without leaking client topologies, scripts, or operational telemetry to third-party cloud providers, the facility runs a dedicated on-premises AI core:
- Hardware: NVIDIA GeForce RTX 4070 SUPER (12 GB VRAM).
- Model Architecture: Custom-tuned 14B parameter reasoning model pre-conditioned with native system administration and PowerShell runbooks.
- Optimization Breakthrough: By rightsizing the active context window from 32k to 16k and pairing Flash Attention with a Q8 quantized KV cache, the entire model was locked into 100% GPU VRAM residency, boosting generation speed from 15.4 to 45.2+ tokens/second (+293% speedup).
4. Network Security & Micro-Segmentation Matrix
Managed by a central security gateway and PoE distribution switches, traffic is segregated into dedicated security zones:
| Segment | Role | Security Controls & Policies |
|---|---|---|
| Management | Gateways & Switches | Isolated administrative access only |
| Workstations | Engineering & Admin Endpoints | Split-DNS resolution via primary Domain Controller |
| Production Servers | Core Virtual Machines & Dev Nodes | Stateful inter-VLAN inspection, zero unauthenticated ingress |
| IoT & Smart Devices | Surveillance & Environmental | Strict client isolation; cloud-egress only; no LAN access |
| Storage Fabric | iSCSI MPIO Data Paths | Non-routable; completely air-gapped from internet egress |
Next Phase: Two-Node High-Availability Cluster
The lab is currently executing the promotion of a secondary Active Directory Domain Controller and automated FSMO role transfer. Once complete, the secondary host will be converted into a second bare-metal hypervisor node, delivering a fully redundant 2-host high-availability pool with zero-downtime live migration across the shared SAN.