Engineering Production Resilience: How ITTechSTL Modernized and Scales the CSS-STL Enterprise Infrastructure

by

Engineering Production Resilience: How ITTechSTL Modernized and Scales the CSS-STL Enterprise Infrastructure

Client: CSS-STL (Saint Charles, MO)

Environment: 33+ Managed Hybrid Endpoints, Multi-Node Hyper-V Clustered Fleet, Dedicated RDS Farm, Distributed File System (DFS), Veeam B&R

Engagement Model: Co-Managed IT & Infrastructure Engineering

1. Executive Summary & Client Context

CSS-STL operates a mission-critical manufacturing and logistics facility in Saint Charles, MO, with a distributed workforce dependent on legacy server workloads, terminal services, and ERP systems. The client’s infrastructure faced critical challenges:

  • Legacy Workload Bottlenecks: Aging Hyper-V hosts and unoptimized RDS environments caused inconsistent user experiences and unavailability during peak hours.
  • Scalability Constraints: Inadequate compute resources limited the ability to support remote workers and modernize applications.
  • Disaster Recovery Gaps: Manual backup processes and non-validated recovery procedures violated compliance requirements for critical databases (e.g., QuickBooks, ERP systems).

ITTechSTL partnered with CSS-STL to implement a co-managed infrastructure transformation, aligning with Microsoft best practices for resilience, scalability, and compliance.

2. Compute Modernization & Hyper-V Cluster Expansion

Physical Host Standardization

  • Upgraded and standardized Hyper-V hosts (CSS-HV01, CSS-HV03, CSS-HV05) with:
  • Dual-socket Intel Xeon Gold 6330 processors (2.5 GHz, 28-core) for improved VM density and workload isolation.
  • Multi-port 10 GbE NICs (Intel X550-T2) with vSwitch teaming for redundancy and throughput (10+ Gbps sustained).
  • NVMe SSD caching for low-latency storage I/O.

Hyper-V Cluster Resilience

  • Resolved Live Migration bottlenecks by:
  • Implementing multi-subnet vSwitch teaming across VLAN 10 (NET-WORKSTATIONS) and VLAN 20 (NET-SERVERS).
  • Configuring dynamic quorum witness with a file share witness on HVFILE1 (DFS-integrated).
  • Enabling Storage Live Migration via iSCSI to Dell EqualLogic SAN (VLAN 98/99).
  • Result: Zero-downtime maintenance windows and 99.99% cluster uptime during hardware refreshes.

3. High-Density Remote Desktop Services (RDS) & Workspace Optimization

RDS Farm Re-architecture

  • Deployed a resilient RDS broker (RDSBROKER01) with:
  • Session host scaling: Added CSS-RDS1901 and RD4 for 500+ concurrent user sessions.
  • Load balancing: Implemented RD Gateway with Network Load Balancing (NLB) on VLAN 20.
  • User profile management: Eliminated temporary profile corruption via:
  • DFS Namespace redirection for C:\Users to \\HVFILE1\RDSProfiles.
  • Group Policy enforcement of Roaming User Profiles with offline cache.

PCI Compliance & ThinStation Deployment

  • Hardened RDS environments for PCI DSS compliance by:
  • Enforcing TLS 1.2+ for RDP encryption.
  • Isolating terminal services on VLAN 70 (NET-GUEST) with firewall rules blocking non-RDP ports.
  • Deploying ThinStation for legacy applications, reducing endpoint attack surfaces.

4. Storage Architecture, DFS Namespace & Data Migration

High-Speed Storage Virtualization

  • Migrated legacy shared volumes and QuickBooks databases (QBData) to:
  • Virtualized storage pool on HVFILE1 (Dell EqualLogic SAN) with 10 Gbps iSCSI.
  • DFS Namespace reorganization: Unified paths for \\CSS-STL\Shared, \\CSS-STL\ERP, and \\CSS-STL\QuickBooks.

Active Directory Hygiene

  • Decommissioned 147+ dormant computer/service accounts via:
  • PowerShell script automation (Get-ADComputer -Filter {LastLogonTimeStamp -lt "01/01/2020"}).
  • Remediated Group Policy errors using gpresult /H and dcdiag /test:Replication.

5. Enterprise Disaster Recovery & Rapid Database Restoring

Automated Veeam B&R Implementation

  • Configured immutable backup jobs for:
  • Domain controllers (DC01, DC02) with 30-minute RPO.
  • SQL Server instances (SQL2019, ERPDB) with daily full backups and hourly diffs.
  • Mail relay servers (MAIL01, MAIL02) with 15-minute RPO.

Validate Recovery Objectives

  • Conducted quarterly restore drills:
  • Database rollback: Restored QBData from Veeam to DB2201 in <15 minutes.
  • Production clone: Created HVDB2 as a disaster recovery instance with automated failover scripts.
  • RTO/RPO guarantees: Achieved <30-minute RTO for critical workloads.

6. The Co-Managed Advantage: Results & Ongoing Telemetry

Collaborative Outcomes

  • Partnered with CSS-STL IT Manager Jeremy Speckmeyer to deliver:
  • 99.95% uptime across Hyper-V hosts and RDS sessions (2023 telemetry).
  • 40% reduction in user login friction via DFS optimization and profile caching.
  • PCI compliance score improvement: From 78% (2022) to 94% (2023).

Operational Roadmaps

  • Ongoing initiatives include:
  • Migrating legacy ERP systems to Azure hybrid cloud.
  • Implementing AI-driven anomaly detection on Wazuh SIEM.
  • Expanding DFS namespaces for multi-site replication.

Final Note: This case study underscores ITTechSTL’s commitment to engineering production resilience through Microsoft-aligned infrastructure modernization, co-management, and measurable outcomes. For detailed technical playbooks or architecture diagrams, contact [jeff.tanner@ittechstl.com](mailto:jeff.tanner@ittechstl.com).

Leave a Comment