Executive Summary: Eliminating CPU RAM spillover by rightsizing context window to 16k and enabling Flash Attention with Q8 KV cache in Ollama Docker.
1. Challenge & Context
Local LLM inference using ittechstl-architect:latest (Qwen 14B) on an NVIDIA GeForce RTX 4070 SUPER was running at only 15.4 tokens/second. GPU utilization diagnostics revealed that over-allocating the context window to 32,768 tokens forced an 18% CPU offload penalty across PCIe, throttling throughput by ~50%.
2. Architectural Solution
Audited Ollama runtime parameters and adjusted local_llm.py to target 16,384 tokens. Paired with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, the entire 10 GB model footprint was locked strictly inside VRAM with zero host RAM spillover.
3. Verified Outcomes & Telemetry
Multi-tier benchmarking across PowerShell script generation, disaster recovery runbooks, and SAN triage verified sustained generation speeds jumping from 15.4 to 45.2 tokens/second (2.93x speedup) with zero throttling.