Zero-Host-Copy Inference: Keeping the CPU Out of the Hot Path in a C++/CUDA Pipeline (165 FPS)
When modern INT8-quantized neural networks execute in single-digit milliseconds, the system bottleneck shifts from raw compute to I/O and pipeline infrastructure. On discrete GPUs, constantly bouncing high-resolution video frames and intermediate payloads (like bounding box tensors) across the PCIe…