-
Book Overview & Buying
-
Table Of Contents
Deep Learning with C++
By :
You’ve trained a model that performs brilliantly offline. Now comes the hard part: shipping it—and keeping it fast, reliable, and cost-effective as usage grows. This chapter is about inference in the real world: turning a checkpoint into a predictable C++ service that meets strict latency/throughput Service Level Objectives (SLOs), scales cleanly, and stays healthy over time.
By “inference,” we mean running a trained model to produce predictions; “serving” is exposing that computation behind an API or stream while managing versions, scaling, and observability. We’ll cover the common modes you’ll encounter: online/real-time (low-latency requests), streaming (e.g., token output for LLMs), batch (throughput-first jobs), and edge/on-device (privacy/latency). What matters in production are the SLOs: latency (p50/p95/p99; time-to-first-byte/token), throughput (QPS/tokens-per-second...