-
Book Overview & Buying
-
Table Of Contents
LLMs in Enterprise
By :
This chapter explored advanced techniques and key considerations for optimizing LLM inference. It began by examining distributed inference, where frameworks such as vLLM and TensorRT-LLM use model parallelism to scale across multiple GPUs. NVIDIA Dynamo was introduced as a novel serving model that separates compute and memory, improving resource efficiency. The chapter then covered hybrid inference methods, such as combining quantization with KV caching to enhance speed and reduce memory use. Speculative decoding was also discussed as a way to boost token generation performance without compromising quality. Ethical and operational factors were highlighted, including licensing shifts such as TGI that pose compliance challenges. Cost-performance optimization emerged as a recurring theme, emphasizing the impact of tool and hardware choices on GPU consumption. Collectively, these insights reflect the need to balance innovation, efficiency, and responsible deployment in the evolving...