Why should I shrink my model, and how?
After learning all about how the power of large models can boost your accuracy, you may be wondering, why would I ever consider shrinking my model? The reality is that large models can be very slow to respond to inference requests and expensive to deploy. This is especially true for language and vision applications, including everything from visual searching to dialogue, image-to-music generation, open-domain question-answering, and more. While this isn’t necessarily an issue for training, because the only person waiting for your model to finish is you, it becomes a massive bottleneck in hosting when you are trying to keep your customers happy. As has been well studied, in digital experiences, every millisecond counts. Customers very strictly prefer fast, simple, and efficient interfaces online. This is why we have a variety of techniques in the industry to speed up your model inference without introducing drops in accuracy. Here, we’...