In a perfect scenario, your VPN users will have high-speed, dedicated connections over some sort of hard line to reach your server. Not only that, these same users will have top-end systems, equipped with an exorbitant amount of RAM, and high clock-speed CPUs equipped with the latest crypto-offload chipsets.
In reality, however, there are a variety of remote locations and devices that users will leverage to connect to a VPN. Some of these are out of necessity, such as a high-latency satellite connection, and others are out of convenience, such as using a mobile device. For the majority of your users, you should be able to provide sufficient cryptographic protections while still maintaining a comfortable performance level.