A 2.8 GHz x86 class processor can saturate a 100 Mbps network link using 3DES, and sustain 300 Mbps using AES without hardware acceleration. Note that these benchmarks were performed without any additional requirements. Most notably, no firewall rules were defined. If you do need to deploy firewall rules, there are a few rules of thumbs that will help you optimize.
Try to branch often, so that you traverse through less firewall rules. Do not use a single top-down list of rules.
Use the related keyword somewhere at the top of your ruleset, so those connections that do not need to traverse the whole tree (again) are accepted.
Put those types of packets that you get most of at the top of your rulesets. Depending on the server, this could be ESP packets, UDP 4500/500 packets, HTTP packets, FTP packets, DNS packets, or even database packets.
Optimizing your firewall is outside the scope of this book, but you should realize it could impact heavily on the performance of your VPN...