Passwords are great, but they're also terrible.
Most people use weak passwords, and while I hope that's not you, there's always the chance that someone in your team doesn't have the discipline you do, and resorts to football99 or similar for connecting to your shared remote host.
With password access enabled, anyone might be able to connect to your server from any country by brute-forcing their way into your machine, given enough time and enough processing power.
I say "might" because as long as you use secure passwords of a decent length, passwords can be hard to guess, even with the power of a sun. Consult your company security policy when deciding these things, or read up on the best practices at the time you're writing the policy yourself.
Here's where keys come in.
SSH keys are based on...