Let's assume that you have chosen a good algorithm and implemented it without too much regard for optimization, as is commonly the case with first attempts. Should you invest the time to optimize it? Performance optimization can be a very costly activity. You must not try to optimize your code unless you must. Your time is valuable, and it's probably better spent doing something else.
Let's say that for some reason, you must make your implementation faster. The first thing you must decide on is how fast is fast enough. Is your algorithm required to simply finish within a couple of hours instead of a couple of days, or do you need to come down to microsecond levels? Is this an absolute requirement or should you simply do the best job you can within a specific time frame? These are important questions that you must consider before optimizing...