There is no such thing as too much refactoring, but there are stages in the development process that act as an impetus for refactoring. Here are a few of them.
When you do something the first time, you just do it to get it done. When doing the same thing a second time, you do the same thing again, though you squirm at having to repeat it. When you do it a third time, you start refactoring.
When you are adding a new feature to someone else's unclear code, it is better to start refactoring it before adding the feature. Refactoring helps you understand the unclear code. Refactored code facilitates the smooth addition of new features.
Bugs normally live in smelly code. They both go hand in hand. Once you start refactoring your code, most of the bugs just jump out into the open.