So, in brief, refactoring is an organizational process that you go through in order to make production possible. If you aren't going toward production when you refactor, you're going to run into lots of different kinds of trouble. I can't even tell you all of the things that are going to go wrong, but they're going to happen. On the other hand, if you just try to produce a system and you never reorganize it, you're going to get yourself into such a mess that production becomes difficult or impossible.
So both of these things have to be done – you must produce a product, and you must organize the system in such a way that the product can be produced quickly, reliably, simply, and well. If you leave out organization, you won't get the product that you want, and if you leave out production, then there's literally no reason to even be doing the refactoring in the first place.
Yes, it's nice to water the lawn, but let's put out some fires, first.
-Max