There's another point that I use to decide when I'm "done" with refactoring a particular piece of code, which is that I think that other people will be able to clearly see the pattern I've designed and will maintain the code in that pattern from then on.
Sometimes I have to write a little piece of documentation that describes the intended design of the system, so that people will follow it, but in general my theory (and this one really is just a theory – I don't have enough evidence for it yet) is that if I design a piece of code well enough, it shouldn't need a piece of documentation describing how it's supposed to be designed. It should probably be visible just from reading the code how it's designed, and it should be so obvious how you'd add a new feature within that design that nobody would ever do it otherwise. Obviously, perfectly achieving that goal would be impossible, but that's a general truth in software design:
There is no perfect design, there is only a...