There are two kinds of breaks:
Your test suite doesn't compile after refactoring
Your test suite still compiles, but no longer turns green
Tests would stop compiling if you change the external API the test is relying on. If that is part of your refactoring, then the compilation error is okay and expected. If you're changing the internal implementation of your functionality, then the test should not be affected and still compile.
A test that fails to compile after refactoring could be a sign of testing in a too granular level and testing implementation details instead of testing the function as a whole.
If your test suite still compiles after refactoring but your test fails, you have the same root problem. You're testing on a too granular abstraction level. This is a common...