A common technique that can make your builds faster is to use a single compilation unit build, or unity build. It won't speed up every project, but it may be worth a shot if there's plenty of code in your header files. Unity builds work by just including all your .cpp files in one translation unit. Another similar idea is to use pre-compiled headers. Plugins such as Cotire for CMake will handle both of these techniques for you. CMake 3.16 also adds native support for unity builds, which you can enable either for one target, set_target_properties(<target> PROPERTIES UNITY_BUILD ON), or globally by setting CMAKE_UNITY_BUILD to true. If you just want PCHs, you might want to take a look into CMake 3.16's target_precompile_headers.
If you feel like you are including too much in your C++ files, consider using a tool named include-what-you-use to tidy them up. Preferring forward declaring types and functions to including header files can also go a long...