Recovering systems after a major crash or outage is not an enjoyable experience. We must reboot or restore one or more servers, perform forensics to determine the root cause of the failure, and attempt to repair or replace corrupt binaries.
This is no less true on systems that rely on Patroni as their high availability solution. However, Patroni automates the more annoying portions of recovering a damaged PostgreSQL database.
Let's see how.
This recipe depends on the presence of the entire stack. Please complete all previous recipes in this chapter before continuing.
We also need a broken server. The easiest way to do this is to break it ourselves. Execute these commands on any system to simulate an unrecoverable server crash:
pkill -9 patronipkill -9 postgresfind /db/pgdata -name '*r*' -o -name '*0*' -delete