As we have been progressing through the chapters and solving many issues for our recent employer, we have also been gaining their confidence in our abilities. Recently, we were even placed on the on call rotation, which means that if issues occur after hours an alert will be sent to our phone by SMS.
Of course, the first night of being on call we get an alert; the alert is not a good one.
ALERT: blog.example.com is no longer responding to ICMP Pings
When we were added to the on call rotation, our team lead informed us that any major incident that occurs after hours must also have an RCA performed. The reason for this is so that others in our group can learn and understand what we did to resolve the issue and how to prevent it from happening again.
As we discussed earlier one of the key components to a useful RCA is listing when things happen. A major event in our timeline is when we received the alert; based on our SMS message we can see that we received the alert on July...