Book Image

Real-World SRE

By : Pavlos Ratis, Nat Welch
Book Image

Real-World SRE

By: Pavlos Ratis, Nat Welch

Overview of this book

Real-World SRE is the go-to survival guide for the software developer in the middle of catastrophic website failure. Site Reliability Engineering (SRE) has emerged on the frontline as businesses strive to maximize uptime. This book is a step-by-step framework to follow when your website is down and the countdown is on to fix it. Nat Welch has battle-hardened experience in reliability engineering at some of the biggest outage-sensitive companies on the internet. Arm yourself with his tried-and-tested methods for monitoring modern web services, setting up alerts, and evaluating your incident response. Real-World SRE goes beyond just reacting to disaster—uncover the tools and strategies needed to safely test and release software, plan for long-term growth, and foresee future bottlenecks. Real-World SRE gives you the capability to set up your own robust plan of action to see you through a company-wide website crisis. The final chapter of Real-World SRE is dedicated to acing SRE interviews, either in getting a first job or a valued promotion.
Table of Contents (16 chapters)
Real-World SRE
Contributors
Preface
Other Books You May Enjoy
Index

Carrying out incident analysis


Incident analysis is complicated. Sometimes you know the answer right away and sometimes it takes hours or days of research. You should approach the system like a paranoid Sherlock Holmes. Start at the scene of the crime and then dig deep into every aspect. Be wary of your preconceived notions, be aware of red herrings, and always test your hypotheses.

Note

A red herring is something that is misleading or distracting from the task at hand. It is often an attractive answer or problem unrelated to the actual issue.

The scene of a crime for an outage is often the thing you rolled back. It may be a bad config or buggy code. Was the outage caused by the change your team was trying to deploy or was it caused by a system interacting with that code? You can then start reading through the code that changed or the code that interacts with the code you deployed. If there wasn't a rollback, then you've probably already figured out the issue, as you will have had to figure...