Designing for failure
Sooner or later, our stream processors will run into trouble. A target system may go down, poison events may start arriving, or someone may make an honest human error. It will happen, and we must be prepared. We will cover monitoring and alerting in Chapter 12, Optimizing Observability. In the meantime, we need to prepare for the worst and design our stream processors to be resilient and self-healing.
We will cover idempotence and order tolerance in Chapter 5, Turning the Cloud into the Database. In the next section, we will look at backpressure, rate limiting, and poison events. And when we do get into trouble, we will emit fault events.
Backpressure and rate limiting
Backpressure is an important characteristic of stream processing. A stream processor should not overload its target resource. In the best case, the target will throw throttling errors, and we will have low throughput, as the stream processor wastes time and resources. In the worst case...