A replica of a service might be in the pending state. There might be quite a few reasons for that, and we won't go through all of them. Instead, we'll explore one of the most common causes behind having a replica pending deployment. A service might have memory reservation that cannot be fulfilled with the current cluster.
Let's say that a service has memory reservation set to 3 GB. All the replicas of that service are running but, at one moment, the system scales that service by increasing the number of replicas by one. What happens if none of the nodes have 3 GB of unreserved memory? Docker Swarm will set the status of the new replica to pending, hoping that 3 GB will be available in the future.
Such a situation might not be discovered with any of the existing alerts. The used memory of each of the nodes might be below...