At some point, be it through network disruptions, natural disasters, or any number of other factors outside your control, it's possible for the data on a replica set member to become stale. What this means is that the data is so radically out of date that it is impossible for it to catch up through the normal synchronization process. If you have determined that network communications are not at fault, it is time to consider performing a manual resynchronization (https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/).
If in the process of monitoring the replica set status, you notice one of the replica members has an increase in replication lag, you need to flag that server for closer inspection. If the lag time continues to increase, first check all network connections, operating system logs, and router logs. It's possible the problem is physical—perhaps even something as simple as a bad cable.
There are two primary...