In the context of a stream of temporal information (such as a log file), the notion of something being statistically rare (occurs at a low frequency) is paradoxically both intuitive and hard to understand. If I were asked, for example, to trawl through a log file and find a rare message, I might be tempted to label the first novel message that I saw as a rare one. But what if practically every message was novel? Are they all rare? Or is nothing rare?
In order to define rarity to be useful in the context of a stream of events in time, we need to agree that the declaration of something as being rare must take into account the context in which it exists. If there are lots of other routine things and a small number of unique things, then we can deem the unique things rare. If there are many unique things, then we will deem that nothing is rare.
When...