Chapter 7 – Adding Custom Metrics
- We should first decide what we need the metric for. For example, if we need it to rank memes in search results or to calculate ad hits, we should separate it from telemetry. Assuming we store the meme download counter in a database for business logic purposes, we could also stamp it on traces or events as an attribute when the counter is updated.
From a telemetry-only standpoint, metric per meme would have high cardinality as we probably have millions of memes in the system and thousands active per minute. With some additional logic (for example, if we can ignore rarely accessed memes), we might even be able to introduce a metric with a meme name as an attribute.
I would start with traces and aggregate spans by meme name in a rich query. Even if traces are sampled, I can still calculate the estimated number of downloads, compare it between memes, and see trends.
- Usually, both, but it depends: we need incoming HTTP request...