lol rip

Result: Clean latency metrics that automatically filter out low-traffic noise. Your dashboards stay focused on clusters that matter.

The earlier graph is now trimmed, showing just the series that matter.

Shoutout to Datadog's query engine team for enabling this kind of creative solution! 🙌

5/5

The actual query looks like this:

cockroachdb.sql.conn.latency * (cutoff_min(cockroachdb.sql.new_conns, 5) / cutoff_min(cockroachdb.sql.new_conns, 5))

When new_conns < 5, the series becomes null and nullifies the latency graph.

4/n

The solution:
1. Use cutoff_min() on your modulating metric (the connection rate, in the example)
2. Multiply your target metric by modulating_metric/modulating_metric

This creates a 1 (keep) or null (filter) multiplier! 🪄

3/n

An example problem: Monitoring SQL connection latency across a fleet of CockroachDB clusters gets noisy when some clusters have quiet workloads. Low-traffic clusters skew dashboards 📊

How do we monitor SQL connection latency, but ONLY when connection rates are high enough to be meaningful?

2/n

Probably, but you'd just have to do everything yourself which may or may not appeal or ultimately be cost effective to maintain.

The ish is that the cheap log tier (flex) doesn't support arbitrary metric creation. The more expensive one (standard, what they used to have exclusively) supports this in full. In practice you can use a mixture of the two tiers, but I do hope they improve this behavior

I'm not sure what you mean by visualizing them over time as a trace, but I think so? For example, you could write a filter on a "trace id" attribute and show log events with a given trace id.

No need to select dimensions, no.

Deriving SLOs and metrics - yes, ish...

Datadog's flex logs system provides a reasonable method in my experience - filter / group by / aggregations across logs with arbitrary attributes, plus graphing on top of that output. Or did you mean something more specific?

bullet journal of best bullet journal illustrations type beat

once a scene starts, it's hard to de-scene it! I'm clearly part of the problem.

goodhang.org/p/scene-me-u...

yes why is that

rip

your what