Suppose that you have an internal master DNS server and a backup for that master server. The two servers are theoretically fed from the same data and so should have the same DNS zone contents, and especially they should have the same DNS zone SOAs for all zones in both of their internal and external views. They both run Bind and you use the Bind exporter , which provides the SOA values for every zone Bind is configured to be a primary or a secondary for. So you can write an alert with an expression like this:

bind_zone_serial{host="backup"}
  != on (view,zone_name)
    bind_zone_serial{host="primary"}

This is a perfectly good alert (well, alert rule), but it has lost all of the additional labels you might want in your alert. Especially, it has lost both host names. You could hard-code the host name in your message about the alert, but it would be nice to do better and propagate your standard labels into the alert. To do this you want to use one of group_left() and group_right(), but which one you want depends on where you want the labels to come from.

( Normally you have to chose between the two depending on which side has multiple matches , but in this case we have a one to one matching.)

For labels that are duplicated between both sides, the group_*() operators pick which side's labels you get, but backwards from their names. If you use group_right(), the duplicate label values come from the left; if you use group_left(), the duplicate label values come from the right. Here, we might change the backup host's name but we're probably not going to change the primary host's name, so we likely want to preserve the 'host' label from the left side and thus we use group_right():

bind_zone_serial{host="backup"}
  != on (view,zone_name)
    group_right (job,host,instance)
      bind_zone_serial{host="primary"}

One reason this little peculiarity is on my mind at the moment is that Cloudflare's excellent pint Prometheus rule linter recently picked up a new 'redundant label' lint rule that complains about this for custom labels such as 'host':

Query is trying to join the 'host' label that is already present on the other side of the query.

(It doesn't complain about job or instance, presumably because it understands why you might do this for those labels. As the pint message will tell you, to silence this you need to disable 'promql/impossible' for this rule.)

When I first saw pint's warning I didn't think about it and removed the 'host' label from the group_right(), but fortunately I actually tested what the result would be and saw that I was now getting the wrong host name.

(This is different from pulling in labels from other metrics , where the labels aren't duplicated.)

PS: I clearly knew this at some point, when I wrote the original alert rule, but then I forgot it by the time I was looking at pint's warning message. PromQL is the kind of complex thing where the details can fall out of my mind if I don't use it often enough, which I don't these days since our alert rules are relatively stable.