Prometheus Relabel Rules and the ‘action’ Parameter

Today I want to talk about learning about the action parameter in the relabel_config and metric_relabel_config elements in Prometheus. This was an epiphany I had when searching for how to dig substrings out the __meta_* label names as returned from service discovery (hint, use action: labelmap)

Relabel configs are composed of the following:

• source_labels
• separator (default ‘;’)
• target_label – mandatory for replace actions. More on this below
• regex (default ‘.*’)
• modulus
• replacement (default ‘$1’)
• action (default ‘replace’)

Some of these elements have defaults, others are required based on the value of the action element.

When first learning about relabel configs in Prometheus you will encounter many examples that looks something like this:

relabel_configs: 
    - source_labels: [__meta_kubernetes_role] 
      action: keep 
      regex: (?:apiserver|node) 
    - source_labels: [__meta_kubernetes_role] 
      target_label: job 
      replacement: kubernetes_$1 
    - action: labelmap 
      regex: __meta_kubernetes_node_label_(.+)

Every example I’ve found the stanza starts with source_labels as the first entry in the list of elements. As it states in the docs:

action defaults to ‘replace’

After reading the code (trying to find labelmap) it occurred to me that action is really the star of the show.

There are many different actions available to the relabel configs (lifted from the docs):

• replace: Match regex against the concatenated source_labels. Then, set target_label to replacement, with match group references
   (${1}, ${2}, …) in replacement substituted by their value. If regex
   does not match, no replacement takes place.
• keep: Drop targets for which regex does not match the concatenated source_labels.
• drop: Drop targets for which regex matches the concatenated source_labels.
• hashmod: Set target_label to the modulus of a hash of the concatenated source_labels.
• labelmap: Match regex against all label names. Then copy the values of the matching labels to label names given by replacement with match group references (${1}, ${2}, …) in replacement substituted by their value.
• labeldrop: Match regex against all label names. Any label that matches will be removed from the set of labels.
• labelkeep: Match regex against all label names. Any label that does not match will be removed from the set of labels.

From a neophyte’s perspective perhaps all the examples for replace relabel rules should start with action, even if it’s redundant. To rewrite the above example:

relabel_configs: 
    - action: keep 
      source_labels: [__meta_kubernetes_role] 
      regex: (?:apiserver|node) 
    - action: replace 
      source_labels: [__meta_kubernetes_role] 
      target_label: job 
      replacement: kubernetes_$1 
    - action: labelmap 
      regex: __meta_kubernetes_node_label_(.+)

By leading with action it is crystal clear what is happening.