1.Alerting with Prometheus is separated in to two parts

1-1Alerting rules

  • Alerting rules is a file that formatted by yml,it's define some exp in to the yaml file.
  • Alerting rules is in Prometheus servers send alerts to an alert manager and alert manager managed those alerts.

the rule document in there: the rules doc

1-2Alerting Manager

  • AlertingManager is a command line tool and the command line tool flags has those functions

    • configure immutable system parameters
    • configuration file defines inhibition rules
    • configuration notification routing
    • configuration notification receivers
    • router editor
  • Alerting notifications steps

    • Setup and configure the alert manager
    • Configure Prometheus to talk to the alert manager
    • Create alerting rules in Prometheus

1-2-1 Configs

  • config.file: specify which configuration file to load./alertmanager --config.file=alertmanager.yml
    • the file written in the YAML format and defined by the scheme described below
      • <duration>: a duration matching the regular expression [0-9]+(ms|[smhdwy])
      • <labelname>: a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
      • <labelvalue>: a string of unicode characters
      • <filepath>: a valid path in the current working directory
      • <boolean>: a boolean that can take the values true or false
      • <string>: a regular string
      • <secret>: a regular string that is a secret, such as a password
      • <tmpl_string>: a string which is template-expanded before usage
      • <tmpl_secret>: a string which is template-expanded before usage that is a secret
      • <int>: an integer value

there is the simple file and some configs

1-2-2 Cores

  • Grouping

    • Grouping categorizes alerts of similar nature into a single notification.
  • Inhibition

    • Inhibition is a concept of suppressing notifications for certain alerts if certain other alerts are already firing
  • Silences

    • Silences are a straightforward way to simply mute alerts for a given time.
    • A silence is configured based on matchers, just like the routing tree.
    • Silences are configured in the web interface of the Alert manager.

1-2-3 Notification template Reference

1-2-3-1 Data structures


ReceiverstringThe notification will be send to
StatusstringDefined as firing if at least one alert if firing.otherwise resolved
AlertsAlertList of all alert objects in this group
GroupLabelskvThe labels these alerts were grouped by
CommonLabelskvThe labels common to all of the alerts
CommonAnnotationskvset of common annotations to all of the alerts. Used for longer additional strings of information about the alert
ExternalURLstringBacklink to the alert manager that send the notification


StatusstringDefines whether or not the alert is resolved or currently firing
LabelskvA set of labels to be attached to the alert
AnnotationskvA set of annotations for the alert
StartsAttime.TimeThe time the alert started firing
EndsAttime.TimeOnly set if the end time of an alert is know
GeneratorURLstringA backlink which identifies the causing entity of this alert


SortedPairs-Pairs(list of key/value string pairs)Return s a sorted list of key/value pairs
Remove[] stringkvReturns a copy of the key/value map without the given keys.
Names-[]stringReturns the names of the label names in the labelset
Values-[]stringReturns the list of values in the labelset


titlestringstrings.Title,capitalises first character of each word
matchpattern,stringRegexp.MatchString.Match a string using Regexp
reReplaceAllpattern,replacement,textRegexp.ReplaceAllString Regexp substitution,unanchored
joinsep string,s []stringstrings.join
safeHtmltext stringhtml/template.HTML
stringSlice...stringReturns the passed strings as a slice of strings


别云剑本星 智多星本剑