Skip to main content

Notifications

Integrationsโ€‹

The integration page allow you to create new integration and modify or delete the existing ones.

info

Bleemeo creates default Integrations that allow you to create a notification rule by email, mobile application or webhook.

Create Integrationโ€‹

You can create new Integration with the + Integration button. The first thing to do on the create new integration is to select the type of integration you need:

Finally you need to write the name of your integration who will be used to create notification rule. If you want to create a new notification rule with this integration go to notifications page.
Once done click the Create button. Your new integration is added at the end of the list. It is enabled by default.

Actions on Integrationโ€‹

You can enable or disable an integration with the toggle button. The 2 small icons on the right allow you to edit or delete an integration.

On this page you can see the notifications that you have configured.

Bleemeo creates a default notification that notify you by email of all errors.

default notification

Create Notification Ruleโ€‹

You can create new notification with the + Notification button.

1st Step: Scopeโ€‹

first view

The first thing to do on the create new notification is to select the scope of notification rule, you can:

  • Be notified on any server(s)

  • Be notified on a specific server

  • Be notified on a group of servers (Server group or Tag)

The second part lets you choose between two triggers for this notification rule:

  • With agent metrics (classic thresholds) or if agent lost its connection to Bleemeo Cloud

  • Use Prometheus Query Language to define thresholds (PromQL)

2nd Step: Problem with metricโ€‹

second view metrics

You have three possibilities of problem with metric:

  • Be notified for any problem on selected server(s)

  • Be notified for specified problem on selected server(s)

  • Be notified when selected server(s) lost connection (This option doesn't exists if you have selected monitor in the previous view)

info

If you select the second option (specific metric), you will be able to save only if this metric has thresholds. It's possible to edit thresholds of selected metric directly in this modal (screenshot above). This is possible only if you have selected a specific agent, or a Server Group in the previous view.

2nd Step: Problem with PromQLโ€‹

second view promql

To create an alerting rule with PromQL, you can use our documentation.

You have a name to choose, this name will be used as the alert name in status dashboard and notifications.

You can add warning and/or critical PromQL and configure the delay. The delay corresponds to the time during which the threshold of the PromQL must be exceeded to change its status.

3rd Step: Targetsโ€‹

third view promql

Finally you need to select who will be notified and how. You can choose directly targets and/or contacts group. If you want to create a new integration, you can by clicking on + More.

If you have selected a contacts group that doesn't exist, in the next view you should edit this contacts group to add targets.

Set a delay for repetitionโ€‹

You can add a repeat delay to a notification rule. This delay is in minutes, it is 0 by default, which means that the event will be notified only once, without repetition.

If you change it to 10 for example, you will receive a notification every 10 minutes for an event until it becomes "OK" again.

You can stop a repetition of an event from two views, in the list of events and in the status dashboard. If the event returns "OK" and returns in error, the repetition starts again.

Set time constraints on Targetโ€‹

You can choose the days and times when you want to receive notifications from each target.

For this example, notifications will be sent only Monday to Friday between 8:00 am and 6:00 pm (UTC).

constraint notification

Or this example:

  • Monday to Friday between 8:00 am and 6:00 pm (UTC) on email
  • Week night on mobile application
  • No notifications between Friday 6:00 pm to Monday 8:00 am

complex notifications

Once done click the Save button. Your new notification rule is added at the end of the list. It is enabled by default.

PromQLโ€‹

You can find documentation on Prometheus Query Language on the web:

PromQL Documentation

PromQL Cheat Sheet

PromQL Example

Alerting Rule Creation with PromQLโ€‹

To create an Alerting Rule with PromQL, go to notification page and create a new notification rule.

Select a scope:

  • If you select Any server, your PromQL will run in Bleemeo Cloud infrastructure, and will be executed for all of your servers.
  • If you select A specific server, your PromQL will run on your Glouton (Agent Bleemeo), and will be executed only for this server.
  • If you select A group of servers, your PromQL will run on each Glouton of this group, and will be executed only for agents of this group.

and select Use PromQL to define conditions to trigger alarm to use PromQL.

first view with promql

You have to choose a name, this name will be used as alert name in status dashboard and notifications.

You can add warning and/or critical PromQL and configure the delay. The delay corresponds to the time during which the threshold of the PromQL must be exceeded to change its status.

second view promql

info

For a PromQL to trigger alerts, it must at least have the format: metric_name > thresholds

Example: Cassandraโ€‹

We have several Cassandra server running on our Kubernetes, we want to trigger a warning alert if the sum of the cpu_used of the containers exceeds 50% during 300 seconds and trigger a critical alert if it exeeds during 600 seconds.

info

It's possible to put the same PromQL for warning and critical but just with different delays.

PromQL

sum(container_cpu_used{item=~"k8s_cassandra_cassandra.*"}) > 50

~"k8s_cassandra_cassandra.*" This allows you to find all the containers whose item name begins with k8s_cassandra_cassandra

promql cassandra

Example: Workers Loadโ€‹

We have a Kubernetes cluster with 3 workers, we want to trigger a warning alert if the sum of the system_load of 3 workers exceeds 15 during 300 seconds and trigger a critical alert if exceed 30 during 300 seconds.

PromQL Warning:

sum(system_load1{instance=~"host-fqdn-1|host-fqdn-2|host-fqdn-3"}) > 15

PromQL Critical:

sum(system_load1{instance=~"host-fqdn-1|host-fqdn-2|host-fqdn-3"}) > 30

promql load

Example: Multi Metricsโ€‹

We want to create a notification rule on any server, we want to create an alerting rule which will trigger a warning alert if on a server we have the cpu_used > 50 and the mem_used_perc > 80 for 120 seconds, and which will trigger a critical alert if on a server we have the cpu_used > 80 and the mem_used_perc > 90 for 300 seconds.

info

If you want use multi metrics in your PromQL, all labels of each metrics must be the same. Else you can filter with on (name_label) like the example below.

PromQL Warning:

cpu_used > 50 and on (instance) mem_used_perc > 80

PromQL Critical:

cpu_used > 80 and on (instance) mem_used_perc > 90

promql multi metrics

Silenceโ€‹

The silence is a setting available on an agent or a metric that allows you not to receive any notification from the agent or the metric during the defined period. You can create a new silence, edit or delete the existing ones, for that you have to click on ๐Ÿ””.

This an example of a dashboard view and you can see the silence button to the right of each metric : silence logo

To create silenceโ€‹

To create a silence you click on the ๐Ÿ”” of the chosen metric or the agent for the silence after that, a page like this appears. You need to fill in the Reason field and setup the start and the end date of the silence :

create silence

To edit or delete silenceโ€‹

To edit or delete a silence you click on the ๐Ÿ”” of the chosen metric or agent and after that you click on an event in the calendar.

  • If you want to edit you can update the reason, the start or end date of the silence and after to save click on the Edit button.
  • If you want delete you can click on Delete button.

This is an example:
create silence