8.14. Configuring retention policy for Prometheus metricsΒΆ

The Prometheus service used for monitoring the cluster runs and stores its data on the management node. By default, Prometheus metrics are stored for seven days. This retention period can be insufficient for troubleshooting purposes. You can increase this period in the Prometheus configuration file by doing the following:

  1. On the management node, open the /etc/sysconfig/prometheus file to edit, set the needed retention period for the STORAGE_RETENTION option, and then save your changes. For example:

    STORAGE_RETENTION="--storage.tsdb.retention.time=30d"

  2. Restart the Prometheus service:

    systemctl restart prometheus.service

If high availability is enabled in the storage cluster, repeat these steps for the other two management nodes.

However, with a long retention period, the root partition where the data is stored may run out of free space. To prevent this, you can define the maximum size for the Prometheus metrics. The oldest data will be removed first. To change the time retention policy to the size retention policy, do the following:

  1. On the management node, open the /etc/sysconfig/prometheus file to edit, change the flag for the STORAGE_RETENTION option, and then save your changes. For example:

    STORAGE_RETENTION="--storage.tsdb.retention.size=10GB"

  2. Restart the Prometheus service:

    systemctl restart prometheus.service

If high availability is enabled in the storage cluster, repeat these steps for the other two management nodes.

Oct 06, 2020
Print