8.13. Configuring Retention Policy for Prometheus MetricsΒΆ

The Prometheus service used for monitoring the cluster runs and stores its data on the management node. By default, Prometheus metrics are stored for 7 days. This retention period can be insufficient for troubleshooting purposes. You can increase this period in the Prometheus configuration file as follows:

  1. On the management node, open the /etc/sysconfig/prometheus file to edit, set the needed retention period for the STORAGE_RETENTION option, and save your changes. For example:

    STORAGE_RETENTION="--storage.tsdb.retention.time=30d"

  2. Restart the Prometheus service:

    systemctl restart prometheus.service

In case high availability is enabled in the storage cluster, repeat these steps for the other two management nodes.

However, with a long retention period, the root partition where the data is stored may run out of free space. To prevent this, you can define the maximum size for the Prometheus metrics. The oldest data will be removed first. To change the time retention policy to the size retention policy, do as follows:

  1. On the management node, open the /etc/sysconfig/prometheus file to edit, change the flag for the STORAGE_RETENTION option, and save your changes. For example:

    STORAGE_RETENTION="--storage.tsdb.retention.size=10GB"

  2. Restart the Prometheus service:

    systemctl restart prometheus.service

In case high availability is enabled in the storage cluster, repeat these steps for the other two management nodes.

Jul 21, 2020
Print