Retention isn't applied

What it means#

A log, metric or trace store doesn’t run with the number of days the installation’s settings give it. A new installation keeps logs for 30 days, metrics for 90 days and traces for 7 days, and you set the days of each in the installation’s settings. A store that keeps more days than set uses more disk, and one that keeps fewer deletes records you expect to have.

What raises it#

It shows at once when the store stopped before it started with the days set, and after two minutes when the store still runs with other days. It’s an error while the store keeps fewer days than set.

What to check#

On a hub server, in the stack’s folder, run the checks, whose retention lines give the days set and the days each store runs with:

$ docker compose run --rm infra weikio-backend doctor

Read the store’s own log, for example the log store’s:

$ docker compose logs --tail 50 victoria-logs

How to fix it#

Restart the store so it starts with the days set:

$ docker compose restart victoria-logs

The metric store is victoria-metrics and the trace store victoria-traces. The backend keeps what the hub receives while a store restarts and sends it on once the store answers.