Redis and Monit Alerting for Double Memory Usage

Tags
Status
Done

When monitoring redis memory with monit, on occasion there are false positive alerts about memory usage. Stating that there is double the memory usage that the limit specificed, while the Redis instance has a memorymax setting of 1500.

[EST Jan  9 14:25:01] warning  : 'redis-server' total mem amount of 3.3 GB matches resource limit [total mem amount > 1.7 GB]

Here’s the monit config

# gridpane monit redis v2
check process redis-server
    with pidfile "/var/run/redis/redis-server.pid"
    start program = "/usr/sbin/service redis-server start"
    stop program = "/usr/sbin/service redis-server stop"
    if cpu usage > 240% for 10 cycles then exec "mon REDIS CPU_HOT warning" AND repeat every 10 cycles
    if cpu usage > 320% for 5 cycles then exec "mon REDIS CPU_RESTART error"
    if totalmem > 1700 MB for 5 cycles then exec "mon REDIS MEM_HIGH warning" AND repeat every 5 cycles
    if totalmem > 1800 MB for 10 cycles then exec "mon REDIS MEM_RESTART error"
    if failed host 127.0.0.1 port 6379 then restart
    if children > 255 for 5 cycles then restart
    if 5 restarts within 5 cycles then exec "mon REDIS FAILED error" AND repeat every 1 cycles

Looking at atop, you’ll see two processes.

image

Digging further reveals that two redis-server processes are spawned at some point, this occurs when redis is configured to save and is using the rdb format. Which results in spawning a second process which stays alive enough for monit to complain.

311844:M 09 Jan 2023 14:25:00.019 * 10 changes in 300 seconds. Saving... 311844:M 09 Jan 2023 14:25:00.052 * Background saving started by pid 739360 739360:C 09 Jan 2023 14:25:06.876 * DB saved on disk 739360:C 09 Jan 2023 14:25:06.903 * RDB: 7 MB of memory used by copy-on-write 311844:M 09 Jan 2023 14:25:06.968 * Background saving terminated with success

You can read more about Redis persistence configuration and using RDB

https://redis.io/docs/management/persistence/#:~:text=as snapshotting.-,How it works,-Whenever Redis needs

If Redis is restarted at a specific time that aligns with monit's check time it's possible that an alert can be sent out. Since the Redis pid doesn’t change, and since the config is totalmem, the secondary spawned instance is counted towards totalmem. There is the option of using AOF, which wouldn't require a restart.