Skip to content

Entries tagged "oom".

Protect KVM processes from OOM killer

9th July 2015
While running clouds on Linux KVM hypervisors it may happen that some of your virtual machines processes get killed by the OOM killer in order to free up memory.

Depending on your situation, the OOM killer may be instructed not to kill certain processes; but if you go this way make sure you know what you are doing and how resources are used.

So, to proceed with protecting KVM processes from out of memory scenarios, we need to run a few commands:
1 - determine the PID of the processes, we can use pgrep for this
2 - protect them from OOM killer by changing the PIDs oom_adj value to -17 (OOM_DISABLE); if you use a 3.x+ kernel then you need to change oom_score_adj to -1000 instead as oom_adj is deprecated

This can be wrapped up in a one-liner such as this:
for PID in $(pgrep qemu-kvm); do echo -17 > /proc/$PID/oom_adj; done

That would work in CentOS 6, but if you are on a newer kernel than that (say 3.x like the one in CentOS 7) then use this:
for PID in $(pgrep qemu-kvm); do echo -1000 > /proc/$PID/oom_score_adj; done

You might want to double check your KVM processes run as qemu-kvm, that's the program's name in CentOS, it may differ in other distributions.

If you do not want to do this manually every time a VM is created you can simply create a cron job to do it for you every X minutes; if you spin up instances very often then you may set it as frequent as 1 minute:
echo '*/1 * * * * root for PID in $(pgrep qemu-kvm); do echo -1000 > /proc/$PID/oom_score_adj; done' > /etc/cron.d/oomprotect

If you run into memory usage issues, do have a look at KSM as it can help optimise memory utilisation (but at the cost of extra CPU usage).
Tags: cloud, kvm, linux, oom. | No comments