Saturday, December 4, 2010

openwrt crashes due to OOM killer

I have an openvpn tunnel running between two OpenWRT-powered routers. One of them is running Kamikaze 7.09 and crashes all the time when transferring lots of data at once. It was getting really annoying (for me and them) having to call the site and have it manually rebooted. I finally discovered that it appears to be dying because it runs out of memory and the OOM killer terminates all the important processes. It'd be nice if the OOM killer would just reboot the thing, but the kernel in Kamikaze 7.09 for this router doesn't support that (I think newer ones do). So I wrote the following watchdog script and installed it in root's crontab to run every 4 minutes. So far, the router hasn't crashed permanently. When memory gets low, it just reboots! Hopefully no more calls to this site...


#!/bin/sh

F="`free | grep Mem | cut -c 37-45`"
if [[ $F -lt 500 ]]; then
logger "Memory Low ($F) - rebooting..."
sleep 5
/sbin/reboot
fi

No comments: