All production servers are normally installed using kickstart files. Unfortunately, the ks file had a bug that it didnt add a swap partition if there was only one logical or physical disk.
Once the database was setup without swap, the configuration was sized such that all processes fit into memory. But we started seeing OOM killing processes due to lack of memory. Though under normal conditions we had 1-2 GB of free memory.
Unfortunately, OOM at times picked to kill mysql. We were running 500GB+ database having myisam tables. We ended up having to repair the tables every time OOM felt like killing mysqld :(. Adding a swap device might help us fix the issue, but for that we might have to resize the logical/physical partitions, which wasn’t cool. Then this following idea popped up, create a file, format it as swap and add it as swap device dynamically and also add it /etc/fstab to survive reboot.
dd if=/dev/zero of=/swapfile bs=1024 count=16777216 # creates a 16GB file
mkswap /var/swap/swapfile # create swap fs on the file
swapon /var/swap/swapfile # add the swap file as swap device
/var/swap/swapfile swap swap defaults 0 0 #add this entry in /etc/fstab
You are all set for swapping. No more process kills by OOM. But of course adding swap device doesn’t save you from leaking process which also eat up the swap space and die with the swap allocation failed error. Any better ideas are welcome. I can try it and post an update.