Oracle Linux: System Out of Memory (OOM) Caused By Incorrect Kernel Memory Tuning of vm.min_free_kbytes

Applies to:

Linux OS - Version Oracle Linux 5.1 and later
Oracle Net Services - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Cloud Infrastructure - Version N/A and later
Linux x86-64
Linux x86

Symptoms

During system initialization after boot, a kernel panic occurred after the startup of the UDEV subsystem, reporting an out of memory ("OOM") condition.

Extract from console:

INIT: version 2.86 booting
                        Welcome to Enterprise Linux Server
                        Press 'I' to enter interactive startup.
Setting clock (utc): Mon Mar 12 19:20:13 EDT 2012         [OK]
Starting udev:                                                                       [OK]
Kernel panic - not syncing: Out of memory and no killable processes...

Changes

Attempted tuning of system memory allocation parameters.

Cause

The system had 8GB of memory. However in /etc/sysctl.conf, the value of min_free_kbytes had been set to 10240000. This equates to approximately 9.7GB, more than avaiable on the system.

Solution

Reduce the setting of "min_free_kbytes" to a value more appropriate to available resource.  A suggested value for a server is 0.5% of physical memory.

/usr/share/doc/kernel-doc-2.6.18/Documentation/sysctl/vm.txt

min_free_kbytes:

This is used to force the Linux VM to keep a minimum number
of kilobytes free.  The VM uses this number to compute a pages_min
value for each lowmem zone in the system.  Each lowmem zone gets
a number of reserved free pages based proportionally on its size.

Following code shows how the default value for min_free_kbytes is caclulated in kernel:

(Note: By default the maximum value of min_free_kbytes is 65535 (KB), but actually we could set it to a higher value if necessary.)

./mm/page_alloc.c

2371 /*
2372 * Initialise min_free_kbytes.
2373 *
2374 * For small machines we want it small (128k min). For large machines
2375 * we want it large (64MB max). But it is not linear, because network
2376 * bandwidth does not increase linearly with machine size. We use
2377 *
2378 * min_free_kbytes = 4 * sqrt(lowmem_kbytes), for better accuracy:
2379 * min_free_kbytes = sqrt(lowmem_kbytes * 16)
2380 *
2381 * which yields
2382 *
2383 * 16MB: 512k
2384 * 32MB: 724k
2385 * 64MB: 1024k
2386 * 128MB: 1448k
2387 * 256MB: 2048k
2388 * 512MB: 2896k
2389 * 1024MB: 4096k
2390 * 2048MB: 5792k
2391 * 4096MB: 8192k
2392 * 8192MB: 11584k
2393 * 16384MB: 16384k
2394 */
2395 static int __init init_per_zone_pages_min(void)
2396 {
2397 unsigned long lowmem_kbytes;
2398
2399 lowmem_kbytes = nr_free_buffer_pages() * (PAGE_SIZE >> 10);
2400
2401 min_free_kbytes = int_sqrt(lowmem_kbytes * 16);
2402 if (min_free_kbytes < 128)
2403 min_free_kbytes = 128;
2404 if (min_free_kbytes > 65536)
2405 min_free_kbytes = 65536;
2406 setup_per_zone_pages_min();
2407 setup_per_zone_lowmem_reserve();
2408 return 0;
2409 }