Applies to:
Oracle Database - Enterprise Edition
Oracle Database - Standard Edition
Linux x86-64
Linux x86
Description
Starting with RedHat6, OL6, SLES11, and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management. Transparent HugePages are similar to the HugePages that have been available in previous Linux releases. The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time.
Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.
Note: This ONLY applies to the new feature Transparent HugePages, Oracle highly recommends the use of standard HugePages that were recommended for previous releases of Linux. See Document 361323.1 for additional information on HugePages.
Note: on UEK2 and above, check the existence of the /sys/kernel/mm/transparent_hugepage/ directory. If this directory does not exist, then the transparent hugepage is removed from the kernel, so there is no need to disable the transparent hugepage
Occurrence
Starting with SLES11, RHEL6, OL6, and UEK2 Kernels, Transparent HugePages are introduced and enabled by default, and this can cause node reboots and performance problems.
On UEK2 and above, the transparent hugepages is already removed if /sys/kernel/mm/transparent_hugepage/ directory directory does not exist. In that case, there is no further need to disable the transparent hugepages.
Symptoms
Node reboots.
The ocssd.log may show some of the threads are blocked (but this does not show all the time):
2013-05-01 14:30:45.255: [ CSSD][224204544]clssscMonitorThreads clssnmvKillBlockThread not scheduled for 7500 msecs
2013-05-01 14:30:46.945: [ CSSD][224204544]clssscMonitorThreads clssnmvWorkerThread not scheduled for 8030 msecs
Workaround
The information provided here for Redhat and SLES needs to be confirmed by Redhat and SLES support because Redhat and SLES support may provide a different ways to disable the transparent hugepages.
Contact Redhat and SLES support on how to disable the transparent hugepages on their respective platform.
The workaround provided here is for Redhat 6 and lower, Oracle Linux 6 and lower, and SLES 11 and lower
For Oracle Linux 7, Oracle published MOS document 2066217.1
Oracle Linux 7 - How to disable Transparent HugePages for RHCK kernel?
Also, add the following line to /etc/default/grub file:
transparent_hugepage=never
For Redhat 7, Redhat published the following note: how to disable transparent hugepages in Redhat 7.
https://access.redhat.com/solutions/1320153
For SLES 12, conatct SLES support for the instruction on how to disable the transparent hugepages
The following instruction are for Redhat 6 or lower, Oracle Linux 6 or lower, and SLES 11 or lower:
To check if the Transparent HugePages are enabled in your server execute the following:
Default/Enabled setting is [always]:
# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] never
Note 1: For RHEL kernel, the file path is different from above:
# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] never
Please modify this file accordingly.
Note 2: For UEK2 kernel, as of 2.6.39-400.116.0 Transparent Hugepages has been removed from the kernel. If it is not compiled into the kernel then /sys/kernel/mm/transparent_hugepage will not exist.
Disabled setting is [never]:
# cat /sys/kernel/mm/transparent_hugepage/enabled
always [never]
If "enabled" is NOT set to "[never]", the Transparent HugePages are being used.
You can also issue:
# grep AnonHugePages /proc/meminfo
If the output contains a line like "AnonHugepages: xxxx kB", with a value > 0kB, the kernel is using Transparent HugePages.
Because the kernel currently uses Transparent HugePages only for the anonymous memory blocks like stack and heap, the value of AnonHugepages in /proc/meminfo is the current amount of Transparent HugePages that the kernel is using.
To disable Transparent HugePages boot time either one of the following 2 methods may be used:
Add the following to the kernel boot line in /etc/grub.conf (a symlink to /boot/grub/grub.conf) and reboot the server (this is the preferred method):
transparent_hugepage=never
Once modified the line will read similar to the following example:
title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img
OR
Add the following lines in /etc/rc.local and reboot the server (this still can be done on Redhat 7 although rc.local is being deprecated):
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
Please change the file path for RHEL kernel to /sys/kernel/mm/redhat_transparent_hugepage/ accordingly.
===========================================================================
The following comment was received about SLES11 SP3. Please note that this is not tested, so this is provided only as a talking point to your sys admin who knows SLES11 SP3 and who should also contact SLES for correct information.
in SLES11 SP3 with YAST/GRUB boot loader /etc/grub.conf overrided usind YAST. parameter transparent_hugepage=never must be set in YAST-bootloader- Edit settings - in line: optional kernel parameter [before showopts].
Patches
none
History
06/05/2013 the alert is written
06/05/2013 the alert is published
06/06/2013 removed "cat /sys/kernel/mm/transparent_hugepage/enabled" from the section where we check the transparent hugepages are disabled.
06/06/2013 changed "Starting SLES11..." to 'Starting with SLES11"
06/07/2013 changed title to include "ALERT", modified to cover SIDB, corrected typo in 2nd solution, expanded the solution for disable via kernel boot line and validation via AnonHugePages.
11/26/2013 Include file path /sys/kernel/mm/redhat_transparent_hugepage/enabled for RHEL kernel and changes for UEK2 kernel
08/07/2017 Included references to Redhat 7, Oracle Linux 7, and SLES 12. Also clarfied that the instruction in this note is only for Redhat 6, Oracle Linux 6, and SLES 11 and lower versions.
References
NOTE:2066217.1 - Oracle Linux 7 - How to disable Transparent HugePages for RHCK kernel?
