G HugePages

G.1.1 Tuning SGA With HugePages

Without HugePages, the operating system keeps each 4 KB of memory as a page. When it allocates pages to the database System Global Area (SGA), the operating system kernel must continually update its page table with the page lifecycle (dirty, free, mapped to a process, and so on) for each 4 KB page allocated to the SGA.

With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, because each page table entry is pointing to pages from 2 MB to 256 MB.

Also, the kernel has fewer pages whose lifecycle must be monitored. For example, if you use HugePages with 64-bit hardware, and you want to map 256 MB of memory, you may need one page table entry (PTE). If you do not use HugePages, and you want to map 256 MB of memory, then you must have 256 MB * 1024 KB/4 KB = 65536 PTEs.

HugePages provides the following advantages:

• Increased performance through increased TLB hits

• Pages are locked in memory and never swapped out, which provides RAM for shared memory structures such as SGA

• Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (for example, SGA)

• Less bookkeeping work for the kernel for that part of virtual memory because of larger page sizes

G.1.2 Configuring HugePages on Linux

Complete the following steps to configure HugePages on the computer:

1. Run the following command to determine if the kernel supports HugePages:

   $ grep Huge /proc/meminfo
   

2. Some Linux systems do not support HugePages by default. For such systems, build the Linux kernel using the CONFIG_HUGETLBFS and CONFIG_HUGETLB_PAGE configuration options. CONFIG_HUGETLBFS is located under File Systems and CONFIG_HUGETLB_PAGE is selected when you select CONFIG_HUGETLBFS.

3. Edit the memlock setting in the /etc/security/limits.conf file. The memlock setting is specified in KB, and the maximum locked memory limit should be set to at least 90 percent of the current RAM when HugePages memory is enabled and at least 3145728 KB (3 GB) when HugePages memory is disabled. For example, if you have 64 GB RAM installed, then add the following entries to increase the maximum locked-in-memory address space:

   *   soft   memlock    60397977
   *   hard   memlock    60397977
   

   You can also set the memlock value higher than your SGA requirements.

4. Log in as oracle user again and run the ulimit -l command to verify the new memlock setting:

   $ ulimit -l
   60397977
   

5. Run the following command to display the value of Hugepagesize variable:

   $ grep Hugepagesize /proc/meminfo
   

6. Complete the following procedure to create a script that computes recommended values for hugepages configuration for the current shared memory segments:

   1. Create a text file named hugepages_settings.sh.

   2. Add the following content in the file:

      #!/bin/bash
      #
      # hugepages_settings.sh
      #
      # Linux bash script to compute values for the
      # recommended HugePages/HugeTLB configuration
      #
      # Note: This script does calculation for all shared memory
      # segments available when the script is run, no matter it
      # is an Oracle RDBMS shared memory segment or not.
      # Check for the kernel version
      KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
      # Find out the HugePage size
      HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
      # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
      NUM_PG=1
      # Cumulative number of pages required to handle the running shared memory segments
      for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
      do
         MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
         if [ $MIN_PG -gt 0 ]; then
            NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
         fi
      done
      # Finish with results
      case $KERN in
         '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
                echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
         '2.6'|'3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
          *) echo "Unrecognized kernel version $KERN. Exiting." ;;
      esac
      # End
      

   3. Run the following command to change the permission of the file:

      $ chmod +x hugepages_settings.sh
      

7. Run the hugepages_settings.sh script to compute the values for hugepages configuration:

   $ ./hugepages_settings.sh
   

   Note:

   Before running this script, ensure that all the applications that use hugepages run.

8. Set the following kernel parameter, where value is the HugePages value that you determined in step 7:

   # sysctl -w vm.nr_hugepages=value
   

9. To ensure that HugePages is allocated after system restarts, add the following entry to the /etc/sysctl.conf file, where value is the HugePages value that you determined in step 7:

   vm.nr_hugepages=value
   

10. Run the following command to check the available hugepages:

    $ grep Huge /proc/meminfo
    

11. Restart the instance.

12. Run the following command to check the available hugepages (1 or 2 pages free):

    $ grep Huge /proc/meminfo
    

    Note:

    If you cannot set your HugePages allocation using nr_hugepages, then your available memory may be fragmented. Restart your server for the Hugepages allocation to take effect.

G.1.3 Restrictions for HugePages Configurations

HugePages has the following limitations:

• You must unset both the MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters. For example, to unset the parameters for the database instance, use the command ALTER SYSTEM RESET.

• Automatic Memory Management (AMM) and HugePages are not compatible. When you use AMM, the entire SGA memory is allocated by creating files under /dev/shm. When Oracle Database allocates SGA with AMM, HugePages are not reserved. To use HugePages on Oracle Database 12c, You must disable AMM.

• If you are using VLM in a 32-bit environment, then you cannot use HugePages for the Database Buffer cache. You can use HugePages for other parts of the SGA, such as shared_pool, large_pool, and so on. Memory allocation for VLM (buffer cache) is done using shared memory file systems (ramfs/tmpfs/shmfs). Memory file systems do not reserve or use HugePages.

• HugePages are not subject to allocation or release after system startup, unless a system administrator changes the HugePages configuration, either by modifying the number of pages available, or by modifying the pool size. If the space required is not reserved in memory during system startup, then HugePages allocation fails.

• Ensure that HugePages is configured properly as the system may run out of memory if excess HugePages is not used by the application.

• If there is insufficient HugePages when an instance starts and the initialization parameter use_large_pages is set to only, then the database fails to start and an alert log message provides the necessary information on Hugepages.

G.1.4 Disabling Transparent HugePages

Transparent HugePages memory is enabled by default with Red Hat Enterprise Linux 6, SUSE 11, and Oracle Linux 6 with earlier releases of Oracle Linux Unbreakable Enterprise Kernel 2 (UEK2) kernels. Transparent HugePages memory is disabled by default in later releases of UEK2 kernels.

Transparent HugePages can cause memory allocation delays at runtime. To avoid performance issues, Oracle recommends that you disable Transparent HugePages on all Oracle Database servers. Oracle recommends that you instead use standard HugePages for enhanced performance.

Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.

To check if Transparent HugePages is enabled run one of the following commands as the root user:

Red Hat Enterprise Linux kernels:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

Other kernels:

# cat /sys/kernel/mm/transparent_hugepage/enabled

The following is a sample output that shows Transparent HugePages is being used as the [always] flag is enabled.

[always] never

To disable Transparent HugePages perform the following steps:

1. Add the following entry to the kernel boot line in the /etc/grub.conf file:

   transparent_hugepage=never
   

   For example:

   title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
           root (hd0,0)
           kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
           initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img