Situation:
When investigating one clientís Oracle database server high swap memory usage, we find the swap space usage is constantly very high on this Linux server, which causes Oracle database a big performance issue.
OS: RHEL 7.3
DB: Oracle 12.2.0.1
Findings:
1)top
Tasks: 352 total, 2 running, 350 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.4%us, 4.1%sy, 0.0%ni, 79.3%id, 2.2%wa, 0.3%hi, 0.8%si, 0.0%st
Mem: 32172820k total, 32015956k used, 156864k free, 14528k buffers
Swap: 16777208k total, 7435428k used, 9341780k free, 11129844k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
137049 oracle 15 0 16.2g 5.3g 5.3g S 20.6 17.4 10:14.17 oracle
72457 oracle 15 0 16.2g 4.7g 4.7g S 15.3 15.3 10:50.15 oracle
…
..
.
2) pmap
$ pmap -x 137049
137049: oracleRACTEST1 (LOCAL=NO)
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 11704 0 r-x– oracle
0000000006419000 444 140 4 rwx– oracle
0000000006488000 148 100 80 rwx– [ anon ]
000000001966e000 532 176 92 rwx– [ anon ]
0000000060000000 16779264 5444888 1485768 rwxs- [ shmid=0x670005 ]
00000032b6a00000 112 108 0 r-x– ld-2.5.so
00000032b6c1c000 4 0 0 r-x– ld-2.5.so
…
..
.
00007fff4c504000 160 136 132 rwx– [ stack ]
00007fff4c5d2000 12 4 0 r-x– [ anon ]
ffffffffff600000 8192 0 0 —– [ anon ]
total kB 16954204 5477952 1490448
3) swappiness
$ cat /proc/sys/vm/swappiness
10
4) /proc/meminfo
$ cat /proc/meminfo
MemTotal: 32172820 kB
MemFree: 158096 kB
Buffers: 12512 kB
Cached: 11124148 kB
SwapCached: 878556 kB
Active: 11885576 kB
Inactive: 376796 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32172820 kB
LowFree: 158096 kB
SwapTotal: 16777208 kB
SwapFree: 9336848 kB
Dirty: 136 kB
Writeback: 0 kB
AnonPages: 487212 kB
Mapped: 11089784 kB
Slab: 104984 kB
PageTables: 2779136 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 24475008 kB
Committed_AS: 20456368 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8192
HugePages_Free: 8192
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
Causes:
The huge pages are not used by Oracle database server, because the number of hugepages set-up is too small.
Solution:
1)Increase the number of hugepages.
$ cat /etc/sysctl.conf|grep -i hugepages
vm.nr_hugepages = 8200
2) Reboot the server.
3) Check the results.
$ cat /proc/meminfo
…
..
.
SwapTotal: 16777208 kB
SwapFree: 16777208 kB
Dirty: 116 kB
Writeback: 8 kB
AnonPages: 1076372 kB
Mapped: 108400 kB
Slab: 132980 kB
PageTables: 37540 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 24466816 kB
Committed_AS: 2249980 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8200
HugePages_Free: 26
HugePages_Rsvd: 19
Hugepagesize: 2048 kB
$ ipcs -ma
—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status
0x00000000 5275649 root 644 80 2
0x00000000 5308419 root 644 16384 2
0x00000000 5341188 root 644 280 2
0xbb94f3d4 5373957 oracle 640 17181966336 87
—— Semaphore Arrays ——–
key semid owner perms nsems
0x17a558e8 131073 oracle 640 250
…
..
.
—
— top
top – 15:15:40 up 20:13, 7 users, load average: 1.13, 0.64, 0.50
Tasks: 289 total, 3 running, 286 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.9%us, 1.4%sy, 0.0%ni, 63.0%id, 23.3%wa, 0.4%hi, 1.0%si, 0.0%st
Mem: 32172820k total, 19801684k used, 12371136k free, 825948k buffers
Swap: 16777208k total, 0k used, 16777208k free, 931716k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
97575 oracle 16 0 16.1g 31m 25m R 16.6 0.1 0:11.88 oracle
97973 oracle 17 0 16.1g 28m 21m D 8.6 0.1 0:01.61 oracle
…
..
..
$ pmap -x 97575
97575: oracleRACTEST1 (LOCAL=NO)
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 17880 0 r-x– oracle
0000000006419000 444 216 56 rwx– oracle
0000000006488000 148 116 116 rwx– [ anon ]
000000000a80a000 532 428 428 rwx– [ anon ]
0000000060000000 16779264 0 0 rwxs- 1 (deleted)
00000032b6a00000 112 108 0 r-x– ld-2.5.so
00000032b6c1c000 4 4 4 r-x– ld-2.5.so
…
..
.
ffffffffff600000 8192 0 0 —– [ anon ]
total kB 16948952 42024 17284
$ sar 1 10
Linux 2.6.18-408.el5 (racnode1) 08/02/2017
03:18:48 PM CPU %user %nice %system %iowait %steal %idle
03:18:49 PM all 5.81 0.00 1.52 1.26 0.00 91.41
03:18:50 PM all 4.77 0.00 1.76 2.01 0.00 91.46
03:18:51 PM all 3.54 0.00 1.26 1.01 0.00 94.19
03:18:52 PM all 2.27 0.00 0.51 1.52 0.00 95.71
03:18:53 PM all 0.25 0.00 0.50 0.50 0.00 98.74
03:18:54 PM all 5.54 0.00 1.01 1.01 0.00 92.44
03:18:55 PM all 2.78 0.00 0.51 0.00 0.00 96.72
03:18:56 PM all 1.01 0.00 0.25 0.25 0.00 98.49
03:18:57 PM all 1.01 0.00 0.25 0.51 0.00 98.23
03:18:58 PM all 2.76 0.00 1.26 0.50 0.00 95.48
Average: all 2.97 0.00 0.88 0.86 0.00 95.29
$ vmstat 1 5
procs ———–memory———- —swap– —–io—- –system– —–cpu——
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 12377776 827836 932156 0 0 1259 387 86 73 5 1 90 5 0
0 0 0 12377776 827856 932140 0 0 15 398 3028 2395 3 2 95 1 0
3 0 0 12373368 827856 932160 0 0 231 270 2065 2786 21 16 63 0 0
0 0 0 12369324 827864 932140 0 0 0 168 1124 676 17 1 82 1 0
0 0 0 12369324 827868 932160 0 0 40 112 1311 754 1 1 98 1 0
