Recently I discovered a critical issue when installing a new Oracle cluster for a customer on Oracle Linux systems. The hardware are shiny new blade systems and I started with the latest and greatest software components which are certified for the Oracle Enterprise Edition and Real Application Cluster. Of course the OS would be Oracle Linux 8.7 with the famous Unbreakable Enterprise Kernel which is at version 7 since this year. In April 2023 Oracle relased the release update 1 which of course brings some enhancements and fixes. Business as usual as you would think.
Going on with the project I installed the operating systems and of course I consulted the certification matrix in the MOS before doing anything. There you can see that with the April 2023 RU of the database - which is release 19.19 and in fact the release I wanted to install - that any kernel version newer than a specific version of UEKR7 was supported. Since I used the latest UEKR7 I thought that I would be in a safe harbour. Things went on and after installation of core components like the GI, configuration of diskgroups, enablement of log maintenance and the creation of a database I came to the point to apply best practices like enabling huge pages. Since I used the glorious oracle-database-preinstall-19c package I just had to set the vm.nr_hugepages system parameter because the memlock setting was already configured in a proper way.
UPDATE:Apparently the statement about the certification state is not entirely correct. It seems that only UEKR6 was certified and the or later clause was only referring to UEKR6. The UEKR7 support seems to be only for some GI components
The bad
Since the system had 768 GB of RAM and about 450 GB of that should be used for SGAs, there is no reason to not insist on having huge pages available. So I moved on to configure the database instance to require them with the following statement. That is something that you should do with any database instance that has more than 4GB of SGA.
After the change the instance would not start when there are not enough huge pages available. So I bounced the database and was wondering why it did not come up and for some reason the host was unreachable too. That is something I have done tens or hundreds of times in the past and I never experienced such an issue. As this is a cluster, at first I disabled the database from the surviving node using srvctl and started to investigate.
The ugly
Further I checked the whole hugepages and memlock configuration. Nope, everything was fine. Then I tried to configure the instance with and without huge pages. Without the latter the instance would come up without any issues, but enabling it reproduced the failure reliably and cluster-wide. Of course I had some kernel(-module) bug in mind and went ahead to dig around. One good thing with Oracle Linux 8 is, that kdump is enabled by default and when such things happen, you can rely on the crash information in /var/crash/<timestamp>. There I found this neat kernel panic report at the end of the vmcore-dmesg.txt file.
Of course I reached out to MOS and requested an investigation of the problem. But until a solution is provided I wanted to get ahead with the project. Since the UEKR7 release update 1 was released in April 2023 I thought about using an older version of the kernel, because there might be some regression that was not yet disovered and a fix was not included in the lates April RU for the database and GI. To list all available kernel versions I used the following dnf command (stripped to the latest available versions).
After Identifing the previous releases I went back one after another by installing and selecting for boot (ignoring the RH kernel, since it is way older than the UEK).
Fortunately there were only two releases of the UEK release update 1, so that I soon came to try the release 5.15.0-8.91.4.1.el8uek.x86_64 which would let me open the database using a version of the UEKR7 release. So there might be some regression in the latest update to the UEKR7 and you should be careful to update. Luckily this happened before the system became productive and I hope that not a lot of people face the same issue. Maybe this is also related to the hardware in use and not a problem everywhere. The general advise to use a test system implementing updates before, is something that roams around not without a reason.
Add Comment
Comments will be shown after approval.