Re: huge CPU load after UEK upgrade

Hi,

Run psnapper or just sudo cat /proc/PID/stack for some of the Oracle & kworker processes (that are in D state) and check if you have *wbt_wait* in the stack. UEK 4 (kernel 4.14) has a Linux kernel write throttling bug/problem and lots of kworkers showing up is one of the symptoms. I've written about drilling down into high Linux system load in general here:

https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/

Incidentally, this is one of the problem scenarios I cover at the upcoming "troubleshooting very complex Oracle performance problems" virtual "conference" (where all speakers are me - https://tanelpoder.com/events ;-)

--
Tanel Poder


On Thu, Aug 12, 2021 at 2:41 AM Laurentiu Oprea <laurentiu.oprea06_at_gmail.com>
wrote:



> Hello everyone,


>

> I know this is more of a linux question but maybe someone had a similar

> issue.

>

> I started to upgrade the kernel version from UEK3 to UEK4 on servers (rac

> 2/3 nodes) running OL6. I had no issue upgrading the kernel on virtual

> machines.

>

> When I upgraded the first rac node located on physical hardware, after

> starting the oracle instances the cpu load went crazy, the whole system was

> far slower. For example the cpu load on the other rac nodes with UEK3 was

> 4/3/3/ while the upgraded node load was 107/104/77 with only half the

> instances started, this workload being created even if the instances had no

> significant workload (they were just started)

>

> I observed that with the new kernel version when oracle instances start

> there is quite a big number of kworker processes. Perf top will show me as

> overhead

> 17.24% [kernel]  _raw_spin_lock

> 4.50% [kernel] acpi_os_write_port

>

> Does anyone have any idea on what could be the issue?

>

> THanks a lot.

>


--
http://www.freelists.org/webpage/oracle-l