Re: Solaris CPU Consumption

On 01/29/2006 11:21:48 PM, Deepak Sharma wrote:
> I am having a tough time finding the right command to
> give me the CPU Usage on our Solaris box.
>
> If there are 100 processes currently on the system, I
> want something that shows 100 lines, each having a
> (non 0.00%) CPU Usage column for that process.
>
> I use 'top', 'vmstat 2 5' or 'sar -u 2 5', for
> example, to find the total CPU usage for the entire
> system.
>
> 1. Using 'top' shows total CPU, say 30% idle (or 70%
> Used), but when I add up all individual processes'
> CPU, it adds up, something like 56%, as an example.
> Several of the processes listed in top show 0.00% CPU
> and the reason could be that they have a 'sleep'
> state. But, adding up all non - 0.00% processed yield,
> as mentioned before, a much lower number than the
> total CPU used shown by Top.
>
> 2. I also tried using '/usr/ucb/ps -auxwww', and
> adding individual processes' CPU. same story, they all
> add up to a total that is much lower than total CPU
> reported by any of the above mentioned commands.
>
> Any help?

The problem is your assumption about the CPU time accounting. On Unix, and Solaris can certainly be regarded as such, there are 3 types of state that CPU can be in:
1) User mode, working on a user process. There is no problem with

   accounting for that time.
2) Kernel mode, when the CPU is processing a system call

   on behalf of a process which invoked the system call. 3) Interrupt stack, when the CPU is servicing an external

   interrupt, coming from some device, outside of a process    context. That means that not a single process is charged    with the CPU time servicing clock interrupt or NIC interrupts.

Believe it or not, a lot of things can be hidden and execute in an interrupt handler. To add to the confusion, interrupt routines are usually scheduled in two parts: upper and lower. Upper part is a high priority interrupt that comes from a device. This part only updates statistics, sets flags, queues lower portion of the interrupt as so called "soft interrupt" and exits, doing only a minimal work. Lower part of the interrupt does the actual processing. Lower part is indistinguishable from the kernel code, but is also not attributed to any process. How does it actually work: 133 PCI bus sends an interrupt to the CPU, say Xeon or P4, 133 million times per second. Each time, the interrupt is recorded. Every now and then, usually using modulo number technique, a soft interrupt is scheduled. The period between the soft interrupt is usually 10 msec and is called "quantum". What is the other name for the lower part of the clock interrupt? It's called "scheduler". The whole run queue management, device management and virtual memory management occurs outside the process context. That means that no single process can be charged with spent CPU, which can be significant. The overall number will rarely be sum of CPU usage of all processes. Of course, you must add to the equation the fact that different programs account differently for the spent CPU. Some will count time spent waiting on an I/O request as "WIO CPU time", some will report idle time. Rounding is also done differently. There are different methodologies and different metrics. My advice is to keep with one program, GlancePlus if possible, which it isn't on SUN boxes. Also useful on Slowaris are pio and topio which you can find here:

http://www.stormloader.com/yonghuang/freeware/pio.html

I can also recommend reading of Adrian Cockroft's book, mentioned on that page, as it will explain you the entrails of the Slowaris system.

-- 
Mladen Gogala
http://www.mgogala.com

--
http://www.freelists.org/webpage/oracle-l