RE: 10046, Unaccounted for Time, Aix 5.3 -- How to Determine Process Pre-Emption
Date: Thu, 3 Dec 2009 01:20:44 -0600
Message-ID: <003501ca73e9$21b848a0$6528d9e0$_at_net>
Thanks for the information. I'd already pointed the people working this towards Cary's paper
(http://www.hotsos.com/dnloads/1/kevents/unaccounted-for.html) and other similar ones. I'm *assuming*, which is dangerous, this is processor preemption based on the "signature" of what I've seen. But the initial goal was to verify it is preemption for this specific process, and seeing system wide stats, vmstat's cs column, for example, doesn't tell us for sure that preemption is the source for this process.
FWIW, normally when I see high values / percentages for unaccounted, the process *usually* has glaring problem(s) with approach / design and can/should be changed. The code/approach gets changed, and even if the unaccounted for remains high *percentage* wise, it is still so low, typically, to be inconsequential to response time. End result is that I've not normally had to dig into this topic that deeply because they were other bigger things to address, and once addressed eliminated the significance of the unaccounted for time.
But, this is vendor code, code will not change (and yeah, some glaring issues, approach wise -- it's more fun to execute SQL statements millions of times as opposed to 3 insert statements that join tables ;-)). And I've been asked to help. I started off by asking them to verify if it is indeed preemption. And that's where things are. And if it is indeed preemption, and maybe we should assume it is since trace file overhead has been ruled out, and changing the process is not an option, then some of the things you (I'd already been looking at aix scheduler topics), and others, bring up may need to be pursued. Currently running mrnl on the 384MB trace file to see what we can gain from that. And I had been hoping the SA's might be able to give a little more insight process wise. Another subject area to learn more about, os tracing.
Larry G. Elkins
> -----Original Message-----
> From: Kerry Osborne [mailto:kerry.osborne_at_enkitec.com]
> Sent: Wednesday, December 02, 2009 8:47 PM
> To: elkinsl_at_flash.net
> Cc: 'Oracle-L'
> Subject: Re: 10046, Unaccounted for Time, Aix 5.3 -- How to Determine
> Process Pre-Emption
>
> I'm not too familiar with aix but Solaris has several scheduling
> algorithms (Real Time, Fair Share, Time Share, etc...) These have
> different ranges of priorities. Do a man on ps. It probably has
> options to show priority and scheduling class (i.e. the algorithm).
> Mixing different algorithms in a single VM can cause problems.
>
> Here's the command in solaris that shows that info:
>
> ps -ef -o user,pid,project,class,zone,pset,pri,nlwp,psr,time,args
>
> As already mentioned, vmstats/sar should also give you a good clue as
> to whether processes are struggling to get on the CPU.
>
> There is a great Solris Internals book that describes all the
> scheduling algorithms among other things. I presume there is a similar
> reference for AIX. Also, I'd recommend Millsap's book for a better
> understanding of how/why unaccounted for time occurs.
>
> Kerry Osborne
-- http://www.freelists.org/webpage/oracle-lReceived on Thu Dec 03 2009 - 01:20:44 CST