RE: Extent allocation time -- process check
Date: Thu, 11 Feb 2010 09:10:53 -0600 (CST)
> Why all the work in trying to figure out the extent allocation. Can't you
> just go into awr and look at what was running during that time. It's going
> to be an insert or a create table.
An insert is most likely, yes (no object creation happens outside of IT during scheduled downtime as verified by the audit trail). However, there are hundreds or thousands of inserts happening per hour. Only a handful caused their respective object -- in this case, a table -- to allocate new extents. I need to verify what extended and when. I believe that the trace did so, but I have no verification of the process I used.
> You say you're suspecting LGC as being
> wrong, did a message get posted to the alert log, or was it just that a
> threshold got crossed? Did users get the "cannot allocate" etc error?
As I mentioned in the OP, Grid Control reported that there was ~9GB free right up to the point where we got the "ORA-1653: unable to extend table" errors. Since that metric runs every 30 minutes (and I failed to mention that it uploads to the OMS every collection), I believe it was wrong and that we really didn't have 9GB free in that TS.
Update: yesterday we had a snafu where there was a tree of blocking locks for over an hour during lunch. I was never notified by the POS Grid Control. After fruitless troubleshooting from MOS articles, I ended up bouncing the agent out of sheer frustration. 20 minutes later I got the notification. (It should be much more frequent than that, but that's a new GC PITA I need to tackle) I did however get notifications this weekend when TEMP ran out during stats collection.
This is at least our third major effing Grid Control failure and the second that led to Production downtime. The product is neither stable nor reliable. It makes Oracle look bad and makes me look bad or at least casts doubt on my assertions or arguments, and I will not stand for that. I've started the (probably long) search for a replacement.
I'm picturing a baseball bat gangsta Grid thrashing ala "Office Space". Need to find a way to do that to software...
RichReceived on Thu Feb 11 2010 - 09:10:53 CST