Re: How I save Cingular Wireless USD 30M

suppose you're going to now ask how I know I solved the problem. I know I solved the problem because after getting rid of the orphaned index segments, committing, aborting and starting the database back up, I let SMON spin but put the application back online. The database never deadlocked again (and I'm told has not since) and queries and batch jobs ran in a fraction of the time they had. My boss, the SA who had been maintaining the database with OEM when he didn't have a DBA there said that he remembered seeing that sort of performance years ago, back when the database was 7.3.4.

As proof of concept I asked the boss (the SAs ran RMAN there) to rest to other boxes the database from a year ago and 6 months before. Hcheck showed that the amount of corruption growth was linear, which is what you'd expect in one where you keep a window of switch data and are daily, weekly, semi-weekly and monthly dropping and creating new IOT partitions for the switch data and summations of it.

It took so long to get sign off from Oracle's Internals group that I was able to record the growth of many types of data dictionary corruption in each of the other databases in addition to timing how long queries and batch jobs took. Nice and linear.

Tom- Hide quoted text -

On 8/25/07, Tom Pall <oracle.list_at_gmail.com> wrote:
>
> I did the traces, ran Staspack till I was blue in the face, set the events
> to trap deadlocks. I did all of the things a DBA would do but decided that
> there was something deeper than just two applications colliding, because as
> I worked the problem over a two week period, I noticed the database slowing
> down. Not waits slowing down, not I/O slowing down, just throughput slowing
> down. Slowing down in ways neither I nor Oracle Support could explain
> before my dream, research in Metalink and discovery of hcheck.sql in
> Metalink.
>
> Is this technical enough?
>
> Tom
>
> On 8/25/07, Niall Litchfield <niall.litchfield_at_gmail.com > wrote:
> >
> > I must say I find this a somewhat bizarre contribution to the list.
> > Normally in a technical posting to this list it would be prudent to mention
> > what the problem was, how you diagnosed it, and how you fixed it. In the
> > case of a deadlock which can be traced, it would be normal to illustrate
> > this process - or why it failed you. Dreams are well and good, the benzene
> > ring structure springs to mind, but normally there's a follow up and a
> > methodology to investigate the hypothesis. The details of this seem to be
> > missing here - you say that you found a script on metalink for example, but
> > don't mention the note number. It found some orphaned indexes (The word
> > segment is redundant here) - I'm not sure what one of those would be - an
> > intact index left when a table was dropped? I've never seen one of those. It
> > would be a curious thing indeed, certainly worthy of describing in more
> > detail. You also describe being able to give a database a life expectancy -
> > I'd love more or indeed any detail on that. In short from a technical point
> > of view you don't actually tell us anything.
> >
> > The alternative I suppose is that this is a disguised job search. The
> > trouble with those of course is they tend not to work that well. I did take
> > a look at your resume though. Stuff that I would look for would be a concise
> > summary of your skills and abilities, your job history shows a number of
> > diverse and in demand skills - but it doesn't really tell me what I'm
> > getting. With the exception of the 30m saving - and believe me as a hiring
> > manager I'd want more confirmation of that than a post to this list merits -
> > your resume doesn't tell me how you will contribute to the business except
> > as a techie. Incidentally since both the resume and this conversation will
> > be indexed by google It would definitely be good to have the Oracle skill
> > sets listed out in an easy to read format on the resume - my gut reaction is
> > that the version 2 reference below is hyperbole given that that version was
> > 12 years old when you started your career (and almost no-one bought it) but
> > if your experience really does cover all of Oracle's history selling that in
> > the resume would be helpful.
> >
> > So in summary - if this was intended as a technical post then perhaps a
> > rethink about what you can and cannot say might produce a more helpful
> > contribution, that is one production dbas can use. if intended as an advert
> > - try not to make a habit of it and perhaps a resume rethink might help.
> >
> > best regards
> >
> > --
> > Niall Litchfield
> > Oracle DBA
> > http://www.orawin.info
>
>

On 8/25/07, JApplewhite_at_austinisd.org <JApplewhite_at_austinisd.org> wrote:
>
>
> Tim,
>
> I'm thinking and thinking - not to mention ruminating - about why the heck
> you'd ruin a great tequila by either diluting it with icewater or tainting
> it with fruit juice. It should be sipped at room temperature, absolutely
> straight, right after deeply inhaling (through the nose) the heavenly and
> stimulating vapors given off.
>
> Cheaper tequilas often benefit from a few jalapeno slices, but we should
> leave the sweet, salty, and dilution modifications to the young or otherwise
> inexperienced masses.
>
> My 2 centavos worth... ;-)
>
> Jack C. Applewhite - Database Administrator
> Austin I.S.D.
> 414.9715 (phone) / 935.5929 (pager)
>
> Aviso del secreto: Este mensaje del email, incluyendo todos los
> accesorios, está para el uso único de los recipientes previstos y puede
> contener la información confidencial del estudiante. El uso o el acceso
> desautorizado se prohíbe bajo las derechas de la educación de la familia y
> acto federales de la aislamiento (20 U.S.C. §1232g; Parte 99 de 34 CFR).
> Si no eres el recipiente previsto, no puedes utilizar, no divulgas, no
> copias ni diseminas esta información. Entrar en contacto con por favor el
> remitente por el email de la contestación o por 414-9715 que entra en
> contacto con inmediatamente y destruir todas las copias del mensaje
> original, incluyendo los accesorios.
>
>
>
> *Tim Gorman <tim_at_evdbt.com>*
> Sent by: oracle-l-bounce_at_freelists.org
>
> 08/25/2007 01:37 PM Please respond to
> tim_at_evdbt.com
>
> To
> oracle-l <oracle-l_at_freelists.org> cc
>
> Subject
> Re: How I save Cingular Wireless USD 30M
>
>
>
>
> In the midst of this engaging account of "looking and looking", "thinking
> and thinking", "ruminating", and subconscious revelations (rather than
> assessing traces, dumps, V$ views, truss/strace, and the like), I was hoping
> to discuss the tradition whereby someone with an unexpected monetary
> windfall celebrates by buying drinks all around...
>
> ...personally, I'm partial to Patron "silver", either on the rocks or
> chilled straight up, with a squeeze of lime...
>
>

--
http://www.freelists.org/webpage/oracle-l