Oracle FAQ Your Portal to the Oracle Knowledge Grid

Home -> Community -> Mailing Lists -> Oracle-L -> Re: Duhvelopers and DB-ehs?

Re: Duhvelopers and DB-ehs?

From: Marc Perkowitz <>
Date: Tue, 04 Sep 2001 07:36:55 -0700
Message-ID: <>

Ok, so right about the time Oracle 6 went production (we were a beta site), I was the lead DBA for a very large project that put retail store systems in around 1000 locations around the country (USA).  Each one of those locations had their own Oracle database on AT&T 3B2 Intel boxes.  I also supported development, testing, and QA too.  But the earliest versions of Oracle 6 had some database corruption problems -- not too often, say only once a year on the average.  However, multiplying that times 1000 yielded about 2 or 3 stores weekly!  I became very familar with workarounds for db corruptions -- those undocumented rollback init.ora parameters, how to find corrupt blocks and create dummy tables to hold them, etc.  We set up nightly exports to /dev/nul to find the corruption sooner than later.  Lots of scripts and plan A, B, C ... depending on where the corruption was.  Tape backups and mainframe downloads ... static data separated from dynamic data in different tablespaces and files. 
During the most intense time period, hotels were booked and meals catered and we didn't leave the building complex for weeks.
Aren't all these stories fun!!?  (Though I'm happy to say I was never on another project that intense again!)  
Marc Perkowitz.
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">

  To: <A
  href="">Multiple recipients of list ORACLE-L   

  Sent: Friday, August 31, 2001 8:26
  Subject: Re: Duhvelopers and
  face=Arial size=2>   

  OK.  I've resisted the temptation so far,   but (puffing out chest)...

  In one job in my distant past, at a turnkey   software/systems shop, four (initially two) of us used to administer   about half a dozen in house Oracle servers and remotely administer   about 2-3 dozen of our clients' database servers.  They were   scattered all over the US and Canada and we all did database and Unix   systems administration "part time".  About 80% of our time was spent in   custom software/systems design and development.  Everybody got Oracle DBA   training, but nobody was a full-time DBA or systems administrator.  (A   very novel approach that resulted in very few deeply imbedded application   performance issues!)

  From late 1997 until mid-1999 I administered up to about 45   Oracle databases (each was 10-200 GB & OLTP) by myself -   including design/modeling, code reviews and tuning, physical server   configuration (sizing systems, creating raw devices, etc.), and   production database administration.  I was the only Oracle DBA in the   entire company for my first 18 months there!  We actually had about   12 distinct Oracle-based application systems, most with distinct dev,   functional test, performance test, and production databases.  These were   spread over more than 40 Sun servers ranging from Ultra 2s to   E10Ks.  Three of the production systems were on Sun PDB cluster and   Oracle parallel server.  Critical?  At least four were the   core components of the trading systems at one of the world's   largest online brokerages - serious 24 x forever with (usually) million+   dollar costs for any downtime.

  Actually the number was about 24 databases   until Jan 1999.  At that time, they gave me some help - the guy who   was the (only) DBA for the Informix system (which was phased out the   following Fall) - to retrain as an Oracle DBA.  Two weeks   later I was asked to create 21 new databases on 17 new servers -   within nine days.  I made it with a little under an hour to   spare.  That brought the total up to about 45.  By August, the   ex-Informix DBA was doing Oracle full time and they had hired some more help   (all with no significant Oracle experience!) - and,  of course, added   another 20-30 databases!  When I left a couple of months ago (no surprise   is it?) we had 42 production Oracle databases on 35 production servers, with a   total of over 160 databases on over 120 servers, administered by 6   DBAs.  We also had over 100 other servers (Web servers, WebLogic servers,   Tuxedo servers, etc.) that had the Oracle client (including Pro*C)installed -   which we also maintained.

  Of course I was working an average of about   95 hours/week much of that time... And sleeping less than half   that.

  For none of these did I ever have any significant   3rd party tools (BMC Patrol, etc.).  I did develop a *LOT* of homegrown   scripts and automation though. [I still prefer, with a very few select   exceptions, homegrown tools over commercial tools.]  The other key was in   *ADAMANTLY INSISTING* on being able to set things up "right" at the   start.  Having enough space on (at least most) test systems to load   (or clone) production-sized databases helped also.

  This sounds kinda like "When I was a kid, we had to walk five miles to   school in neck-deep snow - uphill both ways!" doesn't it?    Unfortunately, its true.  I have references and the complete lack of an   outside life to prove it!  Since I left, I've been decompressing and   going through social rehabilitation  ["Hi, my name is Don.  I   haven't been on a bridge call in over two months."  (Smattering of   applause...)]
  <FONT face=Arial
  size=2> <FONT face=Arial
  -Don Granaman
  [certifiable OraSaurus and shameless

  PS:  Most of, and certainly the best
  of, the DBA's I've ever worked with had significant prior   experience as developers (not "duhvelopers").   <BLOCKQUOTE dir=ltr
  style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">

    Sent: Friday, August 31, 2001 12:18     PM
    Subject: RE: Duhvelopers and

    <FONT face=Arial color=#0000ff
    size=2>Yeah, well, in a previous life I was in charge of 27 databases on 9     servers ALL BY MYSELF. Didn't have anybody working with me, man. Zilch.     Nada.
    <FONT face=Arial color=#0000ff
    <FONT face=Arial color=#0000ff
    size=2>And, of course, I'm not going to talk about the nature of the systems     'cause then it wouldn't sound so impressive....     <FONT face=Arial color=#0000ff
    <FONT face=Arial color=#0000ff
    size=2>--Walt Weaver
    <FONT face=Arial color=#0000ff
    size=2>  Bozeman, Montana     

      <FONT face=Tahoma 
      size=2>-----Original Message-----From: Mohan, Ross 
      []Sent: Friday, August 31, 2001 
      10:38 AMTo: Multiple recipients of list 
      ORACLE-LSubject: RE: Duhvelopers and 
      <FONT face=Arial color=#0000ff 
      size=2>We do about the same number with 6 DBAs...
      <FONT face=Arial color=#0000ff 
      <FONT face=Arial color=#0000ff 
      size=2>(not that that means sh*t without talking about the nature of the 
      systems, of 

course) Received on Tue Sep 04 2001 - 09:36:55 CDT

Original text of this message