Re: 24x7 database

tammy.adler_at_ipaper.com wrote:

]I am interested in hearing from dba's in a 24x7 environment, large database.
]Currently we have a 12 hour maintenance window each weekend. They are going
]to shorten this to a 4 hour window 3 weekends of the month, with no outage on
]the month-end. By fall, it is requested to be true 24x7, with scheduled
]outages every 6 months.
]
]This is a SAP application...the database is 150GB and growing rapidly...will
]probably double by year's end. The box is the IBM Raven, with AIX 4.3.
]
]How much housekeeping should I do in advance? What else should we do in
]preparation. Can a system be truly 24x7 without any scheduled down-time?
]What problems can we expect?

No IBM experience here, but we have Oracle on Sun.

The mine operates 24X7X362, production stops for Easter Monday, Labour Day and Christmas Day. Sometimes we can use these days for computer/network maintenance, but in reality a lot of users want the maintenance application available during this time, so we can't shut down then either.

Host outages are required to replace disks and upgrade hardware, Memory, CPUs etc. Some patches (including disk subsystem, security, RAID Array firmware etc) require at least unmounting all filesystems (i.e. definitely stopping Oracle) and often require a re-boot.

We don't maintain a regular window for these. Instead we negotiate a time with all interested parties, and schedule a "one-off" each time. In our case the users are happier with this than having a regular maintenance window.

We also work very closely with the Network people. Network changes/outages happen probably every two months - either cable problems or Novell crashes. These provide great windows to add that extra disk, slot in that new system board etc.

We find that scheduled downtime is fine, provided there is enough notice and that everyone knows about it. Unscheduled downtime gets expensive fast. We concentrate on avoiding the unscheduled outages.

For instance: the maintenance system is used 24X7, every day of the year. We had problems with the firmware in the disks in theRAID array (disk drive firmware, not array firmware). The disks would wronly report errors to the array, and twice caused the database to stop (unscheduled). When I got an upgrade and patch, I needed 4 hrs !! to do a complete static backup, then upgrade the disks and apply the patches. The uses were all very co-operative to get this done in order to avoid unscheduled outages.

For preparation I'd establish good communication between your support side and the users. Make sure everyone is talking freely and discussing best solutions, not backstabbing.

-- 
 ,-,_|\   George Dau,                                             __
/    * \  Unix (Solaris, DEC Unix, Linux), Oracle, Internet.     (OO)
\_,--\_/  Home: gedau_at_pobox.com      Work: gedau_at_isa.mim.com.au ( \/ )
      v   WWW:  http://pobox.com/~gedau                          W--W