Re: 1 minutes: best downtime story
Date: Mon, 18 Mar 2013 12:11:13 +0100
Message-ID: <5146F651.8090209_at_nordea.com>
On 14.03.2013 22:07, Jeremy Schneider wrote:
> Hey all -
>
> I'm writing a paper about top causes of downtime. As one component
> of research, I'd like to get some input from you!
>
> One minute, two sentences. First sentence: describe what went down.
> Second sentence: describe why. (I have to categorize all of these.)
> Everyone should have at least one downtime story so I'm hoping for a
> lot of feedback!
>
> Answer about any technology - database, operating system, etc.
>
> Thanks!
>
> -Jeremy
>
>
During a big, big project developers need many test environments with 0
time restore on quite big db. Instead of backup etc. there were
snapshots done, then on a snapshot RO there was a pool of storage for
RW. Worked quite fine.
But the storage array had a feature not too well known. which we become
aware afterwards ;-) - if it lacked in space, it deleted the oldest
snapshot. There were actually only those snapshots for this project -
the rest environments had the space allocated explicitly.
Somebody run copying quite large pack to this environment based on
snapshot (or may be by mistake it was even copied the same RO snapshot
mounted somewehere else). Without the feature RW space would run short
ending the procedure with failure, but with it the RO snapshot was the
oldest one and went to trash - no RO, no copy.
As we asked the storage company about this strange behaviour, we have just got well known - "this is not a bug, ...'
-- Pole na kazi ---------------------------------------------------------------------- Remigiusz Sokolowski <remigiusz.sokolowski_at_nordea.com> pos : Senior DBA at DIiUSI addr : Nordea Bank Polska SA, Luzycka 6A st, 81-537 Gdynia, Poland phone : +48 58 667 17 43 mobile: +48 602 42 42 77 Nordea Bank Polska S.A. z siedziba w Gdyni, ul. Kielecka 2, 81-303 Gdynia, wpisana do Rejestru Przedsiebiorców Krajowego Rejestru Sadowego pod numerem: 0000021828, dla której dokumentacje przechowuje Sad Rejonowy Gdansk - Pólnoc w Gdansku, VIII Wydzial Gospodarczy Krajowego Rejestru Sadowego, o kapitale zakladowym i wplaconym w wysokosci: 277.493.500,00 zlotych, NIP: 586-000-78-20, REGON: 190024711-- http://www.freelists.org/webpage/oracle-lReceived on Mon Mar 18 2013 - 12:11:13 CET