Re[2]: Confessions of An Oracle Expert -- What a Fortnight

From: <dgoulet_at_vicr.com>
Date: Sun, 4 Jun 2000 17:53:11 -0400
Message-Id: <10518.107670@fatcity.com>

Joe,

We had a somewhat similar mess with EMC. They had installed some additional drives & were installing the BIN file mods. Seems that what was in the array & what they thought was there were NOT the same. Took almost half an hour before things started happening, but one by one the file systems started disappearing. Three database instances crashed, and three manufacturing lines went down with them. Thankfully the EMC folks were still here & could assist us in correcting the mess. Took the rest of the day, but we got there. We're still talking with EMC on who's paying for the lost production time.

____________________Reply Separator____________________

Subject: Re: Confessions of An Oracle Expert -- What a Fortnight Author: "Joseph S. Testa" <teci_at_oracle-dba.com> Date: 6/2/00 5:47 PM

Good story, mine is somewhat the same,

EMC disk array, sysadmin mapped, system01.dbf and swap to same partition(i've yet to figure out how they did it but its was not of my concern), so for a long time all was well, then data block corruption errors when we were hitting the machine hard.

SA figured it out, apologized and rebuilt both, restored system01.dbf from backup, did restore(we were in archivelog mode) and all was well.

Oh and this was supposed to be a production system :)

I feel for you.

Joe

"MacGregor, Ian A." wrote:
>
> Expert: A person who has made all the conceivable mistakes and is working
> on the inconceivable.
>
> This week I migrated one of our databases connected with one of the physics
> experiments to a new machine and 8.1.6. The day runs all proceeded without
> incident, but after the midnight shift change problems started occurring
> with client connections. I found that I could not
> get the database to accept a certain username and password from a client,
> but it would from the database machine itself through an IPC connection. I
> checked, double-checked, and triple-checked the setting of the TWO_TASK
> variable which was correct. In then tried
> username/password_at_alias form the client and I was in. I had forgotten that
> earlier that
> day one I was working on wrapping sqlpus so it runs in its own shell, and
> the shell was
> setting two_task to something different. This was not the problem the users
> were having,
> theirs were related to switching to shared servers.
>
> A few people came out of the woodwork saying they needed to see and in some
> cases update the data on the new box. Here permissions on an applications
> data is the responsibility of the application's owner. Obviously some
> database links were needed. The links needed to be shared, but couldn't be
> public links. I've not read about this anywhere, but you
> can share a database link via a view. For example user a: creates the link
> called
> from heretothere, creates a view some_table as select * from
> some_table_at_heretothere,
> and grants user b permissions on some_table. User b can then access
> "some_table_at_heretothere
> as a.some_table. I learned this from a self-taught developer a few years
> ago.
>
> A vendor came in this week to install a product which dumps web logs into
> Oracle. Two more 8.1.6 instances on a brand new box. Some flaky things
> started to happen, not with the database, but with the Oracle software. A
> reboot showed super block problems, the file systems were rebuilt and things
> looked okay at first. But soon got flaky. I decided to re-install the
> Oracle software. After using the installer to remove the software, I
> decided to wipeout the directories as well. This was done via the rm -rf
> command which should get rid of any "." files. Yet some would report back
> that they were not empty. An ls -la would report back nothing. The sysadmin
> reformatted the partition. Oracle was installed, the vendor got about 95%
> through his installation all looked well. Last night I rebooted the
> machine again to kill off an automount daemon. The system would not come
> back up. Again a bad super block. Another sysadmin was able to recover
> from this. I got the system
> back sand tried to bring up Oracle using svrmgrl. Svrmgrl could not find
> its message translations. So whenever I got a message on another box I
> would issue oerr mgr "<number>
> and see what it was telling me. This was a bit cumbersome so I tried
> sqlplus /nolog intending to connect after a sysdba. The screen behaved as
> if the linefeed key was permanently pressed. This was not the best
> situation either. Compounding the growing angst was Jared message that he
> couldn't get sqlplus /nolog to work. Something that had worked for me a few
> hours before.
>
> I thought this was a bad disk. There were previous parity problem with the
> disk, but these went away when the SCSI cable with a bent pin was replaced.
> It was not a hardware problem at all. The sysadmin who originally built the
> partitions for the Oracle software and the Vendor's software overlapped
> them. Thus both could access some of the same disk blocks. Disk block
> multiplexing is not yet ready for production systems :). It was the
> vendor's file system which had the super block problem, and fixing it
> trashed part of the Oracle software install.
>
> Ian MacGregor
> Stanfored Linear Accelerator Center
> ian_at_slac.stanford.edu

-- 
Joe Testa  http://www.oracle-dba.com
Oracle 8i documentation online http://www.oracle-dba.com/oracle-docs
-- 
Author: Joseph S. Testa
  INET: teci_at_oracle-dba.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may

Received on Sun Jun 04 2000 - 16:53:11 CDT