Re: OT - Getting fired for database oops

I was a director in a previous life and had a Unix System Admin that, against specific directions not to use expect with root privileges, did so. He came into my office and explained to me that he screwed up using the command to change the root password
on all of our database servers and some application servers and the shadow file got corrupted. This included servers in 4 countries.

Now - in this scenario the systems still run fine but you cannot get root privileges on them and you will eventually be in a pickle. I called a team meeting and we came up with a plan to correct the problem. These were SUN Solaris servers and the internal boot disks were being mirrored.
So, the entire team, with cooperation at each data center from local operators that worked for the company went system by system and to make a long story short we fixed the shadow file and got each and every server back into our control. I sent this administrator to the local data center and he spent the night doing the grunt work while we controlled the process remotely. This took almost 14 hours to complete.

The next morning this person came into the office, profusely thanked everyone for their help and handed out a six pack of each person's favorite for their understanding and help. Then he came into my office and was ready to take the consequences of his actions.

I told him that I wasn't going to fire him for two main reasons. First he was a good admin, got along very well with the team, a very hard worker and just had a bad brain fart. Second was that he was adult enough to come into my office to admit his mistake so we could address the issue on our own terms and internal to the production team, also I was sure he would never make that mistake again.

He turned out to be a very integral part of the team. I do admit I had one of the best teams anyone could ask for.

Mike

Newman, Christopher wrote:
> A colleague and I were just discussing DBA's getting fired for screwing
> something up/failing to do something (Not for being generally poor
> employees like showing up late, etc). Neither of us had heard any
> stories of either scenario. I know we sometimes have slightly off topic
> discussions on the list, so I was wondering if anyone would share any
> stories.
>
> Chris Newman
> Database Specialist
> AITS, University of Illinois
> 217-333-5429
>
> --
> http://www.freelists.org/webpage/oracle-l
>
>
>
>
>

--
http://www.freelists.org/webpage/oracle-l