RE: Protecting production from "us"

From: Herring, David <HerringD_at_DNB.com>
Date: Thu, 3 Dec 2015 20:12:01 +0000
Message-ID: <CY1PR0201MB18688BF134195B83F8CFD8BAD40D0_at_CY1PR0201MB1868.namprd02.prod.outlook.com>



Matt, how do you control placement of putty windows programmatically? I'm assuming that when you open a putty window for host x and "x" is production then that window is placed in a certain region of your screen.

Per Jeremy and Alfredo, I was really hoping for any putty changes to be automated so there's no easy way (or accidental way) to override them. Meaning, everything needs to be done at the host or server-level. Otherwise it'd be a trust factor that DBAs are properly setting foreground/background settings in their saved putty sessions, which at this point I don't think management would approve. This is the first time we've had an accident like this in years and the backlash is now any further mistakes on production will result in immediate termination.

What I liked about foreground color changes through $PROMPT_COMMAND and $PS1 is that "ls --color..." still works independently. I find it helpful to have file color coding but if the background gets changed then that color coding can possibly make it impossible to see certain types, so I'd have to turn that off. Maybe I'll have to figure out how to add sound to a production window so anytime it's one top I get some sort of Lost in Space warning sounded every 15 seconds. :)

Regards,

Dave

From: Matt Adams [mailto:MAdams_at_TroverSolutions.com] Sent: Thursday, December 03, 2015 12:35 PM To: Herring, David; oracle-l_at_freelists.org Subject: RE: Protecting production from "us"

It may seem simplistic, but my solution is....three screens on my destop. Left, laptop (middle) and right.

Production putty session are allowed only on the right, and are the ONLY things ever on the right screen.

From: oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org> [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Herring, David Sent: Thursday, December 03, 2015 11:45 AM To: oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: Protecting production from "us"

Folks,

The whole subject of locking down production, limiting access, etc. comes up periodically in our list so I apologize if this seems to be a repeat but in short I'm looking for anyone who's willing to share, on this list or privately, how they "protect" production from those who support it.

Here's the situation: as with many others, we're (DBA team) asked to support hundreds of environments. In one situation a DBA (let's call him Scapebob) had multiple putty sessions open for hosts supporting stage and production for the same application. In the heat of the moment he typed a "srvctl stop instance..." command in wrong window - production instead of stage. Both stage and production are 4-node RACs and initially no one noticed, not even the client. Scapebob immediately restarted the production instance and all was fine for about an hr but then some locking issues came up that caused an outage at which point upper-management heard of the accidental instance shutdown and our whole team came under fire.

The question/issue/subject I'm researching is how to best avoid this kind of thing happening again.

  • We already have LDAP/RH directory involved on a number of environments but that doesn't differentiate production vs. lower env. All require individual accounts and use "sudo -u oracle" to execute more dangerous commands.
  • Should we look into some kind of additional controls where commands like "srvctl stop..." cannot be run under our own accounts using "sudo -u oracle" but instead need a different account on production? For example, normally our unfortunate DBA would use his "scapebob" Linux account but perhaps to perform a production shutdown he'd need to connect as "scapebob-rw", a new, special account just for dangerous production activities.
  • The problem in our situation was over confusion with multiple windows. Do people set a Linux TMOUT to something short like 10 or 15 minutes, to hopefully avoid accidentally leaving production putty sessions open?
  • Beyond changing the linux prompt and text colors (we set $PS1 with escape sequences and various key, env-specific values) do you do anything else for protection of production?

Thanks in advance for anything shared.

Regards,

Dave

  • This communication may contain privileged and/or confidential information. If you are not the intended recipient, you are hereby notified that disclosing, copying, or distributing of the contents is strictly prohibited. If you have received this message in error, please contact the sender immediately and destroy any copies of this document. ****
--
http://www.freelists.org/webpage/oracle-l
Received on Thu Dec 03 2015 - 21:12:01 CET

Original text of this message