Re: Protecting production from "us"
Date: Thu, 3 Dec 2015 14:36:16 -0700
Message-ID: <CAJzM94DWyGHcsjxS7+tMMaq6GvZSvFn_JfdyG=43jN5Jop82DQ_at_mail.gmail.com>
Something I came across years ago when we were establishing color coded putty sessions -- color blindness. Had a DBA who could not distinguish between prod/lower environments with the chosen color scheme. Worked it out eventually, but it wasn't something the team thought of initially.
Sandy
On Thu, Dec 3, 2015 at 1:49 PM, Alfredo Abate <alfredo.abate_at_gmail.com> wrote:
> I'm disappointed at management's response of *the backlash is now any
> further mistakes on production will result in immediate termination. *I
> don't see how any person (in any field) could work knowing that if they
> make another mistake like this that they are terminated. Especially given
> the track record that it sounds you and the rest of your team has had
> (years between this happening). If someone was making these types of
> mistakes frequently then that is another story all together.
>
> I suppose if the system at hand cost a company tons of money for each
> outage (say a trading system or high volume eCommerce) then things might be
> a little different (maybe this is the case here).
>
> At the end of the day these machines, systems, etc are all operated by the
> all mighty error prone humans.
>
> Alfredo
>
>
>
> On Thu, Dec 3, 2015 at 2:27 PM, Alfredo Abate <alfredo.abate_at_gmail.com>
> wrote:
>
>> I like Jeremy's server side control better for the terminal background
>> colors. I'll have to look into that one.
>>
>> Thanks for that tip.
>>
>> Alfredo
>>
>>
>>
>> On Thu, Dec 3, 2015 at 1:05 PM, Jeremy Schneider <
>> jeremy.schneider_at_ardentperf.com> wrote:
>>
>>> On Thu, Dec 3, 2015 at 11:45 AM, Herring, David <HerringD_at_dnb.com>
>>> wrote:
>>> > · Should we look into some kind of additional controls where
>>> > commands like "srvctl stop…" cannot be run under our own accounts using
>>> > "sudo -u oracle" but instead need a different account on production?
>>> For
>>> > example, normally our unfortunate DBA would use his "scapebob" Linux
>>> account
>>> > but perhaps to perform a production shutdown he'd need to connect as
>>> > "scapebob-rw", a new, special account just for dangerous production
>>> > activities.
>>>
>>> I think that I'd be hesitant to introduce too much variation between
>>> production and test environments when it comes to processes. It's a
>>> major advantage if you can test your processes in the test tier, then
>>> run those same processes verbatim (key-for-key) in production
>>> afterwards.
>>>
>>> > · The problem in our situation was over confusion with multiple
>>> > windows. Do people set a Linux TMOUT to something short like 10 or 15
>>> > minutes, to hopefully avoid accidentally leaving production putty
>>> sessions
>>> > open?
>>>
>>> I feel like a short timeout is likely to cause more frustration in the
>>> trenches than what it's worth, for anyone who spends any significant
>>> amount of time troubleshooting production systems. Often you have
>>> multiple windows open and switch between them... an aggressive timeout
>>> really makes that much more difficult.
>>>
>>> > · Beyond changing the linux prompt and text colors (we set
>>> $PS1 with
>>> > escape sequences and various key, env-specific values) do you do
>>> anything
>>> > else for protection of production?
>>>
>>> Personally, I think background color is your best bet. Only difference
>>> from Alfredo's suggestion would be that I'd prefer having it be
>>> controlled server-side rather than relying on each engineer to setup
>>> all their terminal connections correctly. Not to mention that you
>>> could get the *wrong* bg color if it's client-side and somehow
>>> somebody ssh's between tiers.
>>>
>>> --
>>> http://about.me/jeremy_schneider
>>> --
>>> http://www.freelists.org/webpage/oracle-l
>>>
>>>
>>>
>>
>
-- Sandy B. -- http://www.freelists.org/webpage/oracle-lReceived on Thu Dec 03 2015 - 22:36:16 CET