Re: Protecting production from "us"
Date: Thu, 3 Dec 2015 15:36:42 -0700
Message-ID: <5660C3FA.70000_at_evdbt.com>
Each time I've made a mistake, I have absolutely *longed* for the freedom to walk away and not be responsible for fixing the mess I created, instead of feeling morally obligated to spend the next 30-60 hours working my tail off to make things right.
Sounds like my kinda place, Alfredo! Where do I sign up? :)
Shouldn't management lead by example? Shouldn't they also be subject to termination for any mistake they make?
On 12/3/15 13:49, Alfredo Abate wrote:
> I'm disappointed at management's response of /the backlash is now any
> further mistakes on production will result in immediate termination.
> /I don't see how any person (in any field) could work knowing that if
> they make another mistake like this that they are terminated.
> Especially given the track record that it sounds you and the rest of
> your team has had (years between this happening). If someone was
> making these types of mistakes frequently then that is another story
> all together.
>
> I suppose if the system at hand cost a company tons of money for each
> outage (say a trading system or high volume eCommerce) then things
> might be a little different (maybe this is the case here).
>
> At the end of the day these machines, systems, etc are all operated by
> the all mighty error prone humans.
>
> Alfredo
>
>
>
> On Thu, Dec 3, 2015 at 2:27 PM, Alfredo Abate <alfredo.abate_at_gmail.com
> <mailto:alfredo.abate_at_gmail.com>> wrote:
>
> I like Jeremy's server side control better for the terminal
> background colors. I'll have to look into that one.
>
> Thanks for that tip.
>
> Alfredo
>
>
>
> On Thu, Dec 3, 2015 at 1:05 PM, Jeremy Schneider
> <jeremy.schneider_at_ardentperf.com
> <mailto:jeremy.schneider_at_ardentperf.com>> wrote:
>
> On Thu, Dec 3, 2015 at 11:45 AM, Herring, David
> <HerringD_at_dnb.com <mailto:HerringD_at_dnb.com>> wrote:
> > · Should we look into some kind of additional
> controls where
> > commands like "srvctl stop…" cannot be run under our own accounts using
> > "sudo -u oracle" but instead need a different account on
> production? For
> > example, normally our unfortunate DBA would use his
> "scapebob" Linux account
> > but perhaps to perform a production shutdown he'd need to
> connect as
> > "scapebob-rw", a new, special account just for dangerous
> production
> > activities.
>
> I think that I'd be hesitant to introduce too much variation
> between
> production and test environments when it comes to processes.
> It's a
> major advantage if you can test your processes in the test
> tier, then
> run those same processes verbatim (key-for-key) in production
> afterwards.
>
> > · The problem in our situation was over confusion
> with multiple
> > windows. Do people set a Linux TMOUT to something short like 10 or 15
> > minutes, to hopefully avoid accidentally leaving production
> putty sessions
> > open?
>
> I feel like a short timeout is likely to cause more
> frustration in the
> trenches than what it's worth, for anyone who spends any
> significant
> amount of time troubleshooting production systems. Often you have
> multiple windows open and switch between them... an aggressive
> timeout
> really makes that much more difficult.
>
> > · Beyond changing the linux prompt and text colors
> (we set $PS1 with
> > escape sequences and various key, env-specific values) do you do anything
> > else for protection of production?
>
> Personally, I think background color is your best bet. Only
> difference
> from Alfredo's suggestion would be that I'd prefer having it be
> controlled server-side rather than relying on each engineer to
> setup
> all their terminal connections correctly. Not to mention that you
> could get the *wrong* bg color if it's client-side and somehow
> somebody ssh's between tiers.
>
> --
> http://about.me/jeremy_schneider
> --
> http://www.freelists.org/webpage/oracle-l
>
>
>
>
-- http://www.freelists.org/webpage/oracle-lReceived on Thu Dec 03 2015 - 23:36:42 CET