Re: RE: How do you rotate OEM 12c paging different DBAs on the team....

From: Stojan Veselinovski <stojan.veselinovski_at_gmail.com>
Date: Wed, 16 Apr 2014 17:33:26 +1000
Message-ID: <CALn1tDs_WCkJDf-mHQuL01-tQuju367tOo2NpMaFrc-KcCR5oA_at_mail.gmail.com>



We have an oncall phone and each member takes it for a week. Grid Control is setup to send critical alerts to that particular phone. Critical and Warnings got to all members via email and to HP SM7 via snmp trap and which automatically creates an incident for us to work on.

A command centre/operations bridge 24*7 triages some of our calls like agent unreachable.

Be wary though that it takes a while to cut through the false positives, bugs and other issues associated with EM12C so you may get lots of early morning wake up calls.

Regards,

On Wed, Apr 16, 2014 at 12:06 PM, kellyn.potvin_at_ymail.com < kellyn.potvin_at_ymail.com> wrote:

> I don't see a response to this last email, so I'll chime in.
> Considering the setup you're currenrly using with rule sets, etc. I'd
> enhance this with a notification schedule, (can be located in the Setup
> menu).
>
> A. Decide who will be oncall and for how long, via a calendar and plan out
> for vacations/time off. As far out as possible. If six dbas, then you need
> to build out a six week notification schedule.
> B. Create a super user called dba_page and add the sms text address for
> all dbas in the group used for paging.
> C. Logged in as this user, go to notification schedule. Choose to edit it,
> choosing a rotation that fits your schedule, (weekly, every six weeks, etc)
> when I had 3 dbas, we did a three wk rotation in the notification schedule.
> D. You can bulk fill by day and hour, so weekdays, populate with all the
> email addresses if you want or leave them blank if you want this "user" to
> NOT receive notifications during business hours, (let email for standard
> user handle it...)
> E. For the night time hours, choose the first person to go oncall, (I
> choose a Monday date to start by to ease rotation). Fill in bulk for all
> hours on weekend.
> Choose next week and do the same for the next DBA...
> F. So on and so forth till you've built out a schedule that makes sense
> and will rotate accordingly.
> G. Save the schedule.
> H. View the schedule and make sure the notifications and time look
> correct, along with the first time it starts over is populated correctly
> with that first dba.
>
> Done. I know this sounds like a pain, but once you get the hang of it, you
> should only do it once a quarter to bi-annually outside of a schedule
> change.
>
> Regards,
> Kellyn Pot'Vin
> Consulting Member of EM Technical Team
> Strategic Customer Program
> Oracle
>
> Sent from Yahoo Mail on Android<https://overview.mail.yahoo.com/mobile/?.src=Android>
>
> ------------------------------
> * From: * Cunningham, Mike <mcunningham_at_thedoctors.com>;
> * To: * Chris Grabowy <cgrabowy_at_gmail.com>; 'ORACLE-L' <
> oracle-l_at_freelists.org>;
> * Subject: * RE: How do you rotate OEM 12c paging different DBAs on the
> team....
> * Sent: * Mon, Apr 14, 2014 8:44:22 PM
>
> Yes, everyone get’s CRITICAL alerts. We are all considered prod
> DBA’s. If we get a text message it is usually something that needs
> attentions so one of us will get up and, at least, check to make sure
> someone is taking care of the issue. Just 2 weeks ago one of the DBA’s on
> vacation texted me to double check that I was looking into a CRITICAL alert
> (blocking lock). I told him to concentrate on vacation J, but that’s how
> the life of a DBA goes sometimes.
>
>
>
> In bigger shops I understand there is a more formal procedure, but we have
> just 3 DBA’s and our communication is really good. So far, we have never
> had an issue where something was not addressed.
>
>
>
> *Michael Cunningham*
> *Senior Database Administrator*
> *The Doctors' Company*
> 707.226.0221 - desk
> 707.337.0184 - cell
>
>
>
> *From:* Chris Grabowy [mailto:cgrabowy_at_gmail.com]
> *Sent:* Monday, April 14, 2014 1:30 PM
> *To:* Cunningham, Mike; 'ORACLE-L'
> *Subject:* RE: How do you rotate OEM 12c paging different DBAs on the
> team....
>
>
>
> Thanks Michael.
>
>
>
> So this is more or less what we are doing too.
>
>
>
> How do you handle after hours support? Does everyone get a CRITICAL text
> at midnight and then….?
>
>
>
> Thanks,
>
> Chris
>
>
>
> *From:* Cunningham, Mike [mailto:mcunningham_at_thedoctors.com]
> *Sent:* Monday, April 14, 2014 4:12 PM
> *To:* cgrabowy_at_gmail.com; 'ORACLE-L'
> *Subject:* RE: How do you rotate OEM 12c paging different DBAs on the
> team....
>
>
>
> We use two layers of alerts setup like this.
>
> 1) WARNINGS get emails sent to all DBA’s. The on-call DBA should be
> checking emails. There are times when the on-call DBA may need to step
> away and asks another to watch his back and that is why we all get the
> emails. The subject line is formatted to make easy sorting. This might be
> something like filesystem >= 85%.
>
> 2) CRITICAL alerts send text messages to all DBA’s. This is
> valuable because my phone is sent to silent for emails, but makes an
> obnoxious sound for text messages and I will pay attention to them.
> Regardless of on-call or not our team will communicate and figure out who
> is attending to the alert so we know someone is on it. This is something
> like blocking locks, filesystem >= 95%, etc. In our environment if we get
> a text there will also be an email with more detailed information.
>
>
>
> *Michael Cunningham*
> *Senior Database Administrator*
> *The Doctors' Company*
> 707.226.0221 - desk
> 707.337.0184 - cell
>
>
>
> *From:* oracle-l-bounce_at_freelists.org [
> mailto:oracle-l-bounce_at_freelists.org] *On Behalf Of *Chris Grabowy
> *Sent:* Monday, April 14, 2014 12:44 PM
> *To:* 'ORACLE-L'
> *Subject:* How do you rotate OEM 12c paging different DBAs on the team....
>
>
>
> We are in the middle of setting up OEM 12c. I setup a separate Incident
> Rule set for the production group and critical alerts.
>
>
>
> I am thinking about setting up a different OEM user whose email is the
> texting addresses of the DBA team.
>
>
>
> We do not carry a pager but we have corporate Blackberrys. So I am trying
> to determine how we would rotate who receives the page dependent on who is
> on call.
>
>
>
> So I am wondering what other sites do? Any pointers? Links? Ideas?
>
>
>
> TIA.
>
>
>
> Chris
>
>
>
>
>
>
>
> *Confidentiality Notice*: This message and any attachments hereto may
> contain confidential and privileged communications or information and/or
> attorney client communications or work-product protected by law. The
> information contained herein is transmitted for the sole use of the
> intended recipient(s). If you are not the intended recipient or designated
> agent of the recipient of such information, you are hereby notified that
> any use, dissemination, copying or retention of this e-mail or the
> information contained herein is strictly prohibited and may subject you to
> penalties under federal and/or state law. If you received this e-mail in
> error, please notify the sender immediately and permanently delete this
> e-mail.
>
>
>
> *Confidentiality Notice*: This message and any attachments hereto may
> contain confidential and privileged communications or information and/or
> attorney client communications or work-product protected by law. The
> information contained herein is transmitted for the sole use of the
> intended recipient(s). If you are not the intended recipient or designated
> agent of the recipient of such information, you are hereby notified that
> any use, dissemination, copying or retention of this e-mail or the
> information contained herein is strictly prohibited and may subject you to
> penalties under federal and/or state law. If you received this e-mail in
> error, please notify the sender immediately and permanently delete this
> e-mail.
>

-- 
Stojan
www.stojanveselinovski.com/blog
www.stojanveselinovski.com

--
http://www.freelists.org/webpage/oracle-l
Received on Wed Apr 16 2014 - 09:33:26 CEST

Original text of this message