Oracle FAQ Your Portal to the Oracle Knowledge Grid
HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US
 

Home -> Community -> Mailing Lists -> Oracle-L -> RE: Centralized Alert Log Monitoring

RE: Centralized Alert Log Monitoring

From: Jesse, Rich <Rich.Jesse_at_qtiworld.com>
Date: Mon, 07 Jan 2002 13:51:45 -0800
Message-ID: <F001.003E8ECE.20020107132521@fatcity.com>


Wow! Sounds nice, but it seems like a lot of time invested. I imagine for shops with many DBs (we've got <10 production), this could save time in the long-run.

We just made a simple Korn script that checks all .log and .trc files in the $ORACLE_ROOT tree for certain keywords, like "error" and "ORA-". If it finds one in a file, it renames the file with a date/time stamp and emails the offending lines to our Systems department (including me). I should try it under CygWin to see if it'll work there. Not sure about "mailx"... :) Also, if there was a deadlock, it awks out some data from the trace file so we can determine whether or not it needs further investigation without rummaging thru the somewhat-cryptic trace.

Benefits:

1)  Highly customizable.
2)  Portable (took <2 mins to change "grep -E" to "egrep" for Solaris).
3)  Small and easy for non-Oracle folk to change (<4KB).
4)  It works for us.
5)  I helped write it, so when #4 does occasionally fail, I can fix it.  ;)

Caveats:
1) It only runs every 5 minutes, due to the overhead incurred in looking for/thru all .log and .trc files. This may prove to be a major stumbling block for us in the future, depending on the nature of a couple of our DBs. 2) It won't help if e-mail is down, but this *should* be a rare occasion. And when it does happen, I can manually monitor the logs for a spell. 3) It *may* not catch 100% of errors, but I haven't run across a situation yet where it missed one.
4) It's really a "work-in-progress", but good enough for us for now.

If you get that rev 2 up and running, let us know! :)

Rich Jesse                              System/Database Administrator
Rich.Jesse_at_qtiworld.com                 Quad/Tech International, Sussex, WI
USA -----Original Message-----
Sent: Monday, January 07, 2002 2:35 PM
To: Multiple recipients of list ORACLE-L

> Any ideas?

Yeah. It involves Perl of course.

This sounds like a natural evolution for the stuff I already have.

The distributed version will be in the Perl book. If I had time I would create the centralized version for it, but I can't do it right now.

'Why even mention it?' you ask.

Well, it never hurts to cast about for ideas on how to do it.

The cleanest way to me would be to have a daemon/Service ( one per database, unix or Win32 ) on the db server side that monitors the alert log. All it would do is read the alert log, and ship off new lines that appear ( via TCP socket ) to a centralized analyzer.

The analyzer would be a non-forking ( Win32, remember? ) TCP server accepting the data from the unix daemons and win32 services, writing it out to a master file, data files per database, or to Oracle tables . Lots of ways it could be stored. ( re the non-forking: Perl threads aren't ready for prime time )

The analyzer would check for patterns in the text that indicate that someone should be paged. These could be global or at the database level, probably a combination with db level overriding global where it exists.

Emails, pages, what have you would be sent, recorded, the errors recorded, etc.

Both client and server would require unix and win32 specific versions, as writing a win32 service is sufficiently different from a unix version to warrant a different code. base.

Might make a nice rev 2 tool.

Any thoughts on this?

Jared

*****DISCLOSURE STATEMENT*****
This transmission may contain information which is confidential, proprietary and privileged. If you are not the intended recipient, note that any review, disclosure, copying, retransmission or other use is strictly prohibited. If you received this transmission in error, please notify the sender immediately and delete the material from your system. Received on Mon Jan 07 2002 - 15:51:45 CST

Original text of this message

HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US