Skip navigation.

Arun Bavera

Syndicate content
Few of my technology tips. http://www.linkedin.com/in/arunbavera
Updated: 13 hours 29 min ago

EM12c Filtering ORA-600 till the issue is resolved

Fri, 2015-01-23 15:22



Example alert recieved:
Host=xyz.domain.net
Target type=Database Instance
Target name=mydatabaseinstance1
Categories=Diagnostics, Fault
Message=Internal error (ORA 600 [kwqitnmphe:ltbagi]) detected in /opt/app/oracle/diag/rdbms/hogcprd/hogcprd1/alert/log.xml at time/line number: Sat Jan 17 22:53:57 2015/40570.
Severity=Critical
Event reported time=Jan 17, 2015 11:08:38 PM EST
Target Lifecycle Status=Production
Department=OEM12c
Line of Business=IT
Location=Atlanta
Operating System=Linux
Platform=x86_64
Associated Incident Id=541447
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=adrAlertLogIncidentError:genericInternalErrStack
Metric Group=Incident
Metric=Generic Internal Error
Metric value=Errors in file /opt/app/oracle/diag/rdbms/xyxyx/xyxyxprd1/trace/hogcprd1_q002_31608.trc (incident=365969):~ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi], [1], [0], [], [], [], [], [], [], [], [], []
Key Value=Sat Jan 17 22:53:57 2015/40570
Key Column 1=Time/Line Number
Rule Name=OEM12c_ManagementAgentRuleset,OEM12c_Performance_metric_rule
Rule Owner=EM_ADMIN
Update Details:
Internal error (ORA 600 [kwqitnmphe:ltbagi]) detected in /opt/app/oracle/diag/rdbms/xyxyxy/xyxyxprd1/alert/log.xml at time/line number: Sat Jan 17 22:53:57 2015/40570.




Original:.*ORA-0*(54|1142|1146)\D.*
New One:.*ORA-0*(54|1142|1146|600:.*\[kwqitnmphe[^\]]*\])\D.*

 Reference: EM 12c : How To Filter Specific ORA-00600 Error In Order To Avoid ORA-600 Alerts to be Raised In Console ? [ID 1592764.1]
Categories: Development

Customizing EM12c Email Format for Notification

Fri, 2015-01-23 15:16
Customization highlighted below:
========================BODY=================================
-- Enterprise Manager Default Event Template

-- Subject of an e-mail is rendered in one line.
-- The resulting text from the following logic will be concatenated together into one line.

-- if this is a repeat email
-- show the repeat count
[IF NOTIF_TYPE EQ "NOTIF_REPEAT"]
    \[[REPEAT_LABEL] #[REPEAT_COUNT]\]
[ENDIF]
[EM_EVENT_PREFIX]:

-- if it is an email for success or failure of corrective action
-- show the name and execution status of the corrective action
[IF NOTIF_TYPE EQ "NOTIF_CA"]
    CA:[CA_JOB_NAME]:[CA_JOB_STATUS]
[ELSE] -- Regular email for metric alert
    [TARGET_LIFECYCLE_STATUS]:[TARGET_NAME]:=>[SEVERITY]
[ENDIF]
-- Show message if available
- [TARGET_NAME]:[ORCL_GTP_DEPARTMENT]:[MESSAGE]


===========================SUBJECT==============================

-- Enterprise Manager default template

[IF ORCL_GTP_DEPARTMENT NOT NULL]
 ===========================================================
 *****ITSM Please contact  [ORCL_GTP_DEPARTMENT]  Department for this Incident Alert******
 ===========================================================
[ENDIF]


-- if the email is for the status of corrective action
-- show the details of the execution of the corrective action
[IF NOTIF_TYPE EQ "NOTIF_CA"]
    [CA_JOB_NAME_LABEL]=[CA_JOB_NAME]
    [CA_JOB_OWNER_LABEL]=[CA_JOB_OWNER]
    [CA_JOB_STATUS_LABEL]=[CA_JOB_STATUS]
    [CA_JOB_STEP_OUTPUT_LABEL]=[CA_JOB_STEP_OUTPUT]
[ENDIF]

-- Source object name is the entity raising the issue.
[IF SOURCE_OBJ_NAME NOT NULL]
    [SOURCE_OBJ_TYPE] [NAME_LABEL]=[SOURCE_OBJ_NAME]
    [SOURCE_OBJ_TYPE] [OWNER_LABEL]=[SOURCE_OBJ_OWNER]
[ENDIF]
[IF SOURCE_OBJ_SUB_TYPE NOT NULL]
    [SOURCE_OBJ_TYPE] [TYPE_LABEL]=[SOURCE_OBJ_SUB_TYPE]
[ENDIF]
[IF HOST_NAME NOT NULL]
    [HOST_NAME_LABEL]=[HOST_NAME]
[ENDIF]
-- Target name links to the respective target home page
-- in Enterprise Manager console
[IF TARGET_NAME NOT NULL]
    [TARGET_TYPE_LABEL]=[TARGET_TYPE]
    [TARGET_NAME_LABEL]=[TARGET_NAME]
[ENDIF]


[IF CATEGORIES NOT NULL]
    [CATEGORIES_LABEL]=[CATEGORIES]
[ENDIF]
[MESSAGE_LABEL]=[MESSAGE]
[IF ACTION_MSG NOT NULL]
    [ACTION_MSG_LABEL]=[ACTION_MSG]
[ENDIF]
[SEVERITY_LABEL]=[SEVERITY]
[EVENT_REPORTED_TIME_LABEL]=[EVENT_REPORTED_TIME]

[IF TARGET_LIFECYCLE_STATUS NOT NULL]
    [TARGET_LIFECYCLE_STATUS_LABEL]=[TARGET_LIFECYCLE_STATUS]
[ENDIF]
[USER_DEFINED_TARGET_PROP]

[IF ASSOC_INCIDENT_ID NOT NULL]
    [ASSOC_INCIDENT_ID_LABEL]=[ASSOC_INCIDENT_ID]
    [ASSOC_INCIDENT_STATUS_LABEL]=[ASSOC_INCIDENT_STATUS]
    [ASSOC_INCIDENT_OWNER_LABEL]=[ASSOC_INCIDENT_OWNER]
    [ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER_LABEL]=[ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER]
    [ASSOC_INCIDENT_PRIORITY_LABEL]=[ASSOC_INCIDENT_PRIORITY]
    [ASSOC_INCIDENT_ESCALATION_LEVEL_LABEL]=[ASSOC_INCIDENT_ESCALATION_LEVEL]
[ENDIF]
[EVENT_TYPE_LABEL]=[EVENT_TYPE]
[EVENT_NAME_LABEL]=[EVENT_NAME]
-- if it is a repeat email, show the repeat count
[IF NOTIF_TYPE EQ "NOTIF_REPEAT"]
    [REPEAT_COUNT_LABEL]=[REPEAT_COUNT]
[ENDIF]
-- Event Dedup related Attributes
[IF TOTAL_OCCURRENCE_COUNT NOT NULL]
    [TOTAL_OCCURRENCE_COUNT_LABEL]=[TOTAL_OCCURRENCE_COUNT]
[ENDIF]
[IF CURRENT_OCCURRENCE_COUNT NOT NULL]
    [CURRENT_OCCURRENCE_COUNT_LABEL]=[CURRENT_OCCURRENCE_COUNT]
[ENDIF]
[IF CURRENT_FIRST_OCCUR_DATE NOT NULL]
    [CURRENT_FIRST_OCCUR_DATE_LABEL]=[CURRENT_FIRST_OCCUR_DATE]
[ENDIF]
[IF CURRENT_LAST_OCCUR_DATE NOT NULL]
    [CURRENT_LAST_OCCUR_DATE_LABEL]=[CURRENT_LAST_OCCUR_DATE]
[ENDIF]
[EVENT_TYPE_ATTRS]

[IF RCA_STATUS NOT NULL]
    [RCA_STATUS_LABEL]=[RCA_STATUS]
[ENDIF]

-- Root Cause Analysis details shows up when available. This is
-- normally applies to availability alerts for service targets
[RCA_DETAILS]

[RULE_NAME_LABEL]=[RULE_NAME]
[RULE_OWNER_LABEL]=[RULE_OWNER]
-- Check if any updates
[IF UPDATES NOT NULL]
[UPDATES_LABEL]:[UPDATES]
[ENDIF]
Categories: Development

How to uninstall/remove EM12c Agent

Tue, 2014-11-11 15:59
./emctl stop agent

/opt/app/oracle/agent12c/core/12.1.0.3.0/perl/bin/perl /opt/app/oracle/agent12c/core/12.1.0.3.0/sysman/install/AgentDeinstall.pl -agentHome /opt/app/oracle/agent12c

rm -Rf /opt/app/oracle/agent12c

From host where emcli is installed:
emcli delete_target -name="test.example.com:1836" -type="oracle_emd" -delete_monitored_targets –async

Reference:
https://docs.oracle.com/cd/E24628_01/install.121/e24089/deinstall_agent.htm#CBBCEJHG

Check any left out info:
select target_name,target_type from MGMT_TARGETS_DELETE where DELETE_COMPLETE_TIME is null

select target_name,target_type from MGMT_TARGETS_DELETE where target_name like '%spare%' and target_type='oracle_emd'



You can try:

exec mgmt_admin.delete_target('target_name','target_type');


If the agent install fails to register with OEM than try:

     cd /opt/app/oracle/agent12c/agent_inst/bin/
    /opt/app/oracle/agent12c/agent_inst/bin/emctl secure agent
    /opt/app/oracle/agent12c/agent_inst/bin/emctl start agent
    /opt/app/oracle/agent12c/agent_inst/bin/emctl config agent addInternalTargets

   /opt/app/oracle/agent12c/agent_inst/bin/emctl upload
Categories: Development

Seeing slow startup of SOA OSB and other Java based application then verify Entropy

Mon, 2014-11-10 09:20
We faced slow startup of Domain Creation, slow startup  of Domain and resolved using proper Entropy settings:
You should be able to select the faster-but-slightly-less-secure /dev/urandom on Linux using:
$JAVA_HOME/jre/lib/security/java.security
Default /dev/urandom is configured, but as mentioned this is ignored by Java.
-Djava.security.egd=file:/dev/urandom
However, this doesn’t work with Java 5 and later (Java Bug 6202721). The suggested work-around is to use:
-Djava.security.egd=file:/dev/./urandom(note the extra ‘/./’)
 
You can also set in your environment like below in setDomainEnv.sh
if [ "${USER_MEM_ARGS}" != "" ] ; then
MEM_ARGS="${USER_MEM_ARGS}
export MEM_ARGS
fi
MEM_ARGS="${MEM_ARGS} -Djava.security.egd=file:/dev/./urandom"
 
 
Or at runtime:
export CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"
/u01/app/oracle/product/fmw/wlserver_12.1/common/bin/config.sh
 
References:
http://theheat.dk/blog/?p=1539
http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom

















Categories: Development

Getting too much alerts for Database Time Spent Waiting (%) Metric in EM12c ?

Thu, 2014-11-06 11:17

 

It is Better to set the Database Time Spent Waiting (%)= 99 in Monitoring Templates , so that any new Database discovered will gets synced with the Template collection and changes the default 30% to 99%.

 

Verify that the DBA_THRESHOLDS in target database is updated.

i.e the below query should return no rows:

SELECT metrics_name, critical_value, warning_value, object_name, status
FROM SYS.DBA_THRESHOLDS
where metrics_name like '%Database Time Spent%';

 

Refer:

How to Disable Alerts for The Database Time Spent Waiting (%) Metric? (Doc ID 1500074.1)

Database Server Generated Metrics Alerts Not Cleared In Grid Console Even After Disabling the Metric (Doc ID 786520.1)

"Database Time Spent Waiting (%)" at 100 for Event Class "Other" when Database is not Under Load (Doc ID 1526552.1)

http://docs.oracle.com/cd/E24628_01/em.121/e25160/oracle_database.htm#EMDBM3180

Categories: Development

Find and Replace in a file

Wed, 2014-11-05 14:12

Had to replace all the occurrence of wrong  inventory then use this:

grep -rl 'oraInventory' .  | xargs sed -i 's|'/home/emuser/oraInventory'|'/opt/app/oracle/oraInventory'|g'

Categories: Development

Creating FMW12c Domain using EM12c User Defined Deployment Procedure

Tue, 2014-10-21 16:03
1. Installing Software example: Install JDK, INFRA, OSB, RCU Creation in 5 minutes, 58 secondsimage

2. Second example: creating BAM12c 2 node cluster in 9 minutes,  Create RCU, Create domain, Extend Domain, startNodemanger, start admin, start Nodemanager on Mgrserver, startall ManagedServers
image
2 Nodes, BAM Cluster
Categories: Development