Re: AWS RDS targets and EM Job alerts
From: <peter.sharman_at_westnet.com.au>
Date: Thu, 22 Dec 2016 04:57:03 +0800
Message-Id: <1c244524629904b45eb66176b51b5b2ae9b720ec_at_webmail.westnet.com.au>
Dave I think you're screwed at this point. The agent needs that nmo ability specifically so it CAN do things securely without having to access root. I think you're going to end up with Amazon saying it can't, by design, and Oracle saying it must, by design. The only way out of this I can see is not using dynamic groups, but that will only resolve this issue. There will undoubtedly be others that arise from the nmo standoff though, so that's only a partial and (by the sounds of your description) a painful solution to the issue. I would take this one up with the security PM for EM. When I left Oracle, that was Angeline Dhanarani, but obviously that might have changed. :) Pete ----- Original Message ----- From: HerringD_at_DNB.com To:"oracle-l_at_freelists.org" Cc: Sent:Wed, 21 Dec 2016 20:42:50 +0000 Subject:AWS RDS targets and EM Job alerts Our team recently started monitoring a number of AWS RDS targets and now run into an issue with our existing jobs in OEM. We're running EM 12.1.0.4 and have a number of jobs scheduled in EM that run OS commands. AWS recently added an option they named "OEM_AGENT" which involves the installation of a 12c management agent, which is great in that we have greater monitoring and alerting options for these targets. The problem comes in with OS command EM jobs. They're failing when executing against the AWS RDS hosts related to "nmo" permissions. That's by design from an AWS standpoint. While they have run "root.sh", they go back and modify permissions on the "nmo" binary so that it can't be accessed by the agent, for security reasons. We have thousands of targets promoted to EM and use dynamic groups so by default these AWS RDS hosts are getting added to our "host" group (lets say ORAL_HOST). Our EM jobs that we want to execute OS commands reference group ORAL_HOST, now giving us alerts on each execution against the AWS RDS hosts. Unfortunately at this point I can't figure out a good way of avoiding alerts on this situation. · There isn't a way to exclude specific hosts from a Dynamic Group. You can define additional filtering criteria on Target Properties but for that to work we'd first need to set certain properties for ALL targets and then remember to set them properly for new ones, all so that those properties would be unique to the AWS RDS targets so that they can be filtered out. · A blackout won't work because blackouts apply to targets, not to jobs. You can specify that EM Jobs not run during the blackout but unfortunately you'd end up blacking out all other alerting in the process. · Adding logic to the EM Job command to skip certain hosts won't help because the job first needs to connect to the host as a given user to run the command and it's the connection part that fails, as "nmo" is accessed earlier in the stack. In other words the failure happens before the job has a chance to run anything. · A specialized Incident Rule won't help because by default the alerts we get from jobs aren't managed by Incident Rules. Although I can create a new Incident Rule on Jobs and have it match 1 or more of the OS Command-based jobs, I can't stop the original alert from arriving in our Inbox. Anyone have any other ideas on how to avoid these alerts? Regards, Dave
Date: Thu, 22 Dec 2016 04:57:03 +0800
Message-Id: <1c244524629904b45eb66176b51b5b2ae9b720ec_at_webmail.westnet.com.au>
Dave I think you're screwed at this point. The agent needs that nmo ability specifically so it CAN do things securely without having to access root. I think you're going to end up with Amazon saying it can't, by design, and Oracle saying it must, by design. The only way out of this I can see is not using dynamic groups, but that will only resolve this issue. There will undoubtedly be others that arise from the nmo standoff though, so that's only a partial and (by the sounds of your description) a painful solution to the issue. I would take this one up with the security PM for EM. When I left Oracle, that was Angeline Dhanarani, but obviously that might have changed. :) Pete ----- Original Message ----- From: HerringD_at_DNB.com To:"oracle-l_at_freelists.org" Cc: Sent:Wed, 21 Dec 2016 20:42:50 +0000 Subject:AWS RDS targets and EM Job alerts Our team recently started monitoring a number of AWS RDS targets and now run into an issue with our existing jobs in OEM. We're running EM 12.1.0.4 and have a number of jobs scheduled in EM that run OS commands. AWS recently added an option they named "OEM_AGENT" which involves the installation of a 12c management agent, which is great in that we have greater monitoring and alerting options for these targets. The problem comes in with OS command EM jobs. They're failing when executing against the AWS RDS hosts related to "nmo" permissions. That's by design from an AWS standpoint. While they have run "root.sh", they go back and modify permissions on the "nmo" binary so that it can't be accessed by the agent, for security reasons. We have thousands of targets promoted to EM and use dynamic groups so by default these AWS RDS hosts are getting added to our "host" group (lets say ORAL_HOST). Our EM jobs that we want to execute OS commands reference group ORAL_HOST, now giving us alerts on each execution against the AWS RDS hosts. Unfortunately at this point I can't figure out a good way of avoiding alerts on this situation. · There isn't a way to exclude specific hosts from a Dynamic Group. You can define additional filtering criteria on Target Properties but for that to work we'd first need to set certain properties for ALL targets and then remember to set them properly for new ones, all so that those properties would be unique to the AWS RDS targets so that they can be filtered out. · A blackout won't work because blackouts apply to targets, not to jobs. You can specify that EM Jobs not run during the blackout but unfortunately you'd end up blacking out all other alerting in the process. · Adding logic to the EM Job command to skip certain hosts won't help because the job first needs to connect to the host as a given user to run the command and it's the connection part that fails, as "nmo" is accessed earlier in the stack. In other words the failure happens before the job has a chance to run anything. · A specialized Incident Rule won't help because by default the alerts we get from jobs aren't managed by Incident Rules. Although I can create a new Incident Rule on Jobs and have it match 1 or more of the OS Command-based jobs, I can't stop the original alert from arriving in our Inbox. Anyone have any other ideas on how to avoid these alerts? Regards, Dave
-- http://www.freelists.org/webpage/oracle-lReceived on Wed Dec 21 2016 - 21:57:03 CET