Re: Anyone running a 10.2.0.3 RAC on Linux?
Date: Thu, 28 Feb 2013 09:30:13 -0800 (PST)
Message-ID: <1362072613.30575.YahooMailNeo_at_web121606.mail.ne1.yahoo.com>
On ours the suid is there for 'root' (the owner) but not for the group.
David Fitzjarrell
From: "Christopher.Taylor2_at_parallon.net" <Christopher.Taylor2_at_parallon.net> To: tanel_at_tanelpoder.com; oratune_at_yahoo.com Cc: Dave.Herring_at_acxiom.com; oracle-l_at_freelists.org Sent: Wednesday, February 27, 2013 9:05 PM Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
It's red hat 5.8.
Looking at oradism in the ORACLE_HOME for the database shows:
-r-sr-s--- 1 root oradba 14931 Dec 20 2006 /orab/app/oracle/product/1020/db/bin/oradism
Does that look suspect (it's different than yours but there is a setuid set)
Chris
From:tanel_at_poderc.com [mailto:tanel_at_poderc.com] On Behalf Of Tanel Poder
Sent: Wednesday, February 27, 2013 5:52 PM
To: oratune
Cc: Taylor Christopher - Nashville; Dave.Herring_at_acxiom.com; oracle-l_at_freelists.org
Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?
The actual priorities in kernel-land are positive, so the negative numbers you see are something that the userland tools and libraries calculate/derive for you. -51 is weird though, perhaps it's a bug with ps or some library function. You can use ps to report which scheduler policy is used for that process, perhaps that sheds some light.
Are you on RHEL6+ by any chance? The newer kernels show the actual kernel prio and policy in proc filesystem:
[root_at_oel6 ~]# grep ^p /proc/40695/sched
policy : 2
prio : 79
The policy #2 corresponds to round-robin real time policy as shown in sched.h:
$ grep SCHED /usr/include/linux/sched.h
#ifndef _LINUX_SCHED_H #define _LINUX_SCHED_H #define SCHED_NORMAL 0 #define SCHED_FIFO 1 #define SCHED_RR 2 #define SCHED_BATCH 3
/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5
/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000
Also check the ownership and setuid bit on the oradism binary too. In addition to the Solaris) DISM memory locking work, this binary is used for elevating the priority of the _high_priority_processes processes (as a regular process can't just upgrade its priority to real time).
$ ls -l $ORACLE_HOME/bin/oradism
-rwsr-x--- 1 root dba 71793 Sep 5 2010 /u01/app/oracle/product/11.2.0/db_1/bin/oradism
So if you've cloned/copied Oracle homes just via a copy or untarred as oracle user (not root), then this setuid bit gets lost and Oracle can't hike the required processes priority during startup.
This may explain some of the differences in priorities, but -51 is weird on linux ...
--
Tanel Poder
Enkitec (The Exadata Experts)
Training | Troubleshooting | Exadata | Voicee App
On Wed, Feb 27, 2013 at 11:21 PM, David Fitzjarrell <oratune_at_yahoo.com> wrote:
My guess would be the latter case -- new 'features' and parameters rarely work as expected or intended when introduced. As stated before 10.2.0.3 was a rough release on many fronts.
David Fitzjarrell
From: "Christopher.Taylor2_at_parallon.net" <Christopher.Taylor2_at_parallon.net> To: Dave.Herring_at_acxiom.com; oratune_at_yahoo.com; oracle-l_at_freelists.org Sent: Wednesday, February 27, 2013 1:51 PM Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
My fault - read your version list too fast. I'm betting that parameter is too aggressive in 10.2.0.3 - or doesn't work as expected. Thanks though!
Chris
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com] Sent: Wednesday, February 27, 2013 2:50 PM To: Taylor Christopher - Nashville; oratune_at_yahoo.com; oracle-l_at_freelists.org Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
No 10.2.0.3, just 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3. It looks this is a new param added in 10.2.0.3.
10.2.0.2
No rows.
10.2.0.4
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- --------------------------------_high_priority_processes LMS* TRUE FALSE
11.2.0.1
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------_high_priority_processes LMS*|VKTM TRUE FALSE
11.2.0.3
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------_high_priority_processes LMS*|VKTM TRUE FALSE
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA http://www.acxiom.com/<http://www.acxiom.com/>
[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor> [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173> [cid:image003.png_at_01CE14F9.E81D9C70] <http://twitter.com/acxio>
[cid:image004.png_at_01CE14F9.E81D9C70]
From: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net> [mailto:Christopher.Taylor2_at_parallon.net] Sent: Wednesday, February 27, 2013 2:37 PM To: Herring Dave - dherri; oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
Ran SQL to list hidden parameters:
NAME VALUE ISDEFAULT ISMODIFIED
-------------------------------------------------- -------------------------------------------------- --------- --------------------------------_high_priority_processes LMS* TRUE FALSE
David H - curious if you can run this as sysdba and see if its the same in your 10.2.0.3:
SELECT name,value,ISDEFAULT,ISMODIFIED FROM
(SELECT x.inst_id AS INST_ID,
x.indx + 1 AS NUM,
ksppinm AS NAME,
ksppity AS TYPE,
ksppstvl AS VALUE,
ksppstdvl AS DISPLAY_VALUE,
ksppstdf AS ISDEFAULT,
DECODE (BITAND (ksppiflg / 256, 1), 1, 'TRUE', 'FALSE')
AS ISSES_MODIFIABLE,
DECODE (BITAND (ksppiflg / 65536, 3),
1, 'IMMEDIATE',
2, 'DEFERRED',
3, 'IMMEDIATE',
'FALSE')
AS ISSYS_MODIFIABLE,
DECODE (BITAND (ksppiflg, 4),
4, 'FALSE',
DECODE (BITAND (ksppiflg / 65536, 3), 0, 'FALSE', 'TRUE'))
AS ISINSTANCE_MODIFIABLE,
DECODE (BITAND (ksppstvf, 7),
1, 'MODIFIED',
4, 'SYSTEM_MOD',
'FALSE')
AS ISMODIFIED,
DECODE (BITAND (ksppstvf, 2), 2, 'TRUE', 'FALSE') AS ISADJUSTED,
DECODE (BITAND (ksppilrmflg / 64, 1), 1, 'TRUE', 'FALSE')
AS ISDEPRECATED,
ksppdesc AS DESCRIPTION,
ksppstcmnt AS UPDATE_COMMENT,
ksppihash AS HASH
FROM sys.x$ksppi x, sys.x$ksppcv y
WHERE (x.indx = y.indx)
)
where NAME like lower('%&ENTER_PARM_NAME%')
/
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com] Sent: Wednesday, February 27, 2013 2:26 PM To: David Fitzjarrell; Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
I checked 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3 and all are at "-2". All clusters are on RHEL.
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA http://www.acxiom.com/<http://www.acxiom.com/>
[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor> [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173> [cid:image003.png_at_01CE14F9.E81D9C70] <http://twitter.com/acxio>
[cid:image004.png_at_01CE14F9.E81D9C70]
From: David Fitzjarrell [mailto:oratune_at_yahoo.com] Sent: Wednesday, February 27, 2013 2:23 PM To: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>; Herring Dave - dherri; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?
Yes, it does. My former coworker reports that on their lone 10.2.0.3 RAC on linux the lms processes are running at a priority of 90, which also seems odd as Dave Herring reports the same priority on his 10.2.0.2 RAC databases as I do for our 11.2.0.3 databases (-2). It would appear that 10.2.0.3 is doing some 'interesting' things with some of the cluster services. Then, 10.2.0.3 wasn't very good on statistics, either, causing other performance problems.
David Fitzjarrell
From: "Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>" <Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>> To: Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Sent: Wednesday, February 27, 2013 7:45 AM Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
So does -51 PRIORITY seem crazy to you as it does to me for LMS processes?
Chris
-----Original Message-----
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>]
Sent: Wednesday, February 27, 2013 8:44 AM
To: Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
Chris,
What flavor and rev of Linux? DRM is pretty buggy under 10g and because of crashes related to it we've done a bit of research into it. The best thing we found was to target longer running cursors to avoid remastering in the first place. I know that sounds like a no-brainer - tune long running cursors - but in this case it was tuning cursors involving data more likely to be shared across the nodes and of those cursors, hit the longest running ones first.
I've got a number of clusters on RHEL 4.6. They're all 10.2.0.2:
% for X in 0 1 2 3; do ssh oracle_at_xxx${X} 'ps -u oracle -O pid,user,priority,nice,vsize,rss,share,pcpu,size | grep lms | grep -v grep'; done 18256 18256 oracle -2 - 151180 224496 - 1.9 2152 S ? 18:47:39 asm_lms0_+ASM1 19345 19345 oracle -2 - 161024 17858588 - 4.3 2744 S ? 1-17:09:49 ora_lms0_XXXXXX1 19349 19349 oracle -2 - 152704 17859424 - 3.0 2616 S ? 1-04:43:51 ora_lms1_XXXXXX1 19353 19353 oracle -2 - 152704 17859644 - 3.2 2616 S ? 1-06:23:23 ora_lms2_XXXXXX1 19357 19357 oracle -2 - 152704 17854440 - 3.0 2616 S ? 1-04:58:33 ora_lms3_XXXXXX1 19361 19361 oracle -2 - 152704 17857896 - 3.1 2616 S ? 1-06:08:38 ora_lms4_XXXXXX1 19365 19365 oracle -2 - 152704 17859636 - 3.0 2616 S ? 1-04:36:10 ora_lms5_XXXXXX11004 1004 oracle -2 - 152704 17859388 - 3.2 2616 S ? 1-06:25:44 ora_lms4_XXXXXX2 1012 1012 oracle -2 - 151660 17857436 - 3.1 2616 S ? 1-05:57:22 ora_lms5_XXXXXX2
974 974 oracle -2 - 152704 17859744 - 4.4 2616 R ? 1-17:51:04 ora_lms0_XXXXXX2
982 982 oracle -2 - 151660 17857316 - 3.1 2616 S ? 1-05:42:41 ora_lms1_XXXXXX2
990 990 oracle -2 - 152704 17859196 - 3.3 2616 R ? 1-07:18:02 ora_lms2_XXXXXX2
996 996 oracle -2 - 151660 17857312 - 3.1 2616 S ? 1-05:41:33 ora_lms3_XXXXXX2
32754 32754 oracle -2 - 151180 224500 - 2.0 2152 S ? 18:59:35 asm_lms0_+ASM2 2876 2876 oracle -2 - 152704 17855760 - 3.8 2616 S ? 22:23:14 ora_lms0_XXXXXX3 2880 2880 oracle -2 - 152704 17855616 - 2.4 2616 S ? 14:07:36 ora_lms1_XXXXXX3 2893 2893 oracle -2 - 152704 17855504 - 2.5 2616 S ? 14:52:41 ora_lms2_XXXXXX3 2914 2914 oracle -2 - 151660 17853452 - 2.4 2616 S ? 14:19:18 ora_lms3_XXXXXX3 2918 2918 oracle -2 - 151664 17853728 - 2.4 2620 S ? 14:34:36 ora_lms4_XXXXXX3 2924 2924 oracle -2 - 151664 17853452 - 2.4 2620 S ? 14:25:32 ora_lms5_XXXXXX3 22555 22555 oracle -2 - 151180 221224 - 1.7 2152 R ? 10:13:15 asm_lms0_+ASM3 9304 9304 oracle -2 - 151180 210960 - 2.0 2152 S ? 18:59:35 asm_lms0_+ASM4 9892 9892 oracle -2 - 151792 17853164 - 4.4 2748 S ? 1-18:08:04 ora_lms0_XXXXXX4 9909 9909 oracle -2 - 161028 17853108 - 3.1 2748 R ? 1-06:06:52 ora_lms1_XXXXXX4 9933 9933 oracle -2 - 151792 17852424 - 3.3 2748 S ? 1-07:40:15 ora_lms2_XXXXXX4 9937 9937 oracle -2 - 151792 17852528 - 3.1 2748 S ? 1-06:04:05 ora_lms3_XXXXXX49941 9941 oracle -2 - 151792 17852492 - 3.3 2748 S ? 1-07:27:18 ora_lms4_XXXXXX4 9945 9945 oracle -2 - 151792 17852584 - 3.2 2748 S ? 1-06:18:26 ora_lms5_XXXXXX4
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA http://www.acxiom.com/
-----Original Message-----
From: oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org> [mailto:oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org>] On Behalf Of Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>
Sent: Tuesday, February 26, 2013 6:39 PM
To: oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Subject: Anyone running a 10.2.0.3 RAC on Linux?
I'm curious what priority your LMS processes run in? One of our DBAs here has a system that is giving him fits and I was looking at it and it seems [to me] that LMS priority for his clusters is way too low but I'm still fairly new to RAC and digging into the internals so wanted to get your thoughts?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
...
...
9022 oracle -51 0 12.2g 11g 11g S 0.3 17.6 1636:16 ora_lms0_XXXXXXXP6N1
9026 oracle -51 0 12.2g 11g 11g S 0.3 17.6 1252:31 ora_lms1_XXXXXXXP6N1
16296 oracle -51 0 8352m 2.5g 2.5g S 2.3 3.9 7149:56 ora_lms0_XXXXXXXP1N1 16314 oracle -51 0 8352m 2.5g 2.5g S 2.3 3.9 6761:01 ora_lms1_XXXXXXXP1N1 18902 oracle -51 0 8344m 2.2g 2.2g S 1.0 3.5 1822:21 ora_lms0_XXXXXXXP2N1 18910 oracle -51 0 8344m 2.2g 2.2g S 1.0 3.5 1611:40 ora_lms1_XXXXXXXP2N1 21547 oracle -51 0 10.1g 3.1g 3.1g S 2.3 5.0 2742:04 ora_lms0_XXXXXXXP3N1 21551 oracle -51 0 10.1g 3.1g 3.1g S 2.6 5.0 2444:59 ora_lms1_XXXXXXXP3N1 24777 oracle -51 0 8344m 1.6g 1.5g S 1.0 2.5 2769:58 ora_lms0_XXXXXXXP4N1 24781 oracle -51 0 8344m 1.6g 1.5g S 1.0 2.5 2596:55 ora_lms1_XXXXXXXP4N1 29630 oracle -51 0 275m 69m 55m S 0.0 0.1 845:01.75 asm_lms0_+ASM1 30825 oracle -51 0 8344m 2.0g 1.9g S 0.0 3.1 1656:07 ora_lms0_XXXXXXXP5N1 30832 oracle -51 0 8344m 2.0g 1.9g S 0.3 3.1 1417:34 ora_lms1_XXXXXXXP5N1
Chris
-- http://www.freelists.org/webpage/oracle-l *************************************************************************** The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system. Thank You. **************************************************************************** -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-lReceived on Thu Feb 28 2013 - 18:30:13 CET