RE: Anyone running a 10.2.0.3 RAC on Linux?

From: <Christopher.Taylor2_at_parallon.net>
Date: Wed, 27 Feb 2013 22:05:29 -0600
Message-ID: <F05D8DF1FB25F44085DB74CB916678E88582A9344D_at_NADCWPMSGCMS10.hca.corpad.net>



It's red hat 5.8.
Looking at oradism in the ORACLE_HOME for the database shows:

-r-sr-s--- 1 root oradba 14931 Dec 20 2006 /orab/app/oracle/product/1020/db/bin/oradism

Does that look suspect (it's different than yours but there is a setuid set)

Chris

From: tanel_at_poderc.com [mailto:tanel_at_poderc.com] On Behalf Of Tanel Poder Sent: Wednesday, February 27, 2013 5:52 PM To: oratune
Cc: Taylor Christopher - Nashville; Dave.Herring_at_acxiom.com; oracle-l_at_freelists.org Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?

The actual priorities in kernel-land are positive, so the negative numbers you see are something that the userland tools and libraries calculate/derive for you. -51 is weird though, perhaps it's a bug with ps or some library function. You can use ps to report which scheduler policy is used for that process, perhaps that sheds some light.

Are you on RHEL6+ by any chance? The newer kernels show the actual kernel prio and policy in proc filesystem:

[root_at_oel6 ~]# grep ^p /proc/40695/sched

policy                             :                    2
prio                               :                   79

The policy #2 corresponds to round-robin real time policy as shown in sched.h:

$ grep SCHED /usr/include/linux/sched.h
#ifndef _LINUX_SCHED_H
#define _LINUX_SCHED_H
#define SCHED_NORMAL 0
#define SCHED_FIFO 1
#define SCHED_RR 2
#define SCHED_BATCH 3

/* SCHED_ISO: reserved but not implemented yet */
#define SCHED_IDLE 5

/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */
#define SCHED_RESET_ON_FORK 0x40000000

Also check the ownership and setuid bit on the oradism binary too. In addition to the Solaris) DISM memory locking work, this binary is used for elevating the priority of the _high_priority_processes processes (as a regular process can't just upgrade its priority to real time).

$ ls -l $ORACLE_HOME/bin/oradism
-rwsr-x--- 1 root dba 71793 Sep 5 2010 /u01/app/oracle/product/11.2.0/db_1/bin/oradism

So if you've cloned/copied Oracle homes just via a copy or untarred as oracle user (not root), then this setuid bit gets lost and Oracle can't hike the required processes priority during startup.

This may explain some of the differences in priorities, but -51 is weird on linux ...

--
Tanel Poder
Enkitec (The Exadata Experts)
Training<http://blog.tanelpoder.com/seminar/> | Troubleshooting<http://blog.tanelpoder.com/> | Exadata<http://www.amazon.com/Expert-Oracle-Exadata-Apress/dp/1430233923> | Voicee App<http://voic.ee/>


On Wed, Feb 27, 2013 at 11:21 PM, David Fitzjarrell <oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>> wrote:
My guess would be the latter case -- new 'features' and parameters rarely work as expected or intended when introduced.  As stated before 10.2.0.3 was a rough release on many fronts.

David Fitzjarrell



________________________________
From: "Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>" <Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>>
To: Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>; oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Sent: Wednesday, February 27, 2013 1:51 PM
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

My fault - read your version list too fast.
I'm betting that parameter is too aggressive in 10.2.0.3 - or doesn't work as expected.
Thanks though!

Chris


From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>]
Sent: Wednesday, February 27, 2013 2:50 PM
To: Taylor Christopher - Nashville; oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

No 10.2.0.3, just 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3.  It looks this is a new param added in 10.2.0.3.

10.2.0.2
No rows.

10.2.0.4
NAME                      VALUE      ISDEFAULT ISMODIFIED
------------------------- ---------- --------- --------------------------------
_high_priority_processes  LMS*      TRUE      FALSE

11.2.0.1
NAME                      VALUE      ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------
_high_priority_processes  LMS*|VKTM  TRUE      FALSE

11.2.0.3
NAME                      VALUE      ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------
_high_priority_processes  LMS*|VKTM  TRUE      FALSE

DAVID HERRING
DBA
Acxiom Corporation
EML  dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com><mailto:dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com>>
TEL    630.944.4762
MBL  630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA
WWW.ACXIOM.COM<http://WWW.ACXIOM.COM><http://www.acxiom.com/>

[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor>  [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173>  [cid:image003.png_at_01CE14F9.E81D9C70] <http://twitter.com/acxio>

[cid:image004.png_at_01CE14F9.E81D9C70]

From: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net><mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>> [mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>]
Sent: Wednesday, February 27, 2013 2:37 PM
To: Herring Dave - dherri; oratune_at_yahoo.com<mailto:oratune_at_yahoo.com><mailto:oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

Ran SQL to list hidden parameters:

NAME                                              VALUE                                              ISDEFAULT ISMODIFIED
-------------------------------------------------- -------------------------------------------------- --------- --------------------------------
_high_priority_processes                          LMS*                                              TRUE      FALSE


David H - curious if you can run this as sysdba and see if its the same in your 10.2.0.3<http://10.2.0.3>:

SELECT name,value,ISDEFAULT,ISMODIFIED FROM
(SELECT x.inst_id AS INST_ID,
          x.indx + 1 AS NUM,
          ksppinm AS NAME,
          ksppity AS TYPE,
          ksppstvl AS VALUE,
          ksppstdvl AS DISPLAY_VALUE,
          ksppstdf AS ISDEFAULT,
          DECODE (BITAND (ksppiflg / 256, 1), 1, 'TRUE', 'FALSE')
            AS ISSES_MODIFIABLE,
          DECODE (BITAND (ksppiflg / 65536, 3),
                  1, 'IMMEDIATE',
                  2, 'DEFERRED',
                  3, 'IMMEDIATE',
                  'FALSE')
            AS ISSYS_MODIFIABLE,
          DECODE (BITAND (ksppiflg, 4),
                  4, 'FALSE',
                  DECODE (BITAND (ksppiflg / 65536, 3), 0, 'FALSE', 'TRUE'))
            AS ISINSTANCE_MODIFIABLE,
          DECODE (BITAND (ksppstvf, 7),
                  1, 'MODIFIED',
                  4, 'SYSTEM_MOD',
                  'FALSE')
            AS ISMODIFIED,
          DECODE (BITAND (ksppstvf, 2), 2, 'TRUE', 'FALSE') AS ISADJUSTED,
          DECODE (BITAND (ksppilrmflg / 64, 1), 1, 'TRUE', 'FALSE')
            AS ISDEPRECATED,
          ksppdesc AS DESCRIPTION,
          ksppstcmnt AS UPDATE_COMMENT,
          ksppihash AS HASH
    FROM sys.x$ksppi x, sys.x$ksppcv y
    WHERE (x.indx = y.indx)
)
where NAME like lower('%&ENTER_PARM_NAME%')
/

From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>]
Sent: Wednesday, February 27, 2013 2:26 PM
To: David Fitzjarrell; Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

I checked 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3 and all are at "-2".  All clusters are on RHEL.

DAVID HERRING
DBA
Acxiom Corporation
EML  dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com><mailto:dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com>>
TEL    630.944.4762
MBL  630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA
WWW.ACXIOM.COM<http://WWW.ACXIOM.COM><http://www.acxiom.com/>

[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor>  [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173>  [cid:image003.png_at_01CE14F9.E81D9C70] <http://twitter.com/acxio>

[cid:image004.png_at_01CE14F9.E81D9C70]

From: David Fitzjarrell [mailto:oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>]
Sent: Wednesday, February 27, 2013 2:23 PM
To: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net><mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>>; Herring Dave - dherri; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?

Yes, it does.  My former coworker reports that on their lone 10.2.0.3 RAC on linux the lms processes are running at a priority of 90, which also seems odd as Dave Herring reports the same priority on his 10.2.0.2 RAC databases as I do for our 11.2.0.3 databases (-2).  It would appear that 10.2.0.3 is doing some 'interesting' things with some of the cluster services.  Then, 10.2.0.3 wasn't very good on statistics, either, causing other performance problems.

David Fitzjarrell


From: "Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net><mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>>" <Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net><mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>>>
To: Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com><mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Sent: Wednesday, February 27, 2013 7:45 AM
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

So does -51 PRIORITY seem crazy to you as it does to me for LMS processes?

Chris


-----Original Message-----
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com><mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>>]
Sent: Wednesday, February 27, 2013 8:44 AM
To: Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?

Chris,

What flavor and rev of Linux?  DRM is pretty buggy under 10g and because of crashes related to it we've done a bit of research into it.  The best thing we found was to target longer running cursors to avoid remastering in the first place.  I know that sounds like a no-brainer - tune long running cursors - but in this case it was tuning cursors involving data more likely to be shared across the nodes and of those cursors, hit the longest running ones first.

I've got a number of clusters on RHEL 4.6.  They're all 10.2.0.2<http://10.2.0.2>:

% for X in 0 1 2 3; do    ssh oracle_at_xxx${X} 'ps -u oracle -O pid,user,priority,nice,vsize,rss,share,pcpu,size | grep lms | grep -v grep'; done
18256 18256 oracle    -2  - 151180 224496 -  1.9 2152 S ?  18:47:39 asm_lms0_+ASM1
19345 19345 oracle    -2  - 161024 17858588 -  4.3 2744 S ? 1-17:09:49 ora_lms0_XXXXXX1
19349 19349 oracle    -2  - 152704 17859424 -  3.0 2616 S ? 1-04:43:51 ora_lms1_XXXXXX1
19353 19353 oracle    -2  - 152704 17859644 -  3.2 2616 S ? 1-06:23:23 ora_lms2_XXXXXX1
19357 19357 oracle    -2  - 152704 17854440 -  3.0 2616 S ? 1-04:58:33 ora_lms3_XXXXXX1
19361 19361 oracle    -2  - 152704 17857896 -  3.1 2616 S ? 1-06:08:38 ora_lms4_XXXXXX1
19365 19365 oracle    -2  - 152704 17859636 -  3.0 2616 S ? 1-04:36:10 ora_lms5_XXXXXX1
  974  974 oracle    -2  - 152704 17859744 -  4.4 2616 R ? 1-17:51:04 ora_lms0_XXXXXX2
  982  982 oracle    -2  - 151660 17857316 -  3.1 2616 S ? 1-05:42:41 ora_lms1_XXXXXX2
  990  990 oracle    -2  - 152704 17859196 -  3.3 2616 R ? 1-07:18:02 ora_lms2_XXXXXX2
  996  996 oracle    -2  - 151660 17857312 -  3.1 2616 S ? 1-05:41:33 ora_lms3_XXXXXX2
1004  1004 oracle    -2  - 152704 17859388 -  3.2 2616 S ? 1-06:25:44 ora_lms4_XXXXXX2
1012  1012 oracle    -2  - 151660 17857436 -  3.1 2616 S ? 1-05:57:22 ora_lms5_XXXXXX2
32754 32754 oracle    -2  - 151180 224500 -  2.0 2152 S ?  18:59:35 asm_lms0_+ASM2
2876  2876 oracle    -2  - 152704 17855760 -  3.8 2616 S ? 22:23:14 ora_lms0_XXXXXX3
2880  2880 oracle    -2  - 152704 17855616 -  2.4 2616 S ? 14:07:36 ora_lms1_XXXXXX3
2893  2893 oracle    -2  - 152704 17855504 -  2.5 2616 S ? 14:52:41 ora_lms2_XXXXXX3
2914  2914 oracle    -2  - 151660 17853452 -  2.4 2616 S ? 14:19:18 ora_lms3_XXXXXX3
2918  2918 oracle    -2  - 151664 17853728 -  2.4 2620 S ? 14:34:36 ora_lms4_XXXXXX3
2924  2924 oracle    -2  - 151664 17853452 -  2.4 2620 S ? 14:25:32 ora_lms5_XXXXXX3
22555 22555 oracle    -2  - 151180 221224 -  1.7 2152 R ?  10:13:15 asm_lms0_+ASM3
9304  9304 oracle    -2  - 151180 210960 -  2.0 2152 S ?  18:59:35 asm_lms0_+ASM4
9892  9892 oracle    -2  - 151792 17853164 -  4.4 2748 S ? 1-18:08:04 ora_lms0_XXXXXX4
9909  9909 oracle    -2  - 161028 17853108 -  3.1 2748 R ? 1-06:06:52 ora_lms1_XXXXXX4
9933  9933 oracle    -2  - 151792 17852424 -  3.3 2748 S ? 1-07:40:15 ora_lms2_XXXXXX4
9937  9937 oracle    -2  - 151792 17852528 -  3.1 2748 S ? 1-06:04:05 ora_lms3_XXXXXX4
9941  9941 oracle    -2  - 151792 17852492 -  3.3 2748 S ? 1-07:27:18 ora_lms4_XXXXXX4
9945  9945 oracle    -2  - 151792 17852584 -  3.2 2748 S ? 1-06:18:26 ora_lms5_XXXXXX4

DAVID HERRING
DBA
Acxiom Corporation
EML  dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com><mailto:dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com>>
TEL    630.944.4762
MBL  630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA http://www.acxiom.com/

-----Original Message-----
From: oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org><mailto:oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org>> [mailto:oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org><mailto:oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org>>] On Behalf Of Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net><mailto:Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>>
Sent: Tuesday, February 26, 2013 6:39 PM
To: oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org><mailto:oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>>
Subject: Anyone running a 10.2.0.3 RAC on Linux?

I'm curious what priority your LMS processes run in?  One of our DBAs here has a system that is giving him fits and I was looking at it and it seems [to me] that LMS priority for his clusters is way too low but I'm still fairly new to RAC and digging into the internals so wanted to get your thoughts?
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
...
...
9022 oracle  -51  0 12.2g  11g  11g S  0.3 17.6  1636:16 ora_lms0_XXXXXXXP6N1
9026 oracle  -51  0 12.2g  11g  11g S  0.3 17.6  1252:31 ora_lms1_XXXXXXXP6N1
16296 oracle  -51  0 8352m 2.5g 2.5g S  2.3  3.9  7149:56 ora_lms0_XXXXXXXP1N1
16314 oracle  -51  0 8352m 2.5g 2.5g S  2.3  3.9  6761:01 ora_lms1_XXXXXXXP1N1
18902 oracle  -51  0 8344m 2.2g 2.2g S  1.0  3.5  1822:21 ora_lms0_XXXXXXXP2N1
18910 oracle  -51  0 8344m 2.2g 2.2g S  1.0  3.5  1611:40 ora_lms1_XXXXXXXP2N1
21547 oracle  -51  0 10.1g 3.1g 3.1g S  2.3  5.0  2742:04 ora_lms0_XXXXXXXP3N1
21551 oracle  -51  0 10.1g 3.1g 3.1g S  2.6  5.0  2444:59 ora_lms1_XXXXXXXP3N1
24777 oracle  -51  0 8344m 1.6g 1.5g S  1.0  2.5  2769:58 ora_lms0_XXXXXXXP4N1
24781 oracle  -51  0 8344m 1.6g 1.5g S  1.0  2.5  2596:55 ora_lms1_XXXXXXXP4N1
29630 oracle  -51  0  275m  69m  55m S  0.0  0.1 845:01.75 asm_lms0_+ASM1
30825 oracle  -51  0 8344m 2.0g 1.9g S  0.0  3.1  1656:07 ora_lms0_XXXXXXXP5N1
30832 oracle  -51  0 8344m 2.0g 1.9g S  0.3  3.1  1417:34 ora_lms1_XXXXXXXP5N1

Chris


--
http://www.freelists.org/webpage/oracle-l


***************************************************************************
The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged.

If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.

If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system.

Thank You.
****************************************************************************

--
http://www.freelists.org/webpage/oracle-l



--
http://www.freelists.org/webpage/oracle-l
--
http://www.freelists.org/webpage/oracle-l



--
http://www.freelists.org/webpage/oracle-l
Received on Thu Feb 28 2013 - 05:05:29 CET

Original text of this message