OraFAQ Forum: Server Administration » kkjcre1p: unable to spawn jobq slave process

Home » RDBMS Server » Server Administration » kkjcre1p: unable to spawn jobq slave process (Oracle 10g, 10.2.04 , Sun Solaries(spark) 64bits)

Show: Today's Messages :: Polls :: Message Navigator
E-mail to friend

kkjcre1p: unable to spawn jobq slave process [message #485272]

Tue, 07 December 2010 03:40

halim
Messages: 100
Registered: September 2008

Senior Member

Dear sir,

Yesterday I found in alert log file bellows warning.
and In that time some clients inform us they are cannot log on
to the application. after few times they could log on to
database automatically.

It is dedicated environment .

In parameter file
----------------------
processes = 4000

Content of alert log file.

Mon Dec  6 16:03:00 2010
Thread 1 advanced to log sequence 67690 (LGWR switch)
  Current log# 3 seq# 67690 mem# 0: /d01/oracle/oradata/stlbas/redo03.log
Mon Dec  6 16:20:39 2010
Process J000 died, see its trace file
Mon Dec  6 16:20:39 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:20:39 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Process J000 died, see its trace file
Mon Dec  6 16:20:45 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:20:45 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Mon Dec  6 16:21:01 2010
Process J000 died, see its trace file
Mon Dec  6 16:21:01 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:21:01 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Mon Dec  6 16:21:17 2010
Process J000 died, see its trace file
Mon Dec  6 16:21:17 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:21:17 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Mon Dec  6 16:36:24 2010
Process J000 died, see its trace file
Mon Dec  6 16:36:24 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:36:24 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Process J000 died, see its trace file
Mon Dec  6 16:36:30 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:36:30 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Mon Dec  6 16:36:41 2010
Process J000 died, see its trace file
Mon Dec  6 16:36:41 2010
kkjcre1p: unable to spawn jobq slave process 
Mon Dec  6 16:36:41 2010
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc:

Mon Dec  6 16:38:18 2010
Thread 1 advanced to log sequence 67691 (LGWR switch)
  Current log# 4 seq# 67691 mem# 0: /d01/oracle/oradata/stlbas/redo04.log

contents in the trace file are -

/d04/admin/stlbas/bdump/stlbas_cjq0_1885.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining Scoring Engine
and Real Application Testing options
ORACLE_HOME = /d04/oracle/ora102
System name:	SunOS
Node name:	badb1
Release:	5.10
Version:	Generic_142900-07
Machine:	sun4v
Instance name: stlbas
Redo thread mounted by this instance: 1
Oracle process number: 26
Unix process pid: 1885, image: oracle@badb1 (CJQ0)

*** SERVICE NAME:(SYS$BACKGROUND) 2010-12-01 13:24:30.248
*** SESSION ID:(2191.1) 2010-12-01 13:24:30.248
*** 2010-12-01 13:24:30.248
Process J000 is dead (pid=25006, state=3):
*** 2010-12-01 13:24:37.277
Process J000 is dead (pid=25012, state=3):
*** 2010-12-01 13:59:56.397
Process J000 is dead (pid=1185, state=3):
*** 2010-12-01 15:55:42.277
Process J000 is dead (pid=15686, state=3):
*** 2010-12-06 16:20:39.526
Process J000 is dead (pid=9458, state=3):
*** 2010-12-06 16:20:45.565
Process J000 is dead (pid=9480, state=3):
*** 2010-12-06 16:21:01.641
Process J000 is dead (pid=9518, state=3):
*** 2010-12-06 16:21:17.712
Process J000 is dead (pid=9550, state=3):
*** 2010-12-06 16:36:24.213
Process J000 is dead (pid=11942, state=3):
*** 2010-12-06 16:36:30.238
Process J000 is dead (pid=11970, state=3):
*** 2010-12-06 16:36:41.289
Process J000 is dead (pid=11998, state=3):

content of listener log file like bellows (in that time only)

06-DEC-2010 16:16:21 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=001zohur))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1771)) * establish * STLBAS * 0
06-DEC-2010 16:16:21 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\RWRBE60.exe)(HOST=APPLICATION-02)(USER=154shahadat))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.32)(PORT=1528)) * establish * STLBAS * 0
06-DEC-2010 16:16:21 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\RWRBE60.exe)(HOST=APPLICATION-07)(USER=043sathekur))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1758)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:21 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=038alfee))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1772)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:22 * service_update * stlbas * 0
06-DEC-2010 16:16:24 * service_update * stlbas * 0
06-DEC-2010 16:16:25 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=003aftab))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1794)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:25 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\RWRBE60.exe)(HOST=NEW-SUN-APP)(USER=023sohel))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.37)(PORT=4291)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:27 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=038alfee))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1796)) * establish * STLBAS * 0
06-DEC-2010 16:16:27 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=038alfee))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1797)) * establish * STLBAS * 0
06-DEC-2010 16:16:28 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\RWRBE60.exe)(HOST=APPLICATION-11)(USER=030salah))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.33)(PORT=3982)) * establish * STLBAS * 0
06-DEC-2010 16:16:29 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=stlbas)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=REPORTS_APP)(USER=504refayet))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.233)(PORT=4995)) * establish * stlbas * 0
06-DEC-2010 16:16:29 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=019aporna))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1800)) * establish * STLBAS * 0
06-DEC-2010 16:16:29 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=stlbas)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=REPORTS_APP)(USER=504refayet))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.233)(PORT=4996)) * establish * stlbas * 0
06-DEC-2010 16:16:30 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=003aftab))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1801)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:30 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=STLBAS)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=APPLICATION-07)(USER=019aporna))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.36)(PORT=1802)) * establish * STLBAS * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe
06-DEC-2010 16:16:30 * service_update * stlbas * 0
06-DEC-2010 16:16:30 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=stlbas)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=REPORTS_APP)(USER=501azad))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.233)(PORT=4997)) * establish * stlbas * 0
06-DEC-2010 16:16:31 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=stlbas)(CID=(PROGRAM=D:\OraNT\BIN\ifrun60.EXE)(HOST=REPORTS_APP)(USER=501azad))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.1.233)(PORT=4999)) * establish * stlbas * 0
06-DEC-2010 16:16:32 * service_update * stlbas * 0

What is the cause ?
and
why it shown "Solaris Error: 32: Broken pipe" ? please

Best Regards
Halim

[Updated on: Tue, 07 December 2010 03:59]

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485296 is a reply to message #485272]

Tue, 07 December 2010 05:39

halim
Messages: 100
Registered: September 2008

Senior Member

Thanks

I got my solution.

regards
Halim

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485304 is a reply to message #485296]

Tue, 07 December 2010 06:18

cookiemonster
Messages: 13920
Registered: September 2008
Location: Rainy Manchester

Senior Member

It might be useful for others who encounter the same problem if you share the solution.

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485450 is a reply to message #485304]

Wed, 08 December 2010 02:34

halim
Messages: 100
Registered: September 2008

Senior Member

My Answer is here:-

http://forums.oracle.com/forums/thread.jspa?threadID=2141050

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485492 is a reply to message #485450]

Wed, 08 December 2010 09:52

Michel Cadot
Messages: 68647
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

In this thread you said you got the solution you don't say which one it is among what has been said.
Please explain the solution that was appropriate in your case.

Regards
Michel

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485899 is a reply to message #485492]

Sat, 11 December 2010 01:22

halim
Messages: 100
Registered: September 2008

Senior Member

ok Michel

IT reached max utilization of process parameter.

Found this information by this query,

SQL>
SQL> select * from v$resource_limit ;
 
RESOURCE_NAME                  CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_AL LIMIT_VALU
------------------------------ ------------------- --------------- ---------- ----------
processes                                     1607            4000       4000       4000
sessions                                      1596            4005       4405       4405
enqueue_locks                                 1380            6280      57390      57390
enqueue_resources                              563            2901      19600  UNLIMITED
ges_procs                                        0               0          0          0
ges_ress                                         0               0          0  UNLIMITED
ges_locks                                        0               0          0  UNLIMITED
ges_cache_ress                                   0               0          0  UNLIMITED
ges_reg_msgs                                     0               0          0  UNLIMITED
ges_big_msgs                                     0               0          0  UNLIMITED
ges_rsv_msgs                                     0               0          0          0
gcs_resources                                    0               0          0          0
gcs_shadows                                      0               0          0          0
dml_locks                                       43             718      19380  UNLIMITED
temporary_table_locks                            0               3  UNLIMITED  UNLIMITED
transactions                                   306             722       4845  UNLIMITED
branches                                         1              13       4845  UNLIMITED
cmtcallbk                                        2               4       4845  UNLIMITED
sort_segment_locks                            1594            4451  UNLIMITED  UNLIMITED
max_rollback_segments                          136             361       4845      65535
max_shared_servers                               0               0  UNLIMITED  UNLIMITED
parallel_max_servers                          1324            3600       2560       3600
 
22 rows selected.

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #485901 is a reply to message #485899]

Sat, 11 December 2010 01:34

Michel Cadot
Messages: 68647
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

Thanks
Michel

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #488012 is a reply to message #485901]

Tue, 04 January 2011 03:48

vnkumaresh
Messages: 9
Registered: December 2010

Junior Member

when you hit process value to its max value, you would have received max-process error(ORA-0020). Correct me If i am wrong.

Thanks

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #488020 is a reply to message #488012]

Tue, 04 January 2011 04:06

Michel Cadot
Messages: 68647
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

Yes if it happens during a session creation but not if it happens during the generation of a slave process for a job as it happened here.

Regards
Michel

[Updated on: Sat, 02 July 2011 10:50]

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #514169 is a reply to message #488020]

Sat, 02 July 2011 01:06

halim
Messages: 100
Registered: September 2008

Senior Member

Hi Michel sir,

I have a more question about this.

How can i control this generation of a slave process ?
Here sometimes many session create more then 500 salve processes.
this is not only related to job here.
so is there any parameter or way to control it ?

Thanks
Halim

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #514201 is a reply to message #514169]

Sat, 02 July 2011 10:51

Michel Cadot
Messages: 68647
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

Set parallel_max_servers parameter.

Regards
Michel

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #514252 is a reply to message #514201]

Sun, 03 July 2011 00:32

halim
Messages: 100
Registered: September 2008

Senior Member

Thanks sir

as document says parallel_max_servers value will be following calculation .

parallel_max_servers = CPU_COUNT x PARALLEL_THREADS_PER_CPU x (2 if
PGA_AGGREGATE_TARGET > 0; otherwise 1) x 5

in our system:-
----------------
parallel_max_servers = 32*2*2*5 = 640

so if we reduce the value like 100, is there any performance impact on the system ?

Regards
Halim

Report message to a moderator

Re: kkjcre1p: unable to spawn jobq slave process [message #514256 is a reply to message #514252]

Sun, 03 July 2011 01:55

Michel Cadot
Messages: 68647
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

Yes or no it depends on so many things that no one can say without more information and experience
on your system and workload:
- you have less active processes so less memory used and less concurrency on the CPU
- you have less parallelism so potentially there may be time where CPU that can be used is not
used but this depends on your IO subsystem usage.

Anyway 100 is, I think, a sufficient value.
Check your io subsystem is or not overloaded.
Check the IO and CPU queue lengths.

Regards
Michel

Report message to a moderator

Previous Topic:	Starting up Database
Next Topic:	DB_LINKS uses user account

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sun Apr 28 19:03:47 CDT 2024