Deadlock in java or oracle with full skacktrace included, please help.

From: (wrong string) ølberg <newsbep_at_kukkuk.dk>
Date: Mon, 25 Mar 2002 16:31:43 +0100
Message-ID: <3c9f42e1$0$87765$edfadb0f_at_dspool01.news.tele.dk>


Hi all.

Here is a problem that has been nagging me for ½ a year by now.

Our system, written in Java, hangs but with no SQLException or anything else.
(you can see in the trace below which threads that lockup)

I think it might be a deadlock in oracle, but then again it might just be my own fault and probably is.

We are running IBM aix and java 1.2.2 although i don't think it has anything to do with aix or java version. What happends just before or at the same time when
some of the threads locks up is that say only 5 out of 10 INSERTS is commited. I
do understand the commit system but i have NEVER heard about a partially commit.

Please please please anyone help me ;-)

In case you don't want waped lines at pos 75 please check links:

Online versions of the full thread stacktrace can also be found at : http://www.kukkuk.dk/remote/OCHDeadlock.html

Sriped version with only active threads can be found at : http://www.kukkuk.dk/remote/OCHDeadlockStriped.html

-----Thread dump created with kill -3 on -----

Thu Mar 21 16:02:39 2002

SIGQUIT received at d01eee00 in /usr/lib/libc.a. J2RE 1.2.2 IBM build ca122-20010313
/usr/java_dev2/jre/sh/../bin/java com.company.och2.OCHMain ../etc/och.cfg

System Properties



Java Home Dir:/usr/java_dev2/jre/sh/..
Java DLL Dir:/usr/java_dev2/jre/sh/../bin Sys
Classpath:/usr/java_dev2/jre/sh/../lib/rt.jar:/usr/java_dev2/jre/sh/../lib/i 18n.jar:/usr/java_dev2/jre/sh/../classes User Args:
-Djava.class.path=/usr/company/ms/msapi/v111/lib/ms.jar:/usr/company/lsmag/l ogapi/v112/lib/log.jar:/usr/company/sumo/srv/v108/lib/sumosrv.jar:/usr/compa ny/util/layer/v102/lib/layer.jar:/usr/company/util/dnconf/v101/lib/dnconf.ja r:/usr/company/util/java/v105/lib/java.jar:/usr/company/api/jdbc/oracle/v101 /lib/classes111.zip:/usr/company/api/regexp/v100/lib/gnu-regexp-1.0.8.jar:/u sr/local/vbroker/java/lib/vbjapp.jar:/usr/local/vbroker/java/lib/vbjorb.jar: ../hotfix:../lib/och2srv.jar

Current Thread Details


"OCH2MSWorker-5" (TID:0x80033028, sys_thread_t:0x30820088, state:CW, native
ID:0x1517) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30cf9d30)
  • Monitors Held -----

Operating Environment



Host: xxxxxx.xxxxxx.xxx:xxx.xxx.xxx.xxx OS Level: AIX 4.3.3.0
Processors -
Architecture: POWER_PC (impl: POWER_604, ver: PV_604) How Many: 4
Enabled: 4

User Limits (in bytes except for NOFILE and NPROC) -

RLIMIT_FSIZE:  infinity
RLIMIT_DATA: 1342177280
RLIMIT_STACK:  33554432
RLIMIT_CORE: 1073741312
RLIMIT_NOFILE:  2000

NPROC(max): 2048

Page Space (in blocks) -
/dev/hd6: size=524288, free=282014

Application Environment



Signal Handlers
SIGINT: osnsuiint (libmsapi.so)
SIGQUIT: intrDispatchMD (libhpi.a)
SIGILL: intrDispatchMD (libhpi.a)
SIGTRAP: unknown handler (libjitc.a)
SIGABRT: intrDispatchMD (libhpi.a)
SIGEMT: intrDispatchMD  (libhpi.a)
SIGFPE: intrDispatchMD  (libhpi.a)
SIGBUS: intrDispatchMD  (libhpi.a)

SIGSEGV: intrDispatchMD (libhpi.a)
SIGSYS: intrDispatchMD (libhpi.a)
SIGPIPE: ignored
SIGUSR2: unknown handler (libjitc.a)
Environment Variables
_=/usr/java_dev2/jre/sh/../bin/java
MANPATH=/usr/man:/usr/local/man
LANG=en_US
LOGIN=och2
IMQCONFIGCL=/etc/IMNSearch/dbcshelp
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/en_US/%N:/usr/lib/nls/msg/%L
/%N.cat:/usr/lib/nls/msg/en_US/%N.cat

PATH=/usr/java_dev2/jre/sh:/usr/java_dev2/sh:/usr/bin:/etc:/usr/sbin:/usr/uc b:/usr/company/och2/bin:/usr/bin/X11:/sbin:/usr/local/bin:/bin:/usr/company/ och2/bin:.
LC__FASTMSG=true
CGI_DIRECTORY=/var/docsearch/cgi-bin
IMQCONFIGSRV=/etc/IMNSearch
CLASSPATH=/usr/company/ms/msapi/v111/lib/ms.jar:/usr/company/lsmag/logapi/v1 12/lib/log.jar:/usr/company/sumo/srv/v108/lib/sumosrv.jar:/usr/company/util/ layer/v102/lib/layer.jar:/usr/company/util/dnconf/v101/lib/dnconf.jar:/usr/c ompany/util/java/v105/lib/java.jar:/usr/company/api/jdbc/oracle/v101/lib/cla sses111.zip:/usr/company/api/regexp/v100/lib/gnu-regexp-1.0.8.jar:/usr/local /vbroker/java/lib/vbjapp.jar:/usr/local/vbroker/java/lib/vbjorb.jar:../hotfi x:../lib/och2srv.jar
LOGNAME=och2
LOCPATH=/usr/lib/nls/loc
PS1=somehost:och2:${PWD}$
MS_ENV=p
USER=och2
DOCUMENT_SERVER_MACHINE_NAME=localhost
AUTHSTATE=files
CVSROOT=/usr/company/cvsroot
SHELL=/bin/ksh
ODMDIR=/etc/objrepos
JAVAHOME=/usr/java_dev2/jre/sh/..
DOCUMENT_SERVER_PORT=49213
XFILESEARCHPATH=/usr/java_dev2/jre/sh/../lib/locale/%L/%T/%N%S: HOME=/usr/company/och2
TERM=vt220
ORACLE_HOME=/export/syslink/oracle_home
K5MUTE=1
PWD=/usr/company/och2/srv/current/bin
DOCUMENT_DIRECTORY=/usr/docsearch/html
TZ=NFT-1DFT,M3.5.0/02:00:00,M10.5.0/03:00:00 AIXTHREAD_SCOPE=S
LIBPATH=/usr/java_dev2/jre/sh/../bin:/usr/java_dev2/jre/sh/../bin/classic::/ usr/company/ms/msapi/v111/lib
IBM_JAVA_COMMAND_LINE=/usr/java_dev2/jre/sh/../bin/java com.company.och2.OCHMain ../etc/och.cfg

Loaded Libraries (sizes in bytes)



/usr/java_dev2/jre/bin/libmath.a
filesize: 65693
text start: d41a4000
text size: aeb3
data start: 31b9afa8
data size: 3ac
/export/projects/ms/msapi/v111/lib/libmsapi.so filesize: 2767656
text start: d3fc3000
text size: 1e0339
data start: 306e5478
data size: 14894
/usr/java_dev2/jre/bin/libnet.a
filesize: 51753
text start: d3dbd000
text size: 7b8f
data start: 306043a8
data size: 338
/usr/java_dev2/jre/bin/libjitc.a
filesize: 2219252
text start: d35f4000
text size: 1bb9cc
data start: 30276288
data size: cf1c
/usr/lib/libiconv.a
filesize: 377832
text start: d0034100
text size: 13f3a
data start: f0131da0
data size: a574
/usr/lib/libi18n.a
filesize: 122343
text start: d002c100
text size: 784c
data start: 30234458
data size: 1114
/usr/lib/nls/loc/en_US
filesize: 14340
text start: d0026000
text size: 2d71
data start: d0029130
data size: 2324
/usr/java_dev2/jre/sh/../bin/libzip.a
filesize: 102114
text start: d35e1000
text size: 127e4
data start: 300812f8
data size: 1e98
/usr/java_dev2/jre/sh/../bin/libhpi.a
filesize: 83204
text start: d35d6000
text size: a3e6
data start: 3003fcf0
data size: b5c
/usr/java_dev2/jre/sh/../bin/libxhpi.a
filesize: 8871
text start: d35d5000
text size: f29
data start: 3003e948
data size: 1a8
/usr/java_dev2/jre/sh/../bin/libjava.a
filesize: 367162
text start: d359a000
text size: 3a20c
data start: 30037738
data size: 6428
/usr/java_dev2/jre/sh/../bin/classic/libjvm.a filesize: 1301153
text start: d34bb000
text size: defa4
data start: 30011338
data size: 25318
/usr/lib/libdl.a
filesize: 5248
text start: d015b0f8
text size: 1e0
data start: f02010f8
data size: 0
/usr/lib/libbsd.a
filesize: 62186
text start: d00d34e0
text size: 7deb
data start: f0108b28
data size: 1460
/usr/ibmcxx/lib/libC.a
filesize: 2610937
text start: d0b23360
text size: 5982
data start: f021c760
data size: a08
/usr/ibmcxx/lib/libC.a
filesize: 2610937
text start: d0aa3660
text size: 5fca
data start: f021b460
data size: 83c
/usr/ibmcxx/lib/libC.a
filesize: 2610937
text start: d0ab67c0
text size: 6c174
data start: f0208dc0
data size: 113c4
/usr/ibmcxx/lib/libC.a
filesize: 2610937
text start: d0aaa070
text size: b756
data start: f020731c
data size: 1c
/usr/lib/libpthreads.a
filesize: 712882
text start: d0001000
text size: 2290
data start: f0080000
data size: 41004
/usr/lib/libpthreads.a
filesize: 712882
text start: d0004000
text size: 2020a
data start: f00c2000
data size: 400c
/usr/ibmcxx/lib/libC.a
filesize: 2610937
text start: d0a43100
text size: 33653
data start: f0202d00
data size: 411c
/usr/lib/libcrypt.a
filesize: 11167
text start: d00250f8
text size: 87a
data start: f007f528
data size: 13c
/usr/lib/libc.a
filesize: 6351516
text start: d015f720
text size: 1becf6
data start: f00002c0
data size: 7ea00

"ActionLayer" (TID:0x80032be8, sys_thread_t:0x3087df78, state:CW, native
ID:0x2628) prio=5

at java.lang.Object.wait(Native Method)
at  com.company.util.layer.Layer.hibernate(Layer.java(Compiled Code))
at  com.company.och2.layer.OCHLayer.run(OCHLayer.java:131)
at java.lang.Thread.run(Thread.java:481)
  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x31322d08)
  • Monitors Held -----

"SessionLayerDown" (TID:0x80032c28, sys_thread_t:0x3087dd18, state:CW,
native ID:0x2527) prio=5

at java.lang.Object.wait(Native Method)
at  com.company.util.layer.Layer.hibernate(Layer.java(Compiled Code))
at  com.company.och2.layer.OCHLayer.run(OCHLayer.java:131)
at java.lang.Thread.run(Thread.java:481)
  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x312dcb08)
  • Monitors Held -----

"SessionLayerUp" (TID:0x80032c68, sys_thread_t:0x3087dab8, state:MW, native
ID:0x2426) prio=5
at
oracle.jdbc.driver.OracleConnection.needLine(OracleConnection.java(Compiled Code))
at
oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java(Compi led Code))
at
oracle.jdbc.driver.OracleStatement.doExecute(OracleStatement.java(Compiled Code))
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java (Compiled Code))
at
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatem ent.java(Compiled Code))
at
com.company.och2.store.RuntimeStore.getTransactionParameters(RuntimeStore.ja va(Compiled Code))
at
com.company.och2.layer.session.SessionLayer.handleValidatedTransaction(Sessi onLayer.java(Compiled Code))
at
com.company.och2.layer.session.SessionLayer.handleTransaction(SessionLayer.j ava(Compiled Code))
at
com.company.och2.layer.session.SessionLayer.doLayerWork(SessionLayer.java(Co mpiled Code))
at com.company.och2.layer.OCHLayer.run(OCHLayer.java:137) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd35d9568 in try_suspend at 0xd0005d18 in pthread_mutex_lock at 0xd35da6f8 in cooperative_mutex_lock at 0xd35da530 in sysMonitorEnter at 0xd354ee04 in lkMonitorEnter unable to backtrace through native code - iar 0x30272eb4 not in text area (sp is 0x312954e8)
  • Monitors Held ----- (flat) oracle.jdbc.driver.OraclePreparedStatement_at_80982990/80982998
  • Blocked On ----- oracle.jdbc.driver.OracleConnection_at_80137030/80137038

"TransactionLayer" (TID:0x80032ca8, sys_thread_t:0x3087d858, state:CW,
native ID:0x2325) prio=5

at java.lang.Object.wait(Native Method)
at  com.company.util.layer.Layer.hibernate(Layer.java(Compiled Code))
at  com.company.och2.layer.OCHLayer.run(OCHLayer.java:131)
at java.lang.Thread.run(Thread.java:481)
  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x31250708)
  • Monitors Held -----

"UDPWorker" (TID:0x80032ce8, sys_thread_t:0x30874288, state:R, native
ID:0x2224) prio=5
at java.net.PlainDatagramSocketImpl.receive(Native Method) at
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java(Compil ed Code))
at java.net.DatagramSocket.receive(DatagramSocket.java(Compiled Code)) at
com.company.och2.layer.transaction.UDPWorker.doWork(UDPWorker.java(Compiled Code))
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:176) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd35dbff0 in sysRecvFrom at 0xd34d4e48 in JVM_RecvFrom at 0xd3dbed50 in Java_java_net_PlainDatagramSocketImpl_receive unable to backtrace through native code - iar 0x3111ff18 not in text area (sp is 0x3120a42c)
  • Monitors Held ----- (flat) java.net.DatagramPacket_at_80F3C980/80F3C988 (flat) java.net.DatagramSocket_at_80F3C9E8/80F3C9F0

"ValidationWorker-5" (TID:0x80032d28, sys_thread_t:0x30873f08, state:CW,
native ID:0x2123) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x311c4330)
  • Monitors Held -----

"ValidationWorker-4" (TID:0x80032d68, sys_thread_t:0x3085b2f8, state:CW,
native ID:0x2022) prio=5
at
oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement .java(Compiled Code))
at
oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java(Compiled Code))
at
com.company.och2.store.TransactionManager.commitTransaction(TransactionManag er.java(Compiled Code))
at
com.company.och2.layer.transaction.ValidatorWorker.validateTransaction(Valid atorWorker.java(Compiled Code))
at
com.company.och2.layer.transaction.ValidatorWorker.doWork(ValidatorWorker.ja va(Compiled Code))
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:176) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- unable to backtrace through native code - iar 0x31079cd0 not in text area (sp is 0x31079b84)
  • Monitors Held ----- (flat) com.company.och2.store.TransactionManager_at_801FD110/801FD118
  • Blocked On ----- oracle.jdbc.driver.OraclePreparedStatement_at_80982990/80982998

"ValidationWorker-3" (TID:0x80032da8, sys_thread_t:0x3082ab18, state:CW,
native ID:0x1f21) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x31033f30)
  • Monitors Held -----

"ValidationWorker-2" (TID:0x80032de8, sys_thread_t:0x3082a8b8, state:CW,
native ID:0x1e20) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30fedd30)
  • Monitors Held -----

"ValidationWorker-1" (TID:0x80032e28, sys_thread_t:0x30822bb8, state:CW,
native ID:0x1d1f) prio=5
at
oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement .java(Compiled Code))
at
oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java(Compiled Code))
at
com.company.och2.store.TransactionManager.commitTransaction(TransactionManag er.java(Compiled Code))
at
com.company.och2.layer.transaction.ValidatorWorker.validateTransaction(Valid atorWorker.java(Compiled Code))
at
com.company.och2.layer.transaction.ValidatorWorker.doWork(ValidatorWorker.ja va(Compiled Code))
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:176) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd34d50ac in JVM_Recv at 0xd00123ec in _pthread_event_wait at 0xd000e43c in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f47c in pthread_cond_wait at 0xd35d7ab8 in condvarWait at 0xd35da10c in sysMonitorWait at 0xd354eed0 in lkMonitorEnter unable to backtrace through native code - iar 0x30272eb4 not in text area (sp is 0x30fa78ec)
  • Monitors Held ----- (flat) com.company.och2.store.TransactionManager_at_80192858/80192860
  • Blocked On ----- oracle.jdbc.driver.OraclePreparedStatement_at_80A1FCE0/80A1FCE8

"MS2OCHWorker-5" (TID:0x80032e68, sys_thread_t:0x30822958, state:CW, native
ID:0x1c1e) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x30f60830)
  • Monitors Held -----

"MS2OCHWorker-4" (TID:0x80032ea8, sys_thread_t:0x30821c28, state:CW, native
ID:0x1b1d) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x30d86130)
  • Monitors Held -----

"MS2OCHWorker-3" (TID:0x80032ee8, sys_thread_t:0x308219c8, state:CW, native
ID:0x1a1c) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x30f1b730)
  • Monitors Held -----

"MS2OCHWorker-2" (TID:0x80032f28, sys_thread_t:0x30821768, state:CW, native
ID:0x191b) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x30ed5530)
  • Monitors Held -----

"MS2OCHWorker-1" (TID:0x80032f68, sys_thread_t:0x30821168, state:CW, native
ID:0x181a) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x30e8f330)
  • Monitors Held -----

"OnlineRequestWorker" (TID:0x80032fa8, sys_thread_t:0x30820548, state:CW,
native ID:0x1719) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30e08130)
  • Monitors Held -----

"HooverWorker" (TID:0x80032fe8, sys_thread_t:0x308202e8, state:CW, native
ID:0x1618) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30d3ff30)
  • Monitors Held -----

"OCH2MSWorker-5" (TID:0x80033028, sys_thread_t:0x30820088, state:CW, native
ID:0x1517) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30cf9d30)
  • Monitors Held -----

"OCH2MSWorker-4" (TID:0x80033068, sys_thread_t:0x3081fe28, state:CW, native
ID:0x1416) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30cb3b30)
  • Monitors Held -----

"OCH2MSWorker-3" (TID:0x800330a8, sys_thread_t:0x3081fbc8, state:MW, native
ID:0x1315) prio=5
at
oracle.jdbc.driver.OracleConnection.needLine(OracleConnection.java(Compiled Code))
at
oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java(Compi led Code))
at
oracle.jdbc.driver.OracleStatement.doExecuteWithBatch(OracleStatement.java(C ompiled Code))
at
oracle.jdbc.driver.OracleStatement.doExecute(OracleStatement.java(Compiled Code))
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java (Compiled Code))
at
oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedState ment.java(Compiled Code))
at
com.company.och2.store.RuntimeStore.setTlState(RuntimeStore.java(Compiled Code))
at
com.company.och2.layer.transaction.OCH2MSWorker.sendTransaction(OCH2MSWorker .java(Compiled Code))
at
com.company.och2.layer.transaction.OCH2MSWorker.doWork(OCH2MSWorker.java(Com piled Code))
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:176) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- unable to backtrace through native code - iar 0xf00c4d6c not in text area (sp is 0x30c6c2e0)
  • Monitors Held ----- (flat) oracle.jdbc.driver.OraclePreparedStatement_at_80A1FCE0/80A1FCE8
  • Blocked On ----- oracle.jdbc.driver.OracleConnection_at_8015FA90/8015FA98

"OCH2MSWorker-2" (TID:0x800330e8, sys_thread_t:0x3081bfa8, state:CW, native
ID:0x1214) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30c27730)
  • Monitors Held -----

"OCH2MSWorker-1" (TID:0x80033128, sys_thread_t:0x3081bd28, state:CW, native
ID:0x1113) prio=5
at java.lang.Object.wait(Native Method)
at com.company.och2.misc.OCHWorker.run(OCHWorker.java:180) at java.lang.Thread.run(Thread.java:481)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30be1530)
  • Monitors Held -----

"ActionThread-10" (TID:0x80033168, sys_thread_t:0x307ee4a8, state:CW,
native ID:0x1012) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x1b not in text area (sp is 0x30b9afac)
  • Monitors Held -----

"ActionThread-9" (TID:0x800331d0, sys_thread_t:0x307eebe8, state:CW, native
ID:0xf11) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x30b55708 not in text area (sp is 0x30b54dac)
  • Monitors Held -----

"ActionThread-8" (TID:0x80033238, sys_thread_t:0x307e8278, state:CW, native
ID:0xe10) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x3001c368 not in text area (sp is 0x30b0ebac)
  • Monitors Held -----

"ActionThread-7" (TID:0x800332a0, sys_thread_t:0x307e0808, state:CW, native
ID:0xd0f) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x816 not in text area (sp is 0x30ac89ac)
  • Monitors Held -----

"ActionThread-6" (TID:0x80033308, sys_thread_t:0x307e05a8, state:CW, native
ID:0xc0e) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x816 not in text area (sp is 0x30a827ac)
  • Monitors Held -----

"ActionThread-5" (TID:0x80033370, sys_thread_t:0x307dfea8, state:CW, native
ID:0xb0d) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x3001c368 not in text area (sp is 0x30a3b4ac)
  • Monitors Held -----

"ActionThread-4" (TID:0x800333d8, sys_thread_t:0x307dfc48, state:CW, native
ID:0xa0c) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x3001c368 not in text area (sp is 0x309f63ac)
  • Monitors Held -----

"ActionThread-3" (TID:0x80033440, sys_thread_t:0x307df9e8, state:CW, native
ID:0x90b) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x801a0c34 not in text area (sp is 0x309b01ac)
  • Monitors Held -----

"ActionThread-2" (TID:0x800334a8, sys_thread_t:0x307df788, state:CW, native
ID:0x80a) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x816 not in text area (sp is 0x30969fac)
  • Monitors Held -----

"ActionThread-1" (TID:0x80033510, sys_thread_t:0x307dd118, state:CW, native
ID:0x709) prio=5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at
com.company.och2.layer.action.ActionThread.waitForWork(ActionThread.java(Com piled Code))
at
com.company.och2.layer.action.ActionThread.notifyDoneWorking(ActionThread.ja va(Compiled Code))
at com.company.och2.layer.action.ActionThread.run(ActionThread.java:118)

  • Native Stack ----- unable to backtrace through native code - iar 0x816 not in text area (sp is 0x30923dac)
  • Monitors Held -----

"GarbageCollector" (TID:0x80033578, sys_thread_t:0x30669b08, state:CW,
native ID:0x608) prio=5
at java.lang.Object.wait(Native Method)
at com.visigenic.vbroker.orb.GarbageCollector.run(GarbageCollector.java:64)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x3064c978 not in text area (sp is 0x30785c6c)
  • Monitors Held -----

"AreYouAlive" (TID:0x80033608, sys_thread_t:0x305d9b08, state:CW, native
ID:0x506) prio=5
at java.lang.Object.wait(Native Method)
at com.visigenic.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:65)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x306d93bc)
  • Monitors Held -----

"Thread-0" (TID:0x80033650, sys_thread_t:0x305bf7f8, state:R, native
ID:0x405) prio=5
at java.net.PlainDatagramSocketImpl.receive(Native Method) at
java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java(Compil ed Code))
at java.net.DatagramSocket.receive(DatagramSocket.java(Compiled Code)) at com.visigenic.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:104)

  • Native Stack ----- at 0xd35dbff0 in sysRecvFrom at 0xd34d4e48 in JVM_RecvFrom at 0xd3dbed50 in Java_java_net_PlainDatagramSocketImpl_receive unable to backtrace through native code - iar 0x3111ff18 not in text area (sp is 0x3064a9d0)
  • Monitors Held ----- (flat) java.net.PlainDatagramSocketImpl_at_801F8A78/801F8A80 (flat) java.net.DatagramSocket_at_801F8A90/801F8A98 (flat) java.net.DatagramPacket_at_807CA370/807CA378

"Finalizer" (TID:0x80033728, sys_thread_t:0x300e3368, state:CW, native
ID:0x304) prio=8
at java.lang.Object.wait(Native Method)

at  java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java(Compiled Code))
at  java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java(Compiled Code))
at  java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:176)

  • Native Stack ----- at 0xd00123ec in _pthread_event_wait at 0xd000e43c in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f47c in pthread_cond_wait at 0xd35d7ab8 in condvarWait at 0xd35da10c in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x302267ec)
  • Monitors Held -----

"Reference Handler" (TID:0x80033768, sys_thread_t:0x300d95f8, state:CW,
native ID:0x203) prio=10
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java(Compiled Code)) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

  • Native Stack ----- at 0xd0012010 in _pthread_ksleep at 0xd00123ec in _pthread_event_wait at 0xd000e43c in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f47c in pthread_cond_wait at 0xd35d7ab8 in condvarWait at 0xd35da10c in sysMonitorWait at 0xd354e7f8 in lkMonitorWait at 0xd34df108 in JVM_MonitorWait unable to backtrace through native code - iar 0x0 not in text area (sp is 0x301e172c)
  • Monitors Held -----

"Signal dispatcher" (TID:0x800337a8, sys_thread_t:0x300d8c78, state:CW,
native ID:0x102) prio=5

  • Native Stack ----- at 0xd00123ec in _pthread_event_wait at 0xd000e43c in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f47c in pthread_cond_wait at 0xd35d7ab8 in condvarWait at 0xd35dd778 in sigMonitorWait at 0xd35ddb0c in sysSignalWait at 0xd353eb34 in sigthread at 0xd353c7d4 in xmSaveStackBase at 0xd35d7c68 in _start at 0xd00081fc in _pthread_body unable to backtrace through native code - iar 0x0 not in text area (sp is 0x3019bab0)
  • Monitors Held -----

"OCHMain" (TID:0x800337e8, sys_thread_t:0x3000b248, state:CW, native
ID:0x1) prio=5

at java.lang.Thread.sleep(Native Method)
at  com.company.och2.OCHMain.ochMainLoop(OCHMain.java:407)
at  com.company.och2.OCHMain.main(OCHMain.java:611)

  • Native Stack ----- at 0xd01eee00 in gettimeofday at 0xd00123ec in _pthread_event_wait at 0xd000e310 in _cond_wait_local at 0xd000e814 in _cond_wait at 0xd000f1e0 in pthread_cond_timedwait at 0xd35d7978 in condvarTimedWait at 0xd35da128 in sysMonitorWait at 0xd353d1c0 in xmThreadSleep at 0xd34dcd84 in JVM_Sleep unable to backtrace through native code - iar 0x313cf930 not in text area (sp is 0x2ff2222c)
  • Monitors Held -----

Monitor pool info:
Initial monitor count: 32
Minimum number of free monitors before expansion: 5 Pool will next be expanded by: 121
Current total number of monitors: 243
Current number of free monitors: 76

Monitor Pool Dump (inflated object-monitors): sys_mon_t:0x30009868 infl_mon_t: 0x30009118: java.lang.ref.Reference$Lock_at_8003B750/8003B758: <unowned> Waiting to be notified:
"Reference Handler" (0x300d95f8)

sys_mon_t:0x300098e8 infl_mon_t: 0x30009134: java.lang.ref.ReferenceQueue$Lock_at_8003B3A0/8003B3A8: <unowned> Waiting to be notified:
"Finalizer" (0x300e3368)

sys_mon_t:0x300099e8 infl_mon_t: 0x3000916c: com.visigenic.vbroker.ds.DSUser$AreYouAlive_at_80033608/80033610: <unowned> Waiting to be notified:
"AreYouAlive" (0x305d9b08)

sys_mon_t:0x30009c68 infl_mon_t: 0x300091f8: com.visigenic.vbroker.orb.GarbageCollector_at_80033578/80033580: <unowned> Waiting to be notified:
"GarbageCollector" (0x30669b08)

sys_mon_t:0x30009ee8 infl_mon_t: 0x30009284: com.company.och2.layer.action.ActionThread_at_800333D8/800333E0: <unowned> Waiting to be notified:
"ActionThread-4" (0x307dfc48)

sys_mon_t:0x30009f68 infl_mon_t: 0x300092a0: com.company.och2.layer.action.ActionThread_at_80033308/80033310: <unowned> Waiting to be notified:
"ActionThread-6" (0x307e05a8)

sys_mon_t:0x30009fe8 infl_mon_t: 0x300092bc: com.company.och2.layer.action.ActionThread_at_800332A0/800332A8: <unowned> Waiting to be notified:
"ActionThread-7" (0x307e0808)

sys_mon_t:0x3000a068 infl_mon_t: 0x300092d8: com.company.och2.layer.action.ActionThread_at_800334A8/800334B0: <unowned> Waiting to be notified:
"ActionThread-2" (0x307df788)

sys_mon_t:0x3000a0e8 infl_mon_t: 0x300092f4: com.company.och2.layer.action.ActionThread_at_80033168/80033170: <unowned> Waiting to be notified:
"ActionThread-10" (0x307ee4a8)

sys_mon_t:0x3000a168 infl_mon_t: 0x30009310: com.company.och2.layer.action.ActionThread_at_80033238/80033240: <unowned> Waiting to be notified:
"ActionThread-8" (0x307e8278)

sys_mon_t:0x3000a1e8 infl_mon_t: 0x3000932c: com.company.och2.layer.action.ActionThread_at_80033510/80033518: <unowned> Waiting to be notified:
"ActionThread-1" (0x307dd118)

sys_mon_t:0x3000a268 infl_mon_t: 0x30009348: com.company.och2.layer.action.ActionThread_at_80033370/80033378: <unowned> Waiting to be notified:
"ActionThread-5" (0x307dfea8)

sys_mon_t:0x3000a2e8 infl_mon_t: 0x30009364: com.company.och2.layer.action.ActionThread_at_800331D0/800331D8: <unowned> Waiting to be notified:
"ActionThread-9" (0x307eebe8)

sys_mon_t:0x3000a368 infl_mon_t: 0x30009380: com.company.och2.layer.action.ActionThread_at_80033440/80033448: <unowned> Waiting to be notified:
"ActionThread-3" (0x307df9e8)

sys_mon_t:0x3000a468 infl_mon_t: 0x300093b8: com.company.och2.layer.transaction.HooverWorker_at_800B4F68/800B4F70: <unowned>
Waiting to be notified:
"HooverWorker" (0x308202e8)

sys_mon_t:0x30864548 infl_mon_t: 0x308642b0: com.company.och2.layer.transaction.OnlineRequestWorker_at_800B4B00/800B4B08: <unowned>
Waiting to be notified:
"OnlineRequestWorker" (0x30820548)

sys_mon_t:0x308645c8 infl_mon_t: 0x308642cc: java.util.Vector_at_80290B60/80290B68: <unowned> Waiting to be notified:
"TransactionLayer" (0x3087d858)

sys_mon_t:0x308c7508 infl_mon_t: 0x308b9208: com.company.och2.layer.transaction.ValidatorWorker_at_80100F10/80100F18: <unowned>
Waiting to be notified:
"ValidationWorker-3" (0x3082ab18)

sys_mon_t:0x308c7588 infl_mon_t: 0x308b9224: com.company.och2.layer.transaction.ValidatorWorker_at_80101E90/80101E98: <unowned>
Waiting to be notified:
"ValidationWorker-2" (0x3082a8b8)

sys_mon_t:0x308c7608 infl_mon_t: 0x308b9240: com.company.och2.layer.transaction.ValidatorWorker_at_8010EF98/8010EFA0: <unowned>
Waiting to be notified:
"ValidationWorker-5" (0x30873f08)

sys_mon_t:0x30497c88 infl_mon_t: 0x00000000: oracle.jdbc.driver.OraclePreparedStatement_at_80A1FCE0/80A1FCE8: <unowned> Waiting to be notified:
"ValidationWorker-1" (0x30822bb8)

sys_mon_t:0x30497ff8 infl_mon_t: 0x304993f0: com.company.och2.layer.transaction.MS2OCHWorker_at_800B0908/800B0910: <unowned>
Waiting to be notified:
"MS2OCHWorker-3" (0x308219c8)

sys_mon_t:0x30498238 infl_mon_t: 0x30499460: com.company.och2.layer.transaction.OCH2MSWorker_at_8018B510/8018B518: <unowned>
Waiting to be notified:
"OCH2MSWorker-5" (0x30820088)

sys_mon_t:0x30498be8 infl_mon_t: 0x30499578: com.company.och2.layer.transaction.MS2OCHWorker_at_800AE1F0/800AE1F8: <unowned>
Waiting to be notified:
"MS2OCHWorker-2" (0x30821768)

sys_mon_t:0x30498c68 infl_mon_t: 0x30499594: com.company.och2.layer.transaction.MS2OCHWorker_at_8009A670/8009A678: <unowned>
Waiting to be notified:
"MS2OCHWorker-1" (0x30821168)

sys_mon_t:0x30498ea8 infl_mon_t: 0x304995b0: com.company.och2.layer.transaction.MS2OCHWorker_at_800B16E0/800B16E8: <unowned>
Waiting to be notified:
"MS2OCHWorker-4" (0x30821c28)

sys_mon_t:0x30498f28 infl_mon_t: 0x304995cc: com.company.och2.layer.transaction.MS2OCHWorker_at_800B3448/800B3450: <unowned>
Waiting to be notified:
"MS2OCHWorker-5" (0x30822958)

sys_mon_t:0x3135f018 infl_mon_t: 0x311363b4: oracle.jdbc.driver.OracleConnection_at_8015FA90/8015FA98: owner
"ValidationWorker-1" (0x30822bb8) 1 entry
Waiting to enter:
"OCH2MSWorker-3" (0x3081fbc8)

sys_mon_t:0x31367dc8 infl_mon_t: 0x311365ac: oracle.jdbc.driver.OracleConnection_at_80137030/80137038: owner
"ValidationWorker-4" (0x3085b2f8) 1 entry
Waiting to enter:
"SessionLayerUp" (0x3087dab8)

sys_mon_t:0x315e3848 infl_mon_t: 0x00000000: oracle.jdbc.driver.OraclePreparedStatement_at_80982990/80982998: <unowned> Waiting to be notified:
"ValidationWorker-4" (0x3085b2f8)

JVM System Monitor Dump (registered monitors): Integer lock access-lock: <unowned>
Sleep lock: <unowned>
Waiting to be notified:
"OCHMain" (0x3000b248)

Method trace lock: <unowned>
UTF8 Cache lock: <unowned>
Heap lock: <unowned>
Rewrite Code lock: <unowned>
Monitor Cache lock: owner "OCH2MSWorker-5" (0x30820088) 1 entry Waiting to enter:
"ActionLayer" (0x3087df78)
"SessionLayerDown" (0x3087dd18)
"OCH2MSWorker-4" (0x3081fe28)
"OCH2MSWorker-2" (0x3081bfa8)
"OCH2MSWorker-1" (0x3081bd28)

JNI Pinning lock: <unowned>
JNI Global Reference lock: <unowned>
Classloader lock: <unowned>
Linking class lock: <unowned>
Binclass lock: <unowned>
Monitor Registry lock: owner "OCH2MSWorker-5" (0x30820088) 1 entry Thread queue lock: owner "OCH2MSWorker-5" (0x30820088) 1 entry

Thread identifiers (as used in flat monitors):

ident 28 "ActionLayer"  (0x3087df78) ee 0x3087deac
ident 27 "SessionLayerDown" (0x3087dd18) ee  0x3087dc4c
ident 26 "SessionLayerUp" (0x3087dab8) ee 0x3087d9ec
ident 25 "TransactionLayer" (0x3087d858) ee 0x3087d78c
ident 24 "UDPWorker"  (0x30874288) ee 0x308741bc
ident 23 "ValidationWorker-5" (0x30873f08) ee  0x30873e3c
ident 22 "ValidationWorker-4" (0x3085b2f8) ee 0x3085b22c
ident  21 "ValidationWorker-3" (0x3082ab18) ee 0x3082aa4c
ident 20 "ValidationWorker-2" (0x3082a8b8) ee 0x3082a7ec
ident 1f "ValidationWorker-1"  (0x30822bb8) ee 0x30822aec
ident 1e "MS2OCHWorker-5" (0x30822958) ee  0x3082288c
ident 1d "MS2OCHWorker-4" (0x30821c28) ee 0x30821b5c
ident 1c "MS2OCHWorker-3" (0x308219c8) ee 0x308218fc
ident 1b "MS2OCHWorker-2"  (0x30821768) ee 0x3082169c
ident 1a "MS2OCHWorker-1" (0x30821168) ee  0x3082109c
ident 19 "OnlineRequestWorker" (0x30820548) ee 0x3082047c
ident 18 "HooverWorker" (0x308202e8) ee 0x3082021c
ident 17 "OCH2MSWorker-5"  (0x30820088) ee 0x3081ffbc
ident 16 "OCH2MSWorker-4" (0x3081fe28) ee  0x3081fd5c
ident 15 "OCH2MSWorker-3" (0x3081fbc8) ee 0x3081fafc
ident 14  "OCH2MSWorker-2" (0x3081bfa8) ee 0x3081bedc
ident 13 "OCH2MSWorker-1"  (0x3081bd28) ee 0x3081bc5c
ident 12 "ActionThread-10" (0x307ee4a8) ee 0x307ee3dc ident 11 "ActionThread-9" (0x307eebe8) ee 0x307eeb1c ident 10 "ActionThread-8" (0x307e8278) ee 0x307e81ac
ident f "ActionThread-7"  (0x307e0808) ee 0x307e073c
ident e "ActionThread-6" (0x307e05a8) ee  0x307e04dc
ident d "ActionThread-5" (0x307dfea8) ee 0x307dfddc
ident c "ActionThread-4" (0x307dfc48) ee 0x307dfb7c
ident b "ActionThread-3"  (0x307df9e8) ee 0x307df91c
ident a "ActionThread-2" (0x307df788) ee  0x307df6bc
ident 9 "ActionThread-1" (0x307dd118) ee 0x307dd04c
ident 8 "GarbageCollector" (0x30669b08) ee 0x30669a3c
ident 7 "AreYouAlive"  (0x305d9b08) ee 0x305d9a3c
ident 6 "Thread-0" (0x305bf7f8) ee  0x305bf72c
ident 5 "Finalizer" (0x300e3368) ee 0x300e329c
ident 4 "Reference Handler" (0x300d95f8) ee 0x300d952c ident 3 "Signal dispatcher" (0x300d8c78) ee 0x300d8bac ident 2 "OCHMain" (0x3000b248) ee 0x3000b17c

Java Object Monitor Dump (flat & inflated object-monitors):

com.company.och2.layer.action.ActionThread_at_80033168/80033170
locknflags  80001200 Monitor inflated infl_mon 0x300092f4
com.company.och2.layer.action.ActionThread_at_800331D0/800331D8
locknflags  80001600 Monitor inflated infl_mon 0x30009364
com.company.och2.layer.action.ActionThread_at_80033238/80033240
locknflags  80001300 Monitor inflated infl_mon 0x30009310
com.company.och2.layer.action.ActionThread_at_800332A0/800332A8
locknflags  80001000 Monitor inflated infl_mon 0x300092bc
com.company.och2.layer.action.ActionThread_at_80033308/80033310
locknflags  80000f00 Monitor inflated infl_mon 0x300092a0
com.company.och2.layer.action.ActionThread_at_80033370/80033378
locknflags  80001500 Monitor inflated infl_mon 0x30009348
com.company.och2.layer.action.ActionThread_at_800333D8/800333E0
locknflags  80000e00 Monitor inflated infl_mon 0x30009284
com.company.och2.layer.action.ActionThread_at_80033440/80033448
locknflags  80001700 Monitor inflated infl_mon 0x30009380
com.company.och2.layer.action.ActionThread_at_800334A8/800334B0
locknflags  80001100 Monitor inflated infl_mon 0x300092d8
com.company.och2.layer.action.ActionThread_at_80033510/80033518
locknflags  80001400 Monitor inflated infl_mon 0x3000932c
com.visigenic.vbroker.orb.GarbageCollector_at_80033578/80033580
locknflags  80000900 Monitor inflated infl_mon 0x300091f8
com.visigenic.vbroker.ds.DSUser$AreYouAlive_at_80033608/80033610 locknflags 80000400 Monitor inflated infl_mon 0x3000916c java.lang.ref.ReferenceQueue$Lock_at_8003B3A0/8003B3A8 locknflags 80000200 Monitor inflated infl_mon 0x30009134 java.lang.ref.Reference$Lock_at_8003B750/8003B758 locknflags 80000100 Monitor inflated infl_mon 0x30009118 com.company.och2.layer.transaction.MS2OCHWorker_at_8009A670/8009A678 locknflags 80005e00 Monitor inflated infl_mon 0x30499594 com.company.och2.layer.transaction.MS2OCHWorker_at_800AE1F0/800AE1F8 locknflags 80005d00 Monitor inflated infl_mon 0x30499578 com.company.och2.layer.transaction.MS2OCHWorker_at_800B0908/800B0910 locknflags 80004f00 Monitor inflated infl_mon 0x304993f0 com.company.och2.layer.transaction.MS2OCHWorker_at_800B16E0/800B16E8 locknflags 80005f00 Monitor inflated infl_mon 0x304995b0 com.company.och2.layer.transaction.MS2OCHWorker_at_800B3448/800B3450 locknflags 80006000 Monitor inflated infl_mon 0x304995cc com.company.och2.layer.transaction.OnlineRequestWorker_at_800B4B00/800B4B08 locknflags 80002300 Monitor inflated infl_mon 0x308642b0 com.company.och2.layer.transaction.HooverWorker_at_800B4F68/800B4F70 locknflags 80001900 Monitor inflated infl_mon 0x300093b8 com.company.och2.layer.transaction.ValidatorWorker_at_80100F10/80100F18 locknflags 80003900 Monitor inflated infl_mon 0x308b9208 com.company.och2.layer.transaction.ValidatorWorker_at_80101E90/80101E98 locknflags 80003a00 Monitor inflated infl_mon 0x308b9224 com.company.och2.layer.transaction.ValidatorWorker_at_8010EF98/8010EFA0 locknflags 80003b00 Monitor inflated infl_mon 0x308b9240 oracle.jdbc.driver.OracleConnection_at_80137030/80137038 locknflags 80009c00 Monitor inflated infl_mon 0x311365ac oracle.jdbc.driver.OracleConnection_at_8015FA90/8015FA98 locknflags 80008a00 Monitor inflated infl_mon 0x311363b4 com.company.och2.layer.transaction.OCH2MSWorker_at_8018B510/8018B518 locknflags 80005300 Monitor inflated infl_mon 0x30499460 com.company.och2.layer.transaction.OCH2MSWorker_at_8018B740/8018B748 locknflags 00160000 Flat locked by threadIdent 16. Entrycount 1 com.company.och2.store.TransactionManager_at_80192858/80192860 locknflags 001f0000 Flat locked by threadIdent 1f. Entrycount 1 java.net.PlainDatagramSocketImpl_at_801F8A78/801F8A80 locknflags 00060000 Flat locked by threadIdent 6. Entrycount 1 java.net.DatagramSocket_at_801F8A90/801F8A98 locknflags 00060000 Flat locked by threadIdent 6. Entrycount 1 com.company.och2.store.TransactionManager_at_801FD110/801FD118 locknflags 00220000 Flat locked by threadIdent 22. Entrycount 1 java.util.Vector_at_80290B60/80290B68
locknflags 80002400 Monitor inflated infl_mon 0x308642cc java.util.Vector_at_802AD650/802AD658
locknflags 00280000 Flat locked by threadIdent 28. Entrycount 1 com.company.och2.store.TransactionManager_at_80698860/80698868 locknflags 00270000 Flat locked by threadIdent 27. Entrycount 1 java.net.DatagramPacket_at_807CA370/807CA378 locknflags 00060000 Flat locked by threadIdent 6. Entrycount 1 oracle.jdbc.driver.OraclePreparedStatement_at_80982990/80982998 locknflags 00260000 Flat locked by threadIdent 26. Entrycount 1 oracle.jdbc.driver.OraclePreparedStatement_at_80A1FCE0/80A1FCE8 locknflags 00150000 Flat locked by threadIdent 15. Entrycount 1 java.net.DatagramPacket_at_80F3C980/80F3C988 locknflags 00240000 Flat locked by threadIdent 24. Entrycount 1 java.net.DatagramSocket_at_80F3C9E8/80F3C9F0 locknflags 00240000 Flat locked by threadIdent 24. Entrycount 1

Respect
Benjamin Soelberg
System developer Received on Mon Mar 25 2002 - 16:31:43 CET

Original text of this message