Re: ORA 03113 when initializing Database.

From: Pusztai Viktor <pusztai_at_mail.ktm.hu>
Date: Thu, 08 Mar 2001 15:28:22 +0100
Message-ID: <3AA79706.A497C408@mail.ktm.hu>

Hi,

The important two informations are at the end of this page. Read it carefully!

Viktor

Oracle does not start on RedHat Linux 7.0 after relinking

Versions Affected

Product & Oracle versions affected

8.1.6.1, 8.1.7 (verified)
8.0.5.x, 8.0.6.x, 8.1.5.x (not verified)

Platforms Affected

RedHat Linux 7.0 and any glibc-2.1.9x or glibc-2.2 based distribution.

Description

After upgrading from earlier versions of RedHat Linux to RedHat 7.0 Oracle fails to start up after being relinked. The reason is a generic incompatibility across different major versions of GNU libc that does not guarantee that a binary linked against a version of glibc will keep working if relinked in an environment of a later major glibc release. Since Oracle 8.1.6.1 is linked against glibc-2.1.3 and RedHat 7.0 ships with glibc-2.1.9x, we experience the described problem. This will also happen with 8.0.5, 8.0.6 and 8.1.7; the w/a here described only applies to 8.1.6.1, but its core also works for 8.1.7. This note will be updated shortly to include the 8.1.7 workaround (currently being reviewed).

There will be no w/a for 8.0.5 since it already requires an older compatibility package than the one that is needed for 8.0.6; customers needing to upgrade to newer distributions will also need to upgrade to more recent Oracle versions.

Likelihood of Occurrence

This problem will occur after relinking Oracle to install patches or kernel options or to deinstall them. It will NOT occur if Oracle is not relinked after the OS upgrade.

Possible Symptoms

Starting up the database fails with ORA-3113 after some time spent looping in semop() and a core file is generated. The stack trace taken with GDB on a 8.1.6.1 core is the following:

(gdb) where
#0 0x404c74e1 in __kill () from /lib/libc.so.6
#1 0x8469156 in slcra ()
#2 0x846cd01 in ssexhd ()
#3 0x4048ed60 in pthread_sighandler (signo=11, ctx={gs = 0, __gsh = 0,
fs = 0,

__fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0, edi = 3221208616,

esi = 3221208744, ebp = 3221208516, esp = 3221207644, ebx = 3221207676,

      edx = 0, ecx = 3221207676, eax = 0, trapno = 14, err = 4,
      eip = 138857746, cs = 35, __csh = 0, eflags = 2163270,
      esp_at_signal = 3221207644, ss = 43, __ssh = 0, fpstate = 0x0,
      oldmask = 0, cr2 = 8}) at signals.c:97

#4 <signal handler called>
#5 0x846cd12 in ssexhd ()
#6 0x4048ed60 in pthread_sighandler (signo=11, ctx={gs = 0, __gsh = 0,
fs = 0,

__fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0, edi = 3221209840,

esi = 3221209968, ebp = 3221209740, esp = 3221208868, ebx = 3221208900,

      edx = 0, ecx = 3221208900, eax = 0, trapno = 14, err = 4,
      eip = 138857746, cs = 35, __csh = 0, eflags = 2163351,
      esp_at_signal = 3221208868, ss = 43, __ssh = 0, fpstate = 0x0,
      oldmask = 0, cr2 = 8}) at signals.c:97

#7 <signal handler called>
#8 0x846cd12 in ssexhd ()
#9 0x4048ed60 in pthread_sighandler (signo=11, ctx={gs = 0, __gsh = 0,
fs = 0,

__fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0, edi = 3221211064,

esi = 3221211192, ebp = 3221210964, esp = 3221210092, ebx = 3221210124,

      edx = 0, ecx = 3221210124, eax = 0, trapno = 14, err = 4,
      eip = 138857746, cs = 35, __csh = 0, eflags = 2163347,
      esp_at_signal = 3221210092, ss = 43, __ssh = 0, fpstate = 0x0,
      oldmask = 0, cr2 = 8}) at signals.c:97

#10 <signal handler called>
#11 0x846cd12 in ssexhd ()
#12 0x4048ed60 in pthread_sighandler (signo=11, ctx={gs = 0, __gsh = 0,
fs = 0,

__fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0, edi = 153736192,

esi = 3221213276, ebp = 3221211380, esp = 3221211320, ebx = 3575600,

      edx = 3221211332, ecx = 1745, eax = 502, trapno = 14, err = 6,
      eip = 145158780, cs = 35, __csh = 0, eflags = 2163219,
      esp_at_signal = 3221211320, ss = 43, __ssh = 0, fpstate = 0x0,
      oldmask = 2147483648, cr2 = 1745}) at signals.c:97

#13 <signal handler called>
#14 0x8a6f27c in sskgmstat ()

Workaround

(for 8.1.6.1 on RedHat 7.0) Install the following packages:

compat-egcs-6.2-1.1.2.9
compat-glibc-6.2-2.1.3.2

and modify all occurences of CC and LINK definitions in all env_*.mk makefiles in the $ORACLE_HOME software tree from either $(LDCCOM) or gcc to i386-glibc21-linux-gcc.

To better illustrate the needed changes, the lines such as

    CC=cc
    LINK=$(LDCCOM)
    LINK=$(PURECMDS) gcc $(LDFLAGS) $(COMPOBJS)

will become

    CC=i386-glibc21-linux-gcc
    LINK=i386-glibc21-linux-gcc
    LINK=$(PURECMDS) i386-glibc21-linux-gcc $(LDFLAGS) $(COMPOBJS)

For 817 you only need to change the CC and the second LINK lines.

You will also need to modify the genclntsh shell script and substitute

LD="ld -shared -L${ORACLE_HOME}/lib" LD_RUNTIME="-R${ORACLE_HOME}/lib"
LD_OPT="-h ${CLNT_LIB}"

with

LD="i386-glibc21-linux-gcc -shared -L${ORACLE_HOME}/lib" LD_RUNTIME="-Wl,-R${ORACLE_HOME}/lib" LD_OPT="-Wl,-h${CLNT_LIB}"

Finally, rebuild libclntsh.so by running 'genclntsh' and relink oracle,

svrmgrl etc. using the usual relink commands. You'll get warnings in the relink process but they can be ignored.

This has been tested on env_rdbms.mk, env_sqlplus.mk, env_network.mk, env_oemagent.mk, env_precomp.mk but should be working on any env*.mk makefile.

Other possible workarounds:

downgrade to RedHat 6.2
get a copy of the oracle kernel and utilities such as svrmgrl, imp,

exp etc. from a RedHat 6.2 box and place them in $ORACLE_HOME/bin

if you haven't relinked multiple times, it's enough to copy back the old binary versions in $ORACLE_HOME/bin (such as oracleO, impO etc.) to oracle, imp etc.

Note that the two latter workarounds will not allow you to relink the

Oracle kernel and/or utilities - the resulting binaries will not run

under a glibc-2.1.9x system.

Dirk Reichert írta:

> I'm near suicide.
>
> I installed Oracle 8.1.7 under RH Linux. Installation runs perfect. But when
> I try to create a database I says unable to connect to Oracle.
> When I try to open the prebuild database from the Installation I get ORA
> 03113.
>
> Any idea ?

Received on Thu Mar 08 2001 - 08:28:22 CST