listener crashes

The other day I received a call from one of the sites that I service as a DBA. No one was able to log in with the client apps which access the database (Oracle 7.3.4/Unixware 2.1.2 - I also use multi-threaded servers).

I logged into the server and discovered that the databases were fine when I did a bequeth connection, but nothing was working through the listener. Further investigation show that tnslsnr was running, but all of the dispatchers (10/instance) and server (30/instance) processes were gone. I tried entering "lsnrctl stop" and received a variety of error messages which I didn't write down but basically they were to the effect that the listener could not connect and that there was a tcp/ip problem. This still left tnslsnr running so I got rid of it with 'kill'. At this point the only oracle processes running were the normal dbwr,smon,pmon, etc. I then tried to start the listner with 'lsnrctl start' and received the same error messages as before. I tried it again and then a third time. At this point I did a ps -ef and discovered that I had this incredible number of dispatchers and server processes but no tnslsnr. At that point I was pushed for time and didn't have the luxury of just calmly assessing the situation so I told the sysadmin to reboot the machine. After the reboot everything was fine.

The only other factor in this whole puzzle was that the evening before there was a "show-and-tell" demonstration at a remote site which access the databases using the same software which is used for on-site operations. The network latency was pretty high and apparently in the midst of the test is when the listener crashed. I'm not saying that the demo caused the crash - just that it occurred while the demo was in progress. The individuals who were involved in the test didn't call me until the following morning when I discovered the situation I described above.

Has anyone seen anything like this? Was there a "graceful" way to handle this which would not have required rebooting the server? I didn't have a lot of time because operations were about to begin for the day and the users were not willing to wait while I performed extensive trouble-shooting. Is it possible that the network latency caused the listener to crash?

Ken Received on Tue Jul 06 1999 - 07:25:03 CDT