Re: Strange problem releasing inodes

On Jan 26, 8:02 am, "EdStevens" <quetico_..._at_yahoo.com> wrote:
> On Jan 26, 9:34 am, "Steve Howard" <stevedhow..._at_gmail.com> wrote:
>
>
>
>
>
> > On Jan 26, 10:06 am, "EdStevens" <quetico_..._at_yahoo.com> wrote:
>
> > > OS: SunOS 5.10,Sun-Fire-V440
> > > Oracle: 10.2.0.1.0
>
> > > A couple of months ago we had an issue on this server with the OS
> > > claiming we were running out of space on the file system holding most
> > > of the Oracle files. Closer investigation showed 'some sort of
> > > problem with the inodes'. We stopped the db, the SA ran some
> > > utility, and the space usage dropped from 98% to around 50%.
>
> > > A couple of weeks ago, began getting indications were were in the same
> > > situation. Files are not growing, but reported FS usage is. Another
> > > repair operation was scheduled for last night. As soon as we stopped
> > > the databases the SA reported that reported FS usage immediately
> > > dropped.
>
> > > As far as database usage on this server, we have three databases. One
> > > is rather small, vanilla db backing a packaged app (Clarity/Niku); one
> > > is a physical standby running in recovery mode, and the third is an
> > > 8.1.7.4 that serves as a proxy client between the app and the physical
> > > standby in the event it becomes the primary. (Long story. Short
> > > version is app was written in Pro*C 8.1, then the source code lost.
> > > Solution is to put up an 8.1.7.4 for the app to connect to, and use
> > > synonyms and a db link to pass all activity through to the 10.2 db
> > > where the data actually resides.) We have a shell script that runs
> > > daily (via cron) to delete old archive logs a few days after they have
> > > been applied to the physical standby. Since the archivelogs are
> > > written to a new directory each day
> > > (/db01/archive/epspd/arc/<SID>/archivelog/<yyyymmdd>, the delete is
> > > actually deleting the date-based directory, not the enumerated
> > > archivelog files:
>
> > > find ${TARGET_DIR} \( -name "*" -type d \) -mtime +$RETAIN_DAYS -exec
> > > rm -rf {} \;
>
> > > Any ideas?
>
> > > BTW, today is my last day on this job -- starting a new job on Monday.
> > > Of course, I'll be able to follow and participate in this thread, but
> > > won't be able to work a TAR with MetaLink or actually follow up with
> > > the SA's here. But I thought this was a pretty interesting problem
> > > that was worth some discussion.Hi Ed,
>
> > The only time we have seen this is when someone deleted a trace file to
> > which Oracle was till writing, or deleted an archive log (or any file)
> > to which Oracle was still writing. Do you have access to fuser or lsof
> > to see what files are open by Oracle?
>
> > HTH,
>
> > SteveI've never used either of those, but here's what I see:
>
> first, with fuser:
>
> cdnbwecp.epspd> pwd
> /db01/archive/epspd/arc/EPSPDECP/archivelog
> cdnbwecp.epspd> ls -l
> total 22
> drwxr-x--- 2 oracle dba 2048 Jan 21 23:10 2007_01_21
> drwxr-x--- 2 oracle dba 2048 Jan 22 23:12 2007_01_22
> drwxr-x--- 2 oracle dba 2048 Jan 23 23:11 2007_01_23
> drwxr-x--- 2 oracle dba 2048 Jan 24 23:11 2007_01_24
> drwxr-x--- 2 oracle dba 2048 Jan 25 23:11 2007_01_25
> drwxr-x--- 2 oracle dba 1024 Jan 26 09:13 2007_01_26
> cdnbwecp.epspd> fuser 2007_01_21
> 2007_01_21:
> cdnbwecp.epspd>
>
> So, I'm thinking fuser is showing no processes have that directory. I
> also stepped down into the directory and
>
> cdnbwecp.epspd> fuser o1_mf_1_8881_2v8ksr2t_.arc
> o1_mf_1_8881_2v8ksr2t_.arc:
> cdnbwecp.epspd>
>
> Looks like no processes have the individual file.
>
> It appears lsof is not installed. At least there is no man page for it.

Google for it,
http://ftp.cerias.purdue.edu/pub/tools/unix/sysutils/lsof/

Have you dropped any datafiles in the 8 db? They don't really go away until you
stop the db (actually, close the file), but they aren't visible in the unix directory listing.
There was also a bug in some versions where otrace was accidentally left on, maybe
someone wrote a cron to delete the files (and again, they don't go away until closed,
though this is a bit of a reach - or was that O7)?

Good luck on the new job!

jg

--
@home.com is bogus.
What's next, timesharing?
http://www.signonsandiego.com/uniontrib/20070126/news_1b26luddy.html