Re: Expect to work with terrabytes.. now what?

We run two servers each with 1.5TB of data. THere are two key things to remember:

1     everything takes longer to administrate
2     see 1

The key area this affects is backup and recovery. Depending on your OS and Oracle release, what used to take minutes can take hours or days.

Properly set up, queries should not take any longer than on smaller systems, aside from data volume factors.

Maximize your db_block size (16K or 32K, if your OS supports it). You will need the larger block size to maximize your database size.

On some of the older Oracle releases, 1.5TB is about the maximum you can get because there is a maximum number of data files and most OSs have/had maximum file sizes. On the newer releases, 8.x, this is not as much a factor.

Keep as much data as possible in read only status at the tablespace level. By doing this, you can set Oracle so that it only touches the datafiles when it is doing a query. Otherwize, expect a shutdown normal or startup to take 4-6 hours, each, depending on your storage media.

Placing the data in read only mode also simplifies backup/recovery. Make one copy of the read only files, so if you get file failure you just have to restore the os level copy. Backups are then just your updateable tablespaces, which I hope are only a small percentage of the db.

Recovery of a 1.5TB database will take weeks.

Welcome to the world of VLDBs. It is very different from the land of the littles (say 200GB and less.)
In article <7ld1je$ceq$1_at_nnrp1.deja.com>,   akmal b. chaudhri <akmal_chaudhri_at_my-deja.com> wrote:
> In article <376FE86E.C46DEB60_at_gnncast.net>,
> Daniel Murray <dmurray_at_gnncast.net> wrote:
> > One thing you could possibly do is summarize the data if it is
> possible. It
> > will cut down on the number of entries, but implementing it can be a
> little
> > tricky.
> >
> > Ferdinand Swaters wrote:
> >
> > > Hi All,
> > >
> > > I am currently on a project where the data in my database is going
> to run
> > > into the 1-10 Terabyte range. I have no experience with these
sizes
> of data
> > > in a DBMS, and I am wondering what new pitfalls I might run into.
> > >
> > > The system will do almost no updates on the data, most is storage
> and then
> > > delete after a while. There won't be many queries on the data, but
> the
> > > queries will be large. It is much like a data warehouse situation.
> The DBMS
> > > will be oracle.
> > >
> > > I don't have a good idea on what questions to ask first, because
> I'm mainly
> > > concerned about the things I might forget now, and run into when
> the beast
> > > is operational. One thing that I would like to know is if there is
> a way I
> > > could compress the data while it is in the database. That would
> reduce
> > > storage space required by a huge factor.
> > >
> > > Anyone has some experience they wish to share?
> > >
> > > advTHANKSance,
> > >
> > > Ferd.
> >
> >
>
> The management of large quantities of data is something the group at
> CERN have reported in a number of papers. They have a number of
similar
> requirements to yourself, but use an OODB. However, their experiences
> may still be of value to you. Try searching for RD45 at the CERN web
> site: www.cern.ch. One paper recent paper also appeared here:
>
> http://www.disi.unige.it/conferences/oodbws99/programme.html
>
> Good luck.
>
> akmal
>
> --
> email: akmal(at)bigfoot(dot)com || http://www.bigfoot.com/~akmal/
> "I need a job where my immense ego seems normal." - Dogbert
> DISCLAIMER: I work for Computer Associates and my opinions are my own.
>
> Sent via Deja.com http://www.deja.com/
> Share what you know. Learn what you don't.
>

--
Joseph R.P. Maloney, CCP,CSP,CDP
MPiR, Inc.
502-451-7404
some witty phrase goes here, I think.


Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.