Re: Has anyone developed applications with DB's over 100GB?

David Williams wrote:  

> In article <34C25418.B7525596_at_thunderstone.com.removethis>, THUNDERSTONE
> <bart_at_thunderstone.com.removethis> writes
> >analysis to make it work.
 

> >We've done databases in the size range you describe with our Texis
> >RDBMS, but the
> >key issues were more related to the scope of the queries and how long
> >they would
> >take. The disk managment and backup problems are a big pain at this size
> >too.
 

> Has anyone planned a restore of a database of this size? i.e. what
> happens if BOTH disks in what is assume would be mirrored get
> corrupted. How long would a full restore take? Has the disk layout
> been planned to minimize the downtime in the case of full restores?
> How long do you think it would take?
 

> I see you standin', Standin' on your own, It's such a lonely place for you, For
> you to be If you need a shoulder, Or if you need a friend, I'll be here
> standing, Until the bitter end...
> So don't chastise me Or think I, I mean you harm...
> All I ever wanted Was for you To know that I care

Our news database is on the same order (~120GB) (and our SEC filings database is in the 200GB range but that is on a separate set of machines from the news databases). We use Informix Online Dynamic Server 7.21 (ODS) (upgrading to 7.24 next week) and it handles the database just fine. We service 87000 users load from 30000 to 60000 new news stories daily and respond to an average load of 5000 queries per minute on each of the five sister machines (we can handle peak load on only three machines if needed but five gives the news product manager the warm fuzzies). Informix Standard Engine (SE) and Online version 5.xx cannot handle databases of this size. SE tables are limited to the maximum system file size (usually 2GB) and Online 5.xx is limited to about 56 disk "chunks" of 2 or 4GB each (platform dependent) or about 112GB (or 224GB) with a maximum table size of 32GB (or 64GB).

ODS 7.xx has no practical limitations (2048 chunks or 4TB or 8TB) and table size is similarly unlimited.

We archive using Informix's simplest archive tool (there are currently three) to a single five drive DLT RAID tape array in about 15 hours and have restored the entire database in about 8 hours. When we upgrade to 7.24 we should be able to move to the most sophisticated archive utility (onbar) which will allow backing to multiple tape arrays in parallel and we should be able to cut that in half by adding a second array. For platforms that do not support RAID tape arrays the onbar utility's multiple tape drive capability is the only practical option.

Our platform, currently, is Data General Aviion 10000 (M88K) systems. We are moving to Sun going forward as DG no longer supports the M88K CPU and has moved to Intel based servers which cause too many problems for our environment. With the faster CPUs on the Suns perhaps I can reduce the number of news machines to three or four. Hope that this helps.

Art S. Kagel Received on Tue Jan 20 1998 - 00:00:00 CST