Re: How to store 50 Terabytes per day?

Ian,

Comments inline...

on 9/8/04 10:43 AM, MacGregor, Ian A. at ian_at_slac.stanford.edu wrote:

> Before replying please see:
>
> http://www-search.slac.stanford.edu:8765/query.html?col=3Dslacweb&col=3Dt=
> odays&col=3Dssrlweb&col=3Dslacpubs&col=3Dbabarweb&qt=3Dlsst&charset=3Diso=
> -8859-1&qc=3Dslacweb+todays+ssrlweb+slacpubs+babarweb
>
> Then choose "LSST for SDSC". This is a powerpoint presentation about =
> the project that contains some information on the computing =
> requirements.
>
> Problems with Oracle: What is the maximum number of partitions for a
> database?

65,535 partitions or subparitions, through 9iR2. Don't know if 10g (or 11g?) will raise that limit?

> Will hierarchical file management systems does it work with?

Oracle works with various near-line storage (i.e. NAS, SAMFS, etc), but I'm not sure what HFS it works with...

> What if that system needs to be extended?

With RAC, current SMP servers that scale to 144 processors or more can be clustered together to create truly insane number of processors, if necessary. But, as I earlier described, I don't think that CPU will be the bottleneck in such a system, it'll be storage throughput.

> Is Oracle's one-cluster hence one-architecture technology right for the
> project?

Remember the parable about the tower of Babel? Unless you've really got faith in all this "grid" stuff, I don't see an alternative. At least Oracle provides several viable choices; Teradata doesn't.

> Does Oracle RAC tie you to a single vendor? Can RAC's communicate with other
> RAC's?

I'm guessing that one stack of vendors is more than enough (i.e. RDBMS software, OS, storage SW, server HW, storage network, etc). Can you imagine dealing with more than one vendor at each layer of that stack?

> Does anyone think this will be done using RAID10.

Is there a viable alternative, considering the throughput rates?  

> I'm not sure how much of the data will be online annd how much will be =
> nearline storage

Nearline storage is "online" from Oracle's perspective, just slower. My guess is that all data will have to be moved to nearline after a certain period of time (a couple months?), so if 7 years (just a guess) is retained, then about 3 Pb will be online and the remaining 127 Pb will be nearline. Call 1% versus 99%?

> Ian Macgregor
> Stanford Linear Accelerator Center
> ian_at_slac.stanford.edu =20
>
> =20
>
> =20
>
> -----Original Message-----
> From: oracle-l-bounce_at_freelists.org =
> [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of =
> Jared.Still_at_radisys.com
> Sent: Wednesday, September 08, 2004 9:08 AM
> To: oracle-l_at_freelists.org
> Subject: How to store 50 Terabytes per day?
>
> This quote appears in the following article:=20
> http://www.computerworld.com/hardwaretopics/storage/story/0,10801,95694,0=
> 0.html?nas=3DDM-95694
> " When it's in operation in 2011 at a site still to be determined, the =
> telescope being built for the LSST project will collect data at a rate =
> of about 6GB (equivalent to one DVD) per 10 seconds, generating many =
> petabytes of data over time. One petabyte equals roughly 100 times the =
> printed contents of the Library of Congress. The LSST project "pushes =
> forward database technology dramatically," says Philip Pinto, a physics =
> professor at Steward Observatory at the University of Arizona and a =
> member of the LSST project's board of directors. "The LSST database will =
> probably be the largest known nonproprietary database in the world."=20
> So if you were faced with the task of storing 50 Terabytes per day, what =
> kind of architecture would it require?
> Do you think Oracle would hold up with a transaction rate of 600 =
> Megabytes per second?
> The architecture of such a beast could drive out some interesting =
> developments for more general use.
> Jared

--
To unsubscribe - mailto:oracle-l-request_at_freelists.org&subject=unsubscribe 
To search the archives - http://www.freelists.org/archives/oracle-l/