Re: How are you backing up your V,VLDB????

> That's indeed true. However, I would want to add one
> more "the most significant strategic point" and
> that's proper analysis of real needs.
> Something like: Do we really need all this data
> loaded in our database and stored for next hundred
> of years?
>
> In past I've encountered few times some databases specially
> data warehouses, where its proud owner told me: "We're storing
> 2TB right now and it will grow even further". And after that
> I've realized that inside is a lot of irrelevant crap, which
> is inside just because "maybe we will need this one day"
> and no one was enough brave to told managers that this particular
> information is an useless piece of sh*t.
> Not mentioning redundant data, because data model is bad and/or
> data cleaning was not sufficient or is not implemented at all.

Good points, but as you can see from one of my other replies. This is *not* a data warehouse. We are not archiving data either. This database will hold spatial data. More specifically, our largest data sets in the database will be "raster" spatial data. These data sets can grow to be quite large. We are getting data from satellites (mostly) that take images of the Earth. For instance, one of our datasets holds the National Elevation Dataset (NED) for the entire US. The Continental US is 50GB alone, not including Alaska, Hawaii, and other US territories. Each pixel in the dataset represents the elevation above sea level. And each pixel represents that elevation for a 30 meter portion of the US. As you can probably imagine, this gets to be quite large, quite quickly.

To make matters worse, we are in the process of loading and storing satellite images of selected cities in the US. These images are quite amazing in the amount of detail that one can see. We recently loaded the entire city of Chicago as one large image. The image had enough detail to see vents on the tops of buildings, or lines painted on a tennis or basketball court. You could even make out the net on the tennis court. This is not military-quality satellite imagery either. So this is not spy satellite stuff. Just to hold the entire city of Chicago required about 560GB of storage space inside the database. We will be loading 133 cities in all, with the New York City (including Newark, NJ) area being the largest. It is projected to be over 1TB just for this one urban area! All told, we are projecting 20TB for just the imagery of these cities. This doesn't include all of the other spatial datasets that we want to load.

All of this data will be used for many federal agencies for a variety of reasons. The Forestry Service is already using some of our data for fighting forest fires in the summer. Various federal departments will use this data for their needs as well. So you can probably see that this isn't the type of project where someone says, "Let's store this data because we might need it in the future. And let's warehouse this data over time so that we have a history". So the solution I'm looking for is not one where we just scrap our data rather than figure out ways of backing it up. We must back this data up. All of it.

Thanks,
Brian Received on Thu Jan 30 2003 - 10:28:14 CST