Cray, BIG Data, Virtual Memory - Virtual Performance

From: Taylor, Chris David <ChrisDavid.Taylor_at_ingrambarge.com>
Date: Tue, 31 Jan 2012 08:54:09 -0600
Message-ID: <C5533BD628A9524496D63801704AE56D6ADD1619EE_at_SPOBMEXC14.adprod.directory>



Mark,

I was unfamiliar with Cray's stance until you mentioned it and I'm curious where this leads us when discussing big data? Specifically, "sets" of big data. Since we (as humans) are storing more and more data (Hoarders?), and we want to be able to analyze it in any number of ways, I wonder what are the implications of storing and processing large amounts of data?

It seems to me that as data stored reaches gargantuan proportions, that by necessity the jobs that process such data will all require gargantuan amounts of resources to be able to complete successfully.

It also seems to me, that in the large scheme of things we are very early in the computing age, and in an even earlier (infantile?) developmental stage of data storage.

I wonder what are some theoretical solutions to break down gargantuan data into manageable chunks? Even partitioned data will become unmanageable (I think) at the current rate of data storage.

Honestly, I'm kind of dreading being a DBA over the next 25 years (give or take 20) when contemplating how much data I may be responsible for managing (performance with backup and recovery). I'm excited about seeing what breakthroughs we come up with technology wise, but I'm not looking forward to having to manage it. (If that makes sense)

Chris Taylor
Sr. Oracle DBA
Ingram Barge Company
Nashville, TN 37205

"Quality is never an accident; it is always the result of intelligent effort." -- John Ruskin (English Writer 1819-1900)

CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender immediately and delete the contents of this message without disclosing the contents to anyone, using them for any purpose, or storing or copying the information on any medium.

-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Mark W. Farnham Sent: Tuesday, January 31, 2012 8:17 AM
To: oracle-l_at_freelists.org
Subject: RE: Killer SQL and PGA

ulimit?

The tradeoff here is whether to utilize virtual memory to allow gargantuan jobs to theoretically be processed at all. Seymour Cray nailed that in one when he was asked why his systems did not support virtual memory.

It will always be possible to concoct queries and problems that cannot be solved within the limits of any arbitrary amount of real memory.

--
http://www.freelists.org/webpage/oracle-l
Received on Tue Jan 31 2012 - 08:54:09 CST

Original text of this message