Re: Storage System Bakeoff test plans

Oracle Orion is a IO simulation tool that uses the same code path that the Oracle database uses for its IO calls. It can be quite useful, but you have to understand how to use it. It's very important to understand the IO pattern of the application/database you want to simulate. Specifically for a warehouse this means understanding your query workload. Are the queries using partition scans or index access? Are you using parallel query or not? Are you using nologging/direct path operations or not? How much of the workload fits into each of those categories? What is the IOPS and I/O bandwidth requirement? etc, etc, etc.

I'm a frequent user of Orion to get a "max observable" I/O bandwidth number for the kits I use in data warehouse benchmarks (I use dd as well). When I work with a hardware vendor we generally target a given GB/s I/O throughput number. Some math is worked on paper, and we validate our math with the Orion and dd tests. This gives us a max observable I/O bandwidth number for the hardware using a low level I/O tool. If the observed number is significantly different from the calculated number, investigation is done to understand why. For these tests I use a 1MB IO size which is the largest IO size that the Oracle database currently requests. This effort allows me to understand how much of the potential resource is being consumed by the database when the real workload is running. Obviously it is not possible to drive more I/O with a database than low level tool.

I could see you spending a month doing tests and going through the results. Consider the number of combinations of vendors, LUN configurations, RAID configurations, I/O multipathing, I/O load balancing, simulations, and the fact that you will want to probably run the tests more than once to make sure there are no anomalies.

It's educational to run some micro benchmarks but I would also run tests that reflect the important parts of your business processing. If "all the users in the world" don't use your database, then why test it? Try and keep your tests clean and incremental. No sense of running the "everything" test when you haven't tested each individual part. If the results don't make sense, understand why, maybe it's not the storage that is the bottleneck. I often tell people benchmarking is like high school chemistry: First you work out the experiment on paper, then you go into the lab and see if you get the expected results. If the actual results don't reflect the expected results, understand where it went wrong.

Best of luck.

On Wed, Apr 9, 2008 at 5:53 AM, April Wells <awells_at_netspend.com> wrote:
> We are looking at doing a 'bakeoff' between three different storage vendors
> and I 'get' to do the test plan (as well as implement most of the plan).
>
> Has anyone used the Oracle Orion tool? The claim is that is all that we
> should have to use to load test the data warehouse on these three platforms.
> I'm not sure I buy it, but I would welcome the input of anyone who has
> actually used the tool. How is it at OLAP profiling?
>
> I am of the opinion that we ought to run actually "stuff" through the
> warehouse to test the performance (you know, things like the nightly ETL job
> at the same time we run the query from… well… you know) but it looks like I
> may be out voted. Opinions?
>
> We are planning on running one day of normal type processing, one day of
> 'the heaviest season of the year' and one day of 'all the users in the world
> are hitting it'… but all programmed with Orion… Suggestions?

-- 
Regards,

Greg Rahn
http://structureddata.org
--
http://www.freelists.org/webpage/oracle-l