Re: raw devices

From: Don Granaman <granaman_at_home.com>
Date: Tue, 21 Aug 2001 01:06:06 -0700
Message-ID: <F001.00371DB9.20010821003523@fatcity.com>

Raw versus filesystem is almost a religious issue. There are very good arguments both ways. The main advantages of filesystems are ease of use and flexibility. I tend to favor raw devices for critical production systems - for write-intensive OLTP at least. For development and functional test servers, at least, filesystems are simpler, more flexible and usually more appropriate. (Incidentally, one CAN clone a raw database to filesystem without any special magic.)

For a more thorough discussion of some of the performance issues, see Jonathan Lewis' site at http://www.jlcomp.demon.co.uk , specifically see http://www.jlcomp.demon.co.uk/raworfs_i.html Also see Steve Adams' site at http://www.ixora.com.au , specifically see http://www.ixora.com.au/tips/creation/raw_datafiles.htm . Search on "raw device" to find more there.
(Sorry if this is some sort of infraction by posting direct links to the topic pages!) There are a number of other such references - on both sides of the argument. (I'm looking forward to Gaja's well-reasoned arguments against raws!)

The following is not at all comprehensive, but contains a few considerations not often mentioned.

Filesystem
-advantages

    Flexibility - easy to add, delete, and resize files. Autoextend, etc.     File system read buffering MAY actually be an advantage for some Oracle systems.

    Easy to back up with anything - including cp or copy     Everybody "understands" files (but often less well than they think - see next)
-disadvantages

    Files and filesystems can get extremely fragmented! (Think about it - resize, autoextend,...)

    Write performance penalty
    OS-level datafile locking issues
    May not support async I/O or may not support it well     (UFS) fsck of large filesystems can take FOREVER after a crash!

Raw device
-advantages

    Significant write performance advantage     File system buffering MAY be a disadvantage     Guaranteed sequential datafile space - if the raws are built that way     Much more difficult to accidentally munge (e.g. "$ rm *" OOPS!!!)     Async I/O support [on any (?) platform] -disadvantages

    Requires much more preparation work for layout (This is a disadvantage? ;-)

    More administrative overhead - at least initially     Backups must be done with something that handles raw devices     Not as easy to move things about and perhaps to resize or autoextend them

Note that there is no particular distinction made above between UFS and journaled filesystems (e.g VxFS). Due to space and time constraints, I am arbitrarily declaring that "out of scope" for my part of this discussion! The URLs above do consider that distinction in some detail.

In general, filesystems are easier, but are, at least in my opinion and experience, less secure and perform worse - at least for write-intensive systems. Some of the traditional arguments against raws are largely moot today. For example:

Backups - most modern backup utilities can handle raws transparently.

Administration - modern volume managers make setting up raws actually a bit easier than setting up filesystems. However, you will still have a lot more of them since you need a raw device for every datafile, redo log, or control file you place on raw.

Raws do still introduce some constraints. Chief among them is up front planning for the layout and its prerequisite analysis.

My personal regimen for making raw devices usable includes:

• Get the system administrators onboard early. I have often found initial resistance, then their conversion to advocates.
• Create a dedicated "disk group" (or whatever it might be called in your volume manager) for your database disks. Perhaps one for each database if multiple Oracle databases exist, (Aside: One of the worst raw device fiascoes I ever saw was when a rookie sys admin at a client site, an major telecom, overwrote several raw datafiles. He was asked to add space to some filesystem and was unaware of the procedures and conventions in place to prevent this from happening - like reserving everything in oradg for Oracle!)
• Keep the number of distinct raw device sizes small. Perhaps 100M, 500M, 1000M, & 2000M. (Note: Pick your own favorites. This is just an example. Also, the "real" size may need to be a bit bigger - you can't actually create 20 * 100M extents in a tablespace on a 2000M raw device (or a 2000M UFS file for that matter.) Also, reconsider whether or not you actually need datafiles larger than 2 GB. (I don't like "custom sizing" datafiles - not even very large ones. I would much rather have a tablespace with a number of 2 GB datafiles than with one huge datafile.)
• Go through a very serious I/O pattern analysis and a thorough space utilization/growth analysis in preparation of the layout. This is, by far, the most difficult and time-consuming part. But, its something you want to / should do anyway. Raws are a convenient excuse!
• Create additional "spare" raw devices up front. If something unforeseen happens (eventually, it will) and you get blindsided by suddenly running out of space somewhere, you can simply "alter tablespace add datafile '/dev/vx/rdsk/oradg/dsk01_11_2001M' size ..." instead of having to start from scratch building a raw device.
• Build scripts to dynamically identify raw devices in use and free. For Veritos, I use a shell script to: 1) '$find /dev/vx/rdsk/oradg -type c -print' | sort # creates a list of all reserved raw devices 2) call a SQL script to select the filenames of all datafiles, redo log files, control files, and temp files from the data dictionary and sort the output. # creates a list of used raw devices 3) diff the output from 1 & 2 # creates a list of free raw devices. [Note: If the disk group is used by multiple databases, be sure to check them all!]
• Realize that you CAN resize a datafile within a raw device - but only up to the raw size.
• Establish good naming conventions for disks, disk groups, volumes, etc. so you can tell by the name something about the physical nature. An example from Veritos - dedicated disk group for the database named "oradg" and disks named so as the provide information about controller, array, etc. Volumes named so as to provide information about the physical configuration. This is going to vary with the hardware and needs, but one convention I often use is to create "stripe sets" - sets of perhaps four disks to stripe across. Then I might name the raw volumes something like "ss01_04_1001m", "ss01_05_1001m", ..., "ss01_12_2001m", "ss04_01_01501M", etc. Volumes on single disks might be named something like 'dsk1_04_0101M", etc. The first element (e.g. "ss01", or "dsk1") would indicated the physical disk or stripe set. The second element (e.g. "05") indicates the location on the stripe set or disk. The third element is (an approximation of) the size. There are a number of variations on this theme - pick whatever makes sense to you. The basic idea is just to have the name mean something useful. If you want to quickly add space to a tablespace on a particular disk or stripe set without disturbing the I/O distribution, this makes it easy. You do not need (or want) to follow OFA here! You can create OFA-compliant links (e.g "$ ln -s ...") if you wish. Unfortunately, one of the most common "mistakes" is to just create a mess of raw devices named "vol01", "vol02", "vol03", ..., "vol75". This "convention" requires a journey of discovery whenever anything needs to be done with them.

Subjectively, I've had far fewer problems with raws over the last decade+ than with any type of filesystem - far fewer corruptions, mysterious disappearances, bugs, etc.

A very thorough discussion of all aspects of this would be huge - perhaps a book (but undoubtedly a very poor seller)!

-Don Granaman
[certifiable Orasaurus]

• Original Message ----- To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com> Sent: Monday, August 20, 2001 12:22 PM

> Don & Oracle Gurus,
>
> Going by your answers, I have  a question for you?  What you have

explained
> is excatly the same environment we had back in our previous work place. But
> here we have everything what you have MINUS the raw devices. The question I
> have for Don and the forum folks is, What is the benefits of having flat > files over raw? I have been working with RDBMS for over 10 years and I have

> always worked with RAw and I have raw as recommended by MS,SYBASE and
> INformix.
>
> The reason for this question is because we are moving to RMAN/Veritas and
> EMC Timefinder, I suggested things would be better if we move to raw.  But

I

> could not prove for the fact that Raw is better of Flat FS.
>
> Please Suggest from your past experiences.
>
> Thanx
>
> Srikanth

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.com
-- 
Author: Don Granaman
  INET: granaman_at_home.com

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Received on Tue Aug 21 2001 - 03:06:06 CDT