Kevin Closson
IOPS in a Very High-End NFS Environment?
Since I’m on site at a Beta customer (testing the product I work on at Oracle), this will be a quick blog entry. I’ve been meaning to direct folks to Gear6 for quite some time now. I have no stake in Gear6, so this is not a shameless plug. I think they solve interesting problems so if you are a large NFS shop, I’d recommend checking them out. They offer a plug-in NFS read-through cache and while I haven’t had first hand experience with their product, I know folks that have and they had good things to say about Gear6.
If any of you are confused about what NFS has to do with Oracle, I recommend this list of Oracle on NFS related posts.
Oracle Clusterware for Non-Real Application Clusters Purposes.
Quite some time back I made a blog entry about deploying Oracle Clusterware for non-RAC purposes. As I pointed out in that entry, there were license ramifications. That was then, this is now.
In this press release about Oracle Clusterware, Oracle is announcing that Oracle Enterprise Linux (with Unbreakable Linux Support) can deploy Oracle Clusterware to provide high availability services for any purpose they so desire.
Now that, is interesting.
Some additional, related links:
Making Applications Highly Available Using Oracle Clusterware
Considerations for “Stretch Clusters” with Oracle Real Application Clusters
Little Things Doth Crabby Make Part III. Non-Erroring Errors and Erroneous Experiments.
No worries, we won’t have to lower the Cone of Silence. True, you will see use of an “underbar” init.ora parameter in this post, but its use is not the central theme. No, no Silver Bullets here. This is another post in the Little Things Doth Crabby Make series.
I routinely brag about the sophistication level of my blog readers, so, folks, don’t let me down. Let’s start a thread about why the contents of the following session output would make my Little Things list. OK, come on…
SQL> set timing on
SQL>
SQL> alter session set "_parallel_broadcast_enabled" = FALSE
2
SQL> select count(*) from ap_ae_lines_all where AE_LINE_ID > 1397437860 ;
COUNT(*)
----------
0
Elapsed: 00:01:21.70
SQL>
SQL> alter session set "_parallel_broadcast_enabled" = FALSE;
Session altered.
Elapsed: 00:00:00.00
SQL>
SQL> select count(*) from ap_ae_lines_all where AE_LINE_ID > 1397437860 ;
COUNT(*)
----------
0
Elapsed: 00:01:30.46
Attempted Murder of a 4-Socket AMD Opteron Server with RHEL4. Oracle Can’t Kill It.
But my, oh my, how I’ve tried. OK, I guess my new name is Fan Boy. I know for a fact that I’ve been pretty relentless on this particular server for over 100 days of its current 215-day life.
-sh-3.00$ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 3) -sh-3.00$ uptime 14:41:17 up 215 days, 14:32, 15 users, load average: 37.85, 37.48, 25.89
And, top(1):
top - 14:40:44 up 215 days, 14:31, 15 users, load average: 40.91, 38.05, 25.62 Tasks: 309 total, 30 running, 278 sleeping, 0 stopped, 1 zombie Cpu0 : 92.8% us, 7.2% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu1 : 90.1% us, 9.9% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu2 : 89.3% us, 9.8% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.9% hi, 0.0% si Cpu3 : 90.1% us, 9.9% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu4 : 89.2% us, 9.9% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.9% hi, 0.0% si Cpu5 : 89.1% us, 10.9% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu6 : 92.8% us, 7.2% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu7 : 93.7% us, 6.3% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 10393736k total, 9347616k used, 1046120k free, 1892k buffers Swap: 10288440k total, 838236k used, 9450204k free, 6264396k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14919 kclosson 15 0 120m 84m 7076 S 30.3 0.8 0:17.67 sqlldr 14942 kclosson 15 0 119m 84m 7068 S 29.4 0.8 0:17.75 sqlldr 14940 kclosson 15 0 120m 84m 7068 S 28.6 0.8 0:16.21 sqlldr 15008 kclosson 16 0 668m 35m 29m R 28.6 0.3 0:16.48 oracle 14924 kclosson 15 0 119m 84m 7076 R 26.8 0.8 0:16.39 sqlldr 14932 kclosson 16 0 120m 84m 7068 R 26.8 0.8 0:17.07 sqlldr 14959 kclosson 15 0 668m 34m 29m S 25.9 0.3 0:15.96 oracle 14961 kclosson 16 0 668m 34m 29m R 25.9 0.3 0:14.90 oracle 14945 kclosson 15 0 119m 84m 7076 S 25.0 0.8 0:16.07 sqlldr 14980 kclosson 15 0 668m 34m 29m S 25.0 0.3 0:15.09 oracle 14935 kclosson 16 0 119m 84m 7068 S 24.1 0.8 0:15.05 sqlldr 14947 kclosson 16 0 119m 84m 7072 R 24.1 0.8 0:15.90 sqlldr 14943 kclosson 15 0 119m 84m 7076 R 23.2 0.8 0:14.75 sqlldr 14938 kclosson 16 0 120m 84m 7068 S 22.3 0.8 0:14.35 sqlldr 14941 kclosson 15 0 119m 84m 7076 R 22.3 0.8 0:15.96 sqlldr 14951 kclosson 15 0 120m 84m 7068 S 22.3 0.8 0:16.96 sqlldr 14921 kclosson 16 0 120m 84m 7068 R 21.4 0.8 0:17.84 sqlldr 14934 kclosson 15 0 120m 84m 7076 S 21.4 0.8 0:16.13 sqlldr 14929 kclosson 15 0 119m 84m 7076 R 20.5 0.8 0:17.70 sqlldr 14950 kclosson 16 0 119m 84m 7068 R 20.5 0.8 0:13.63 sqlldr 14922 kclosson 15 0 120m 84m 7068 S 19.6 0.8 0:17.40 sqlldr 14977 kclosson 15 0 668m 34m 29m R 18.7 0.3 0:16.38 oracle 15002 kclosson 16 0 668m 34m 29m R 18.7 0.3 0:15.00 oracle 14920 kclosson 16 0 119m 84m 7076 R 17.8 0.8 0:17.97 sqlldr 14923 kclosson 16 0 119m 84m 7068 R 17.0 0.8 0:13.44 sqlldr 14925 kclosson 16 0 120m 84m 7068 S 17.0 0.8 0:13.06 sqlldr 14927 kclosson 16 0 119m 84m 7076 R 17.0 0.8 0:15.05 sqlldr 14931 kclosson 16 0 119m 84m 7076 R 17.0 0.8 0:15.18 sqlldr 14957 kclosson 15 0 668m 34m 28m S 17.0 0.3 0:14.16 oracle 14930 kclosson 16 0 120m 84m 7068 R 16.1 0.8 0:15.31 sqlldr 14986 kclosson 15 0 668m 34m 29m R 16.1 0.3 0:14.37 oracle 14936 kclosson 15 0 119m 84m 7068 S 15.2 0.8 0:15.58 sqlldr 14964 kclosson 15 0 668m 34m 29m S 15.2 0.3 0:17.10 oracle 15014 kclosson 15 0 668m 34m 28m S 12.5 0.3 0:12.83 oracle 14949 kclosson 16 0 120m 84m 7076 S 7.1 0.8 0:15.70 sqlldr 14955 kclosson 16 0 666m 35m 31m R 4.5 0.4 0:03.11 oracle 14966 kclosson 16 0 666m 35m 31m R 4.5 0.3 0:02.80 oracle 14998 kclosson 15 0 666m 35m 31m S 4.5 0.3 0:02.68 oracle
Things You Do With a Modicum of Trepidation
There are 120 spindles downwind of this….drumroll please….
$ asmcmd ASMCMD> ls SDATA1/ SDATA2_1/ SDATA2_2/ SDATA2_3/ ASMCMD> rm -fr * ASMCMD>
Eek!
In other news, I just learned that Greg Rahn is about to get his hands on a set of HP DL580 G5 (Tigerton goodies). I’d sure like to get Silly Little Benchmark numbers from that sweetheart of a system…wish in one hand, you-know-what in the other I suppose.
Data Center? We Don’t Want No Stinkin’ Data Center.
I’ve often wondered why companies really want to own their computers. HP has voiced the same rhetorical question by launching AIaaS.
Why Do They Put Those Data Centers There?
But Atlanta? Is power really that cheap in Atlanta? It must be all that cheap hydroelectric power down there.
What’s This Have To Do With Oracle?
This is an Oracle-related blog, but I can’t answer that question. I’m watching just like you all are.
The Irony
I find it very ironic that HP announced this hosting initiative at their Technology@work Conference in Barcelona Spain considering the other hot news about the HP DL785 which is an 8-socket “Barcelona” Socket-F based Server. I am happy to see Proliant 8-socket Servers.
More on the “Barcelona” Quad-Core.
NetApp. New Name, Same Stuff.
Network Appliance has renamed their company a nickname. That just seems strange to me. Different name, same stuff.
And, yes, NFS works nicely for Oracle Database.
Maybe Oracle should be renamed/rebranded to “scott/tiger?”
Little Things Doth Crabby Make Part II and Introducing the Oracle Data Administrator.
Some rambling thoughts…
Oracle Database Administrator or Data Administrator?
Oracle continues to add features/functionality that makes the dutiful Database Administrator look a lot more like a Data Administrator. Some would argue it started with ASM since volume management was placed into the hands of the DBA. I’d argue it started with RMAN since that feature was one of the most significant, early features that took functionality away from the OS (and 3rd Party software) and put control in the hands of the DBA. After all, Oracle backups before RMAN had very little to do with Oracle (version 7 and earlier). I’d argue further that features like SecureFiles also make the Database Administrator more of a Data Administrator-or at least it seems it should make a DBA start to think like a “Data Administrator.” I think that trend will continue. I think that increases the value proposition of the Oracle DBA in the datacenter. I think it is a good thing.
DBAs might even find themselves performing lower-level storage operations such as creating OS-level logical drives. One such example would be HP Smart Array logical drives and the associated CLI is the hpacucli command on Linux.
I’ve been Meaning to Blog This One…
Quite some time ago I was fiddling with hpacucli. Yes, it was late at night. Yes, I was tired and crabby. Why would what’s happened in the following screen output make me crabby?
# hpacucli Array Configuration Utility CLI 7.15.19.0 Detecting Controllers...Done. Type "help" for a list of supported commands. Type "exit" to close the console. => ctrl all show status Error: Expecting “controller” in target. Type “help” for more information. => ctrl all show status Smart Array P400 in Slot 1 Controller Status: OK Cache Status: Not Configured Battery Status: OK
… the CLI doesn’t parse out leading whitespaces.
Maybe somebody, someday will be battling the hpacucli CLI and google their way to this post.
P.S., For goodness sake, please don’t go running to your local HP rep to tell him I’m bashing HP. Yes, HP Smart Array is great technology. I just don’t think whitespaces are as significant as hpacucli does 
Little Things Doth Crabby Make Part I. Enterprise Linux 5/RHEL5 Output Format Change for the iostat Command
Yes, I admit, sometimes little things make me crabby. Or is it that being crabby makes me easily irritated at small things? I think it’s the former rather than the latter.
Angry Email Makes Me Crabby
I am surprised at how many emails I’ve received from angry Storage Administrators about my position against Fibre Channel. I generally just ask them to not take it so personally. After all, there is more to storage than whatever storage networking protocol any particular server is plumbed with. Anyone who reads my Manly Man series about Fibre Channel will see that I’m just not a big fan of the current misapplication of Fibre Channel technology. As I point out in installment number two in the Manly Man series, Fibre Channel was originally intended to solve the difficulties associated with connecting hundreds (or thousands) of low capacity drives to a small number of large servers. These days it is being used to provide storage connectivity for hundreds of servers to a couple of huge FC storage arrays. And these hundreds of servers have both Ethernet and FC connectivity-overkill, plain and simple.
A lot is changing in the storage world where Oracle is concerned. I hope to come out of the foxhole to blog more about it over the next few months. Well, maybe I should say a few months from now. In the meantime I’ll be making some datacenter visits. I always learn something interesting when I get a chance to do that.
Little Changes Make Me Crabby
Like, for instance, the following example of the change to iostat -x output from RHEL4 to RHEL5. Notice how the rkB/s and wkB/s columns are missing? Yes, it is simple math to divide [rw]sec/s by 2 to get the same data, but there must be at least 42 of you folks that are feeding iostat -x output through some text processing, right? Time to adjust your scripts. Or is it? I wonder how many folks are really diving into OEL5/RHEL5? It seems a conservative IT shop with a functional RHEL4 configuration would most likely stay put. Any readers care to voice their experience with OEL5/RHEL5 adoption? Is it happening? Are folks waiting for their 11g adoption with plans to team that up with a move to OEL5/RHEL5?
# uname -r
2.6.18-53.ELsmp
# iostat -x 1 1
Linux 2.6.18-53.el5 (host1) 03/07/2008
avg-cpu: %user %nice %system %iowait %steal %idle
0.08 0.00 0.04 0.03 0.00 99.85
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.16 10.38 0.33 0.95 10.55 90.64 79.47 0.02 14.12 0.86 0.11
sda 0.00 0.00 0.00 0.00 0.01 0.00 21.05 0.00 3.18 2.87 0.00
dm-0 0.00 0.00 0.45 11.33 10.47 90.64 8.58 0.39 32.67 0.09 0.11
dm-1 0.00 0.00 0.00 0.00 0.01 0.00 8.00 0.00 5.05 0.51 0.00
cciss/c0d1 0.00 0.00 0.00 0.00 0.01 0.00 10.12 0.00 4.56 2.63 0.00
cciss/c0d2 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 4.30 2.52 0.00
cciss/c0d3 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 6.17 3.35 0.00
cciss/c0d4 0.00 0.00 0.00 0.00 0.01 0.00 10.38 0.00 5.50 3.22 0.00
cciss/c0d5 0.00 0.00 0.00 0.00 0.01 0.00 10.64 0.00 3.38 2.59 0.00
cciss/c0d6 0.00 0.00 0.00 0.00 0.01 0.00 10.64 0.00 4.05 3.31 0.00
cciss/c0d7 0.00 0.00 0.00 0.00 0.01 0.00 10.12 0.00 4.44 2.29 0.00
$ uname -r
2.6.9-34.ELsmp
$ iostat -x 1 1
Linux 2.6.9-34.ELsmp (host2) 03/07/2008
avg-cpu: %user %nice %sys %iowait %idle
1.79 0.00 1.28 0.30 96.63
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.06 11.83 0.14 3.21 23.07 120.39 11.53 60.19 42.77 0.68 201.33 4.60 1.54
External Tools for Oracle Data Bulk Unloading
Jonah H Harris has provided a good list of external data unloading tools that people use with Oracle. It is worth a gander at:
I first heard about Jonah’s blog via OakTable Network email from Tanel Poder. I’ve been reading Jonah’s blog and it looks like good stuff so far.
A Good Resource: Morgan’s Library.
…just a quick blog entry to plug Daniel Morgan’s (of Puget Sound Oracle User Group) very useful collection of scripts and other goodies:
Proof-Positive: Memory is Faster Than Disk. Don’t Need No Book Learnin’ to Cipher That One.
I’ve been reading a lot of blogosphere content about Data Warehousing these days. I’ve taken a lot of interest in such technology as Netezza, GreenPlum, DATAllegro and others and blog reading proves to be an interesting way to augment one’s knowledge. Who’d have thought I’d learn so much about OLTP through this reading.
Memory is Faster than Disk, So Let’s Do a Complete Rewrite
Why, just today I found out that it is time for a total rewrite of commercial RDBMS products. Uh huh. More interestingly, though, I learned:
- Memory is faster than disk. Really, truely, it is!
- A dual-core (2.8GHz) server with 4GB memory and 4 250GB SATA drives can perform 51,000 TpmC
- Disabling transaction logging entirely in a commercial RDBMS will increase throughput (TpmC) about three-fold
I found these pearls of wisdom while reading a Stonebraker paper referred to on this blog post. Yes, I know that blog is basically a store-front for Vertica, but I like to learn about different things that are going on in database technology. Unfortunately this time I was wasting my time. The URL in that blog post points to the VLDB front page, but a little sleuthing found the paper posted here: The End of an Architectural Era (It’s Time for a Complete Rewrite).
Recite after me:
If you get two orders of magnitude performance gain, you are either not doing it or you’ve moved it closer to the processor.
Dang, and I ain’t even got no too pretty good pedigree. Pshaw, I dasn’t fidget ‘mungst the quality!
Central versus De-centralized versus Shared-Nothing
No, it isn’t time for a re-write, especially one that requires a complete shared-nothing database approach. Now don’t get me wrong, I’m all for de-coupling and grid architecture-most particularly where storage is concerned. If I hear of another poor production site that is head-saturated on a $500,000 storage array when driving a measly 15 or so 15K RPM drives, I’ll BAARF. Please see the following post for what I’m talking about:
My Blog Posts Prove Oracle Doesn’t Support NFS!
In my post called Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, I linked to a paper Oracle maintains that explains how to use an NFS export from a small Unix/Linux server as storage for a third voting disk in a stretch RAC cluster. I pointed out that the paper instructs on how to use the noac mount option for Linux RAC clusters in spite of the many resources that suggest actimeo=0 will do. The authors of the document are standing fast that if you are building a Linux-based RAC stretch cluster and are using an NFS mount as a third voting device you do indeed need to mount that particular NFS filesystem with noac. That nugget of truth contradicts so many different documents that I don’t care to list. Instead, I’ll list a resource from Metalink that helps clarify the issue. In fact, I would say that no matter what the sundry Installation Guides or Release Notes say, refer to Metalink 359515.1 when the topic of Oracle Database 10g on NFS filesystems comes up.
Datafiles or CRS Files? The Mount Options Differ.
Metalink 359515.1 is a really helpful note. It spells out the RAC-related mount options for 10gR2 on Solaris, AIX, HP-UX and Linux. Most importantly, it spells out the options for the datafiles and the CRS files in two separate columns. Lo and behold, Metalink 359515.1 clearly spells out that noac is needed for CRS files, but not for datafiles.
In the comment section of Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, a reader points out that the Third Party Vote on NFS paper has a dead end URL (http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html) in the section that aimed to point out the fact that you cannot just use some Unix/Linux server NFS exports for any other purpose than this unique third voting disk setup in a stretch cluster scenario. He is right, that URL is a dead end, but I’d rather point folks to Linux RTCM (RAC Technology Certification Matrix) or the Unix RTCM, both of which clearly spell out a list of supported NFS File servers. Missing from the list is, of course, some plain old Linux or Unix server dishing out NFS exports-because the only supported application of simple, Unix/Linux NFS exports is the third voting disk scenario in a stretch cluster.
The reader also added this comment:
Many people say Oracle doesn’t support NFS, we need to verify. Searching oracle.com, “We did not find any search results for: vendors_nfs.html” and the references from google all seem to point at that one mysteriously missing doc.
Gee whiz. Where to start? Yes, for the eleventeenth time, NFS filesystems are supported for Oracle Database (including RAC). Let’s not get so easily confused; NFS is a protocol and the storage is NAS. Let’s all enter the following formula in our decoder rings before reading Oracle documents:
(Some Stupid Little Linux/Unix Server Exporting Filesystems via NFS) != NAS
The only supported application of non-NAS NFS is described in the following paper: Using NFS for a Third CRS Voting Device
Now, for some light reading about Oracle on NFS with 11g, I submit:
Oracle Database 11g on NFS filesystems:
Using Network Attached Storage or NFS File Systems Installation Guide for Microsoft Windows
Using Network Attached Storage or NFS File Systems Installation Guide for HP-UX
Using Network Attached Storage or NFS File Systems Installation Guide for AIX 5L Based Systems (64-Bit)
Using Network Attached Storage or NFS File Systems Installation Guide for Solaris Operating System
Using Network Attached Storage or NFS File Systems Installation Guide for Linux
And, of course: Configuring Direct NFS Storage for Datafiles
But, let’s not forget:
- Manly Men Only Deploy Oracle with Fibre Channel Storage - Part I. Oracle Over NFS is Weird.
- Manly Men Only Deploy Oracle with Fibre Channel - Part II. What’s So Simple and Inexpensive About NFS for Oracle?
- Manly Men Only Deploy Oracle with Fibre Channel - Part III. Did I Hear EMC Say NAS?
- Manly Men Only Deploy Oracle with Fibre Channel - Part IV. SANs are Simple, RAC is Difficult!
- Manly Men Only Deploy Oracle with Fibre Channel - Part V. What About Oracle9i on RHAS 2.1? Yippie!
- Manly Men Only Deploy Oracle with Fibre Channel - Part VI. Introducing Oracle11g Direct NFS!
- Manly Men Only Deploy Oracle with Fibre Channel - Part VII. A Very Helpful Step-by-step Install Guide for RAC on NFS.
- Manly Men Only Deploy Oracle With Fibre Channel - Part VIII. After All, Oracle Doesn’t Support Asynchronous I/O On NFS!
Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!
The topic of “stretch clusters” has been interesting to a lot of folks for quite some time. A stretch cluster is one where one or more cluster nodes, one of more portions of the SAN or both are geographically remote. Geographically remote could be within eye-sight (1-2km) or a long distance away. YottaYotta (Robin Harris of StorageMojo.com will notice that name) reached out to me (with hardware to offer) several years ago to set up a 3500km stretch cluster with three 10gR2 RAC nodes. Two of the RAC nodes were co-located and the third was put at 3500km distance using communications hardware that simulates the latency imposed by such great distance. And, yes, it is a valid simulation. It was an interesting exercise and with the YottaYotta distributed block server, the PolyServe (HP) and RAC were totally oblivious to the topology. It was a cool project, but that technology has had a difficult time catching on. In the interim, mainstream vendors have stepped up to offer stretch clustering technology and in the name of business continuity, folks are considering these sorts of solutions-but they are expensive. To that end, most shops would tend to buy, at most, a two-legged SAN. Therein lies the problem. Such a configuration could suffer a disaster on the leg of the SAN that has the majority of the CRS voting disks resulting in a total outage of the solution.
The remedy for this problem is to implement a third leg of storage for more voting disks to ensure an n+1 majority are available, but at what cost? The solution is to implement an inexpensive NFS share in which to host these additional voting disks. And, yes, you can use a simple low end Unix/Linux host as the NFS server for this purpose-so long as the host is running Solaris, AIX or HP-UX, or Linux. The following is a link to a paper that covers Oracle’s recommended/supported approach to this solution with Oracle Database 10g Release 2.
Using NFS for a Third CRS Voting Device
The paper is clear about the fact that using some plain Unix/Linux server to host NFS shares for Oracle files is limited to this specific purpose:
Oracle does NOT support standard NFS for any files, with the one specific exception documented in this white paper.
The paper appears to have a small contradiction about mount options-specifically stating that the noac option is required for Linux (see Figure 1) servers which seems to contradict Metalink 279393.1. I’ve sent an email to the authors about that. We’ll see if it changes.
Those Oracle Installs Just Keep Getting More and More Difficult
In my recent rant about Oracle database installation difficulties, I provided a link to a video in which fellow OakTable Network members Morten Egan and Mogens Norgaard captured how difficult the task really is.
Well, they’re at it again. You’ll see Morten “The Nose” Egan start out this new video taped Oracle installation by configuring a SAN with what looks like the HP Array Configuration Utility, but then my eyes are getting as bad as my blogging frequency. I couldn’t miss the Windows Disk Manager though-not even on fast forward.
I think we should start calling him Morten “The Hair” Egan. The link to the video follows:
Unconventional Oracle Installation Part II
