Feed aggregator

Fast-Start Failover - It is reliable

Mihajlo Tekic - Sun, 2008-08-10 20:45
Last Friday there was one post on OTN forums that brought my attention. The OP was wondering “Is Data Guard Buggy” with attention to Fast-Start Failover (FSFO) feature that provides an ability of automatic failover to the standby database if the primary database is not available for certain time.

He had some concerns about FSFO being unreliable, very difficult to be implemented and actually doesn't work properly.

The OP got some immediate response from the OTN users. I couldn't agree more with Joseph Meeks's comment on the topic.

In my opinion, FSFO is very nice feature that plays big role in Oracle's Maximum Availability Architecture. There might be some valid reasons not to implemented it, but if the automatic failover is a requirement, FSFO is the way to go. Should one have any problems implementing it, the best way to go is to get Oracle Support involved.

In this post, I'd like to show that implementation of the FSFO should not be a difficult task once you have the Data Guard environment set up.

I configured data guard physical standby environment on my laptop. Due to hardware limitations, I'll have the observer running on the same machine with the standby databases. Keep in mind this is just an example. In practice, the primary database, the standby database and the observer should run on different hosts. This example also answers one of the questions OP asked: Will it be possible to set it up on one machine? The answer would be it is possible, as shown in the example :-), but it is not the right way to go.

I use DGMGRL utility in the example..

So, I configured a Data Guard environment where DB1.MYDOMAIN.COM is primary database and STDB.MYDOMAIN.COM is physical standby. Both databases are 10.2.x
By default, protection mode of the Data Guard configuration is set to MAXIMUM PERFORMANCE

DGMGRL> connect sys/***@db1
DGMGRL> show configuration

Name: DRTest
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
db1 - Primary database
stdb - Physical standby database

Current status for "DRTest":

Enabling Fast-Start Failover requires the following pre-requisites to be met:
Flashback Database feature is enabled on both the primary and the standby database.
The protection mode of the configuration must be set to MAXIMUM AVAILABILITY
tnsnames.ora in the ORACLE_HOME where the observer runs must be set to see both databases, the primary and the standby.
DGMGRL must be available on the observer host.

Enable Flashback Database
I will enable flashback database feature on both databases. This assumes that the flash recovery area is configured.
The flashback database feature provides an ability for an easy reinstatement of the failed primary database to new standby database.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.

Total System Global Area 285212672 bytes
Fixed Size 1261348 bytes
Variable Size 100663516 bytes
Database Buffers 176160768 bytes
Redo Buffers 7127040 bytes
Database mounted.
SQL> alter database flashback on;

Database altered.

SQL> alter database open;

Database altered.

Set the protection mode to MAXIMUM AVAILABILITY
The Fast-Start Failover can be enabled only if the protection mode is set to MAXIMUM AVAILABILITY.
LogXptMode has to be set to 'SYNC'

DGMGRL> edit database 'db1'
> set property LogXptMode='SYNC';
Property "logxptmode" updated
DGMGRL> edit database 'stdb'
> set property LogXptMode='SYNC';
Property "logxptmode" updated

DGMGRL> edit configuration set protection mode as MaxAvailability;
DGMGRL> show configuration

Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: DISABLED
db1 - Primary database
stdb - Physical standby database

Current status for "DRTest":

SQL> conn sys/***@db1 as sysdba
SQL> select protection_mode, protection_level from v$database;

-------------------- --------------------

SQL> conn sys/***@stdb as sysdba
SQL> select protection_mode, protection_level from v$database;

-------------------- --------------------


I ensured that tnsnames.ora are set correctly as well DGMGRL is installed.

Now, since all the prerequisites are met, lets move forward and enable the FSFO.
Before enabling it, make sure each of the databases in the configuration has set a fast start failover target. This is achieved by setting the FastStartFailoverTarget parameter.

DGMGRL> edit database 'db1' set property faststartfailovertarget='stdb';
Property "faststartfailovertarget" updated
DGMGRL> edit database 'stdb' set property faststartfailovertarget='db1';
Property "faststartfailovertarget" updated

Another important parameter that has to be set is FastStartFailoverThreshold. This parameter specifies the amount of time (in seconds) the observers attempts to reconnect to the primary database before starting the fast-start failover to the standby database. The default value is set to 30 seconds. In the example I set this parameter to 120 seconds.

DGMGRL> edit configuration set property FastStartFailoverThreshold=120;
Property "faststartfailoverthreshold" updated

Now lets enable the fast start failover:


If you check the configuration at this moment you may find the following warning: ORA-16608: one or more databases have warnings.
If you check the status of one of the databases, you can see this warning: ORA-16819: Fast-Start Failover observer not started

DGMGRL> show configuration

Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
db1 - Primary database
stdb - Physical standby database
- Fast-Start Failover target

Current status for "DRTest":
Warning: ORA-16608: one or more databases have warnings

DGMGRL> show database verbose 'db1'

Name: db1
Enabled: YES
Intended State: ONLINE

InitialConnectIdentifier = 'db1.mydomain.com'
LogXptMode = 'SYNC'
Dependency = ''
DelayMins = '0'
Binding = 'OPTIONAL'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '300'
NetTimeout = '180'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'auto'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '2'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = 'stdb, db1'
LogFileNameConvert = 'stdb, db1'
FastStartFailoverTarget = 'stdb'
StatusReport = '(monitor)'
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
HostName = 'miki-laptop'
SidName = 'db1'
LocalListenerAddress = '(ADDRESS=(PROTOCOL=tcp)(HOST=miki-laptop)(PORT=1521))'
StandbyArchiveLocation = 'dgsby_db1'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = '%t_%s_%r.dbf'
LatestLog = '(monitor)'
TopWaitEvents = '(monitor)'

Current status for "db1":
Warning: ORA-16819: Fast-Start Failover observer not started

So lets start the observer. I will repeat again, the observer should run on a different host, however for the sake of this example it will run on the same machine as the databases.
In order to start the observer one should start DGMGRL utility and login to the data guard configuration.
Once logged in, issue START OBSERVER command. This will start the observer.
Optionally you can set a log file destination while invoking DGMGRL utility and specify name for the observer configuration file (for more information check Data Guard Command-Line Interface Reference).
Once the observer is started, the control is not returned to the user until the observer is stopped.

DGMGRL for Linux: Version - Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/***@db1
DGMGRL> start observer;
Observer started

So lets check the configuration now.

$ dgmgrl
DGMGRL for Linux: Version - Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/***@db1
DGMGRL> show configuration verbose;

Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
db1 - Primary database
stdb - Physical standby database
- Fast-Start Failover target

Fast-Start Failover
Threshold: 120 seconds
Observer: miki-laptop

Current status for "DRTest":


As of this moment my configuration has FSFO enabled.

Now lets test if the FSFO really works.
One should be aware of the conditions that must be satisfied for the observer to attempt FSFO.
The full list of conditions can be found in Data Guard Broker user guide under What Happens When Fast-Start Failover and the Observer Are Running?

I will issue SHUTDOWN ABORT to the primary database (SHUTDOWN NORMAL/IMMEDIATE/TRANSACTIONAL would not trigger the failover).
Once I crash the database, the status of the configuration will return Error: ORA-16625: cannot reach the database. Be aware that since the primary database is down, the only way to check the configuration is to connect using the standby database credentials.

SQL> conn sys/***@db1 as sysdba
SQL> shutdown abort
ORACLE instance shut down.

DGMGRL> connect sys/***@stdb
DGMGRL> show configuration

Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
db1 - Primary database
stdb - Physical standby database
- Fast-Start Failover target

Current status for "DRTest":
Error: ORA-16625: cannot reach the database

After waiting for two minutes (FSFO threshold was set to 120 seconds), I checked the observer log file and found out that it started the failover.

[W000 08/09 17:58:49.75] Observer started.

18:05:38.31 Saturday, August 09, 2008
Initiating fast-start failover to database "stdb"...
Performing failover NOW, please wait...
Failover succeeded, new primary is "stdb"
18:06:16.82 Saturday, August 09, 2008

DGMGRL> show configuration verbose


Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED
db1 - Physical standby database (disabled)
- Fast-Start Failover target
stdb - Primary database

Fast-Start Failover

Threshold: 120 seconds
Observer: miki-laptop

Current status for "DRTest":

Warning: ORA-16608: one or more databases have warnings


So the observer started the fast-start failover and has successfully converted the former standby to new primary database.

Once the failover is done the observer will attempt to automatically reinstate the former primary database to new physical standby. Certain requirements have to be met for the automatic reinstatement to take place. If not, the database can be manually reinstated. Once the reinstatement is successful you can see that the former primary database became new physical standby.

DGMGRL> show configuration


Name: DRTest
Enabled: YES
Protection Mode: MaxAvailability
Fast-Start Failover: ENABLED

db1 - Physical standby database
- Fast-Start Failover target
stdb - Primary database

Current status for "DRTest":



Finally, you can do switchover to perform role change and to place the configuration in its initial state.

The example above shows that enabling fast-start failover is pretty straight forward. There are many good documents with information how to enable FSFO feature. Some of them are listed below:

Oracle® Data Guard Broker 10g Release 2 (10.2) - 5.5 Fast-Start Failover
What Happens When Fast-Start Failover and the Observer Are Running?

7.6 Scenario 5: Enabling Fast-Start Failover and the Observer


Oracle Maximum Availability Architecture – MAA

I hope I didn't miss something.


APEX with eBusiness Suite survey reminder

Anthony Rayner - Thu, 2008-08-07 09:58
Readers of the APEX blogosphere will already know, we are currently conducting a survey in an attempt to better understand how people are using Application Express with Oracle eBusiness Suite / Oracle Applications. For example, have you implemented a killer reporting module with our 'Interactive Reports' to empower your users to get better information from their eBusiness Suite application data? Have you had other integration successes?

If so, this is just a quick reminder to anyone who hasn't yet completed David Peake's single page survey, please if you can take a moment to fill this in and tell us your thoughts, we would much appreciate it. Information provided is completely confidential, unless you state otherwise.

Thank you!
Categories: Development

How to Configure OEM 10g Database Control after Clone

Aviad Elbaz - Thu, 2008-08-07 05:32

After cloning an EBS environment I had tried to start the OEM 10g Database Control (DBConsole) and I got this message:

[oratest@testsrv1 ~]$ emctl start dbconsole
TZ set to Israel
OC4J Configuration issue.
/oracle/test/db/testdb/10.2/ oc4j/j2ee/OC4J_DBConsole_testsrv1.domain_TEST not found.

However, it was an acceptable message since DB Console wasn't configured to this instance yet.

Using emca I've tried to recreate the DB Control configuration files as follow:

[oratest@testsrv1 ~]$ emca -config dbcontrol db

INFO: Database Control started successfully
INFO: >>>>>>>>>>> The Database Control URL is http://testsrv1.domain:5500/em <<<<<<<<<<<
Enterprise Manager configuration completed successfully

Well, I thought it looks fine, but when I opened the DB Console I saw this message at the top of the page: 

java.lang.Exception: Exception in sending Request :: null

After some research, logs digging and Metalink searches, I found that the Agent component should be relinked manually.
But before there are two issues in env_sysman.mk file that should be fixed before executing the relink command.

I will show my actions step by step:

  1. Stop DB Console:
    emctl stop dbconsole
  2. cd $ORACLE_HOME/sysman/lib
  3. Edit env_sysman.mk and look for "JRE_LIB_DIR"
    In my instance JRE_LIB_DIR was populated with the old environment name..
    I changed it from:
  4. One more issue for env_sysman.mk, look for "$(LIBNMEM)  $(LIBNMER)"
    And change from:
    $(LIBNMEM)  $(LIBNMER) \
  5. Now relink the Agent component by issuing this command:
    make -f ins_sysman.mk agent
  6. Recreate the DB Control configuration files (Again, even if you did it before)
    emca -config dbcontrol db

Following this procedure will make the DBConsole configured and work for the new cloned instance.

Categories: APPS Blogs

A really recommended ASM patch – failing lun

Geert De Paep - Wed, 2008-08-06 10:22

The following is a real life experience about failing disks/luns and how ASM reacts to this. We used on Solaris with a HDS storage box and MPXio. We made an ASM diskgroup of 2 mirrorred disks. Then we made the luns unavailable to the hosts (hide lun). The result was not really what we expected.After hiding the lun, the ASM alert file shows the following ‘normal’ messages:

15:26:57+: Errors in file /app/oracle/admin/+ASM/bdump/+asm1_gmon_10611.trc:
15:26:57+: ORA-27091: unable to queue I/O
15:26:57+: ORA-27072: File I/O error
15:26:57+: SVR4 Error: 5: I/O error
15:26:57+: Additional information: 4
15:26:57+: Additional information: 2048
15:26:57+: Additional information: -1

and some time later

15:32:23+: WARNING: offlining disk 0.4042332304 (ARESDATAA) with mask 0x3
15:32:23+: WARNING: offlining disk 0.4042332304 (ARESDATAA) with mask 0x3


15:32:33+: WARNING: kfk failed to open a disk[/dev/oracle/asm/aresdata-a]
15:32:33+: Errors in file /app/oracle/admin/+ASM/udump/+asm1_ora_18313.trc:
15:32:33+: ORA-15025: could not open disk ‘/dev/oracle/asm/aresdata-a’
15:32:33+: ORA-27041: unable to open file
15:32:33+: SVR4 Error: 5: I/O error
15:32:33+: Additional information: 3
15:32:33+: NOTE: PST update: grp = 1, dsk = 0, mode = 0x6
15:32:35+: NOTE: PST update: grp = 1, dsk = 0, mode = 0x4
15:32:35+: NOTE: group ARESDATA: relocated PST to: disk 0001 (PST copy 0)
15:32:35+: NOTE: PST update: grp = 1, dsk = 0, mode = 0x4
15:32:35+: NOTE: cache closing disk 0 of grp 1: ARESDATAA
15:32:35+: NOTE: cache closing disk 0 of grp 1: ARESDATAA
15:33:50+: WARNING: PST-initiated drop disk 1(780265568).0(4042332304) (ARESDATAA)
15:33:50+: NOTE: PST update: grp = 1
15:33:50+: NOTE: group ARESDATA: relocated PST to: disk 0001 (PST copy 0)
15:33:50+: NOTE: requesting all-instance membership refresh for group=1
15:33:50+: NOTE: membership refresh pending for group 1/0x2e81e860 (ARESDATA)
15:33:50+: SUCCESS: refreshed membership for 1/0x2e81e860 (ARESDATA)
15:33:53+: SUCCESS: PST-initiated disk drop completed
15:33:53+: SUCCESS: PST-initiated disk drop completed
15:33:56+: NOTE: starting rebalance of group 1/0x2e81e860 (ARESDATA) at power 1
15:33:56+: Starting background process ARB0
15:33:56+: ARB0 started with pid=21, OS id=19285
15:33:56+: NOTE: assigning ARB0 to group 1/0x2e81e860 (ARESDATA)
15:33:56+: NOTE: F1X0 copy 1 relocating from 0:2 to 1:2
15:33:56+: NOTE: F1X0 copy 2 relocating from 1:2 to 0:2
15:33:56+: NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 65534:4294967294
15:33:56+: NOTE: X->S down convert bast on F1B3 bastCount=2

15:34:14+: NOTE: group ARESDATA: relocated PST to: disk 0001 (PST copy 0)
15:34:14+: WARNING: offline disk number 0 has references (1394 AUs)
15:34:14+: NOTE: PST update: grp = 1
15:34:14+: NOTE: group ARESDATA: relocated PST to: disk 0001 (PST copy 0)

However, every time we do a query on the v$asm_disk view, we experience hangs of 30..90 seconds. Same when adding a tablespace in a database. It looks as if ASM, whenever trying to access the failed disk, waits for OS timeouts. I assume that every operation that needs access to ASM disks (e.g. data file autoextend that needs to allocate space in the asm disk, creation of archivelog, …) suffers from these timeout. Not really acceptable for a production environment. I do want to mention that dd, and even orion for asynch IO, detect the error immediately without waiting for any timeout.

You can clearly see the ASM waits when you truss the server process of the asm sqlplus session when you do a select on v$asm_disk. For each failed disk you get:

9377/1: 17.3106 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 18.3195 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 18.3198 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 19.3295 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 19.3298 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 20.3395 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 20.3398 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 21.3495 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 21.3497 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 22.3595 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 22.3598 open(“/dev/oracle/asm/aresdata-a”, O_RDWR|O_DSYNC) Err#5 EIO
9377/1: 22.3605 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 23.3695 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 23.3697 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 24.3795 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 24.3798 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 25.3895 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 25.3897 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 26.3995 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 26.3998 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 27.4095 nanosleep(0xFFFFFFFF7FFF8FE0, 0xFFFFFFFF7FFF8FD0) = 0
9377/1: 27.4097 open(“/dev/oracle/asm/aresdata-a”, O_RDONLY|O_DSYNC) Err#5 EIO
9377/1: 27.4105 write(5, ” * * * 2 0 0 8 – 0 7 -“.., 27) = 27
9377/1: 27.4106 write(5, “\n”, 1) = 1
9377/1: 27.4109 write(5, ” W A R N I N G : k f k”.., 62) = 62
9377/1: 27.4109 write(5, “\n”, 1) = 1
9377/1: 27.4111 close(6) = 0
9377/1: 27.4111 open(“/app/oracle/admin/+ASM/bdump/alert_+ASM2.log”, O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0660) = 6
9377/1: 27.4118 time() = 1217417698
9377/1: 27.4119 writev(6, 0xFFFFFFFF7FFF8080, 3) = 88

So it tries to access each disk 6 times in read write mode and 6 times again in read only mode. A loss of 12 valuable seconds….

At the same time, the os messages file generates the following messages every second:

Jul 30 11:01:05 node1 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g60060e800562f400000062f4000000d5 (ssd63):
Jul 30 11:01:05 node1 offline or reservation conflict

Jul 30 11:01:06 node1 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/ssd@g60060e800562f400000062f4000000d5 (ssd
Jul 30 11:01:06 node1 offline or reservation conflict

We would expect that ASM is intelligent enough to detect that the disk failed, but obviously it keeps trying to access it including the waits and timeouts.
FYI, the state of the disks after the failure has become:

DISKGROUP  PATH                            Total   Used  %Usd ST  Header    FAILGROUP   STATE      DiskName
---------- ------------------------------ ------ ------ ----- --- --------- ----------- ---------- ---------------
ARESDATA   /dev/oracle/asm/aresdata-b      46068   1396     3 ONL MEMBER    B           MOUNTED    ARESDATAB

ARESDATA                                   46068   1396     3 OFF UNKNOWN   A           MOUNTED    ARESDATAA

I opened an SR on Metalink and I uploaded all possible traces I could generate (*). And guess what, due to some reason (maybe (*)), I immediately came to an excellent engineer who identified the problem immediately as a known bug, and asked development to provide a patch for (which did not exist yet at that time). It took only 5 days for the patch to be available, and that patch solves the problem completely. After applying it, every select on v$asm_disk returns immediately.

This is it:

Patch 6278034
Product RDBMS Server
Select a Release
Platform: Sun Solaris SPARC (64-bit)
Last Updated 04-AUG-2008
Size 97K (99336 bytes)
Classification General

The patch exist as well for I would recommend to install it on your oracle_home where asm runs. However I have no idea if the problem is applicable as well to non-Solaris environments.

Note: To resync the disks after the lun is available again, use the ALTER DISKGROUP ADD FAILGROUP x DISK ‘/dev/path/somedevice’ NAME some_new_name [FORCE] command. Not so straightforward, it turns out that trying to offline or drop the disk will not work. I.e.:

==================== OVERVIEW OF ASM DISKS ======================================

———- —————————— —— —— —– — ——— ———– ———- ————— ——-
ARESARCHA /dev/oracle/asm/aresarch-a 30708 11226 36 ONL MEMBER AREAARCHA NORMAL AREAARCHA CACHED
ARESARCHB /dev/oracle/asm/aresarch-b 30708 11202 36 ONL MEMBER ARESARCHB NORMAL ARESARCHB CACHED
ARESDATA /dev/oracle/asm/aresdata-a 46068 1412 3 ONL MEMBER A NORMAL ARESDATAA CACHED


Trying to add it with the same name as before:
SQL> alter diskgroup ARESDATA add failgroup B disk ‘/dev/oracle/asm/aresdata-b’ name ARESDATAB force;
alter diskgroup ARESDATA add failgroup B disk ‘/dev/oracle/asm/aresdata-b’ name ARESDATAB force
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15010: name is already used by an existing ASM disk

Adding it using a new name:

SQL> alter diskgroup ARESDATA add failgroup B disk ‘/dev/oracle/asm/aresdata-b’ name ARESDATAB2 force;
alter diskgroup ARESDATA add failgroup B disk ‘/dev/oracle/asm/aresdata-b’ name ARESDATAB2 force
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15034: disk ‘/dev/oracle/asm/aresdata-b’ does not require the FORCE option

SQL> alter diskgroup ARESDATA add failgroup B disk ‘/dev/oracle/asm/aresdata-b’ name ARESDATAB2;

Diskgroup altered.

I assume that the need to use the ‘force’ option depends on the kind of error you got.

Sometimes I see that the rebalance does not start automatically. Then you get the following status:
———- —————————— —— —— —– — ——— ———– ———- ————— ——-
ARESARCHA /dev/oracle/asm/aresarch-a 30708 11322 36 ONL MEMBER AREAARCHA NORMAL AREAARCHA CACHED
ARESARCHB /dev/oracle/asm/aresarch-b 30708 11298 36 ONL MEMBER ARESARCHB NORMAL ARESARCHB CACHED
ARESDATA /dev/oracle/asm/aresdata-a 46068 1412 3 ONL MEMBER A NORMAL ARESDATAA CACHED
/dev/oracle/asm/aresdata-b 46068 2 0 ONL MEMBER B NORMAL ARESDATAB2 CACHED


In that case, start it manually using:

SQL> alter diskgroup ARESDATA rebalance;

Diskgroup altered.

How I got picked for special attention in Denver International Airport

Moans Nogood - Wed, 2008-08-06 09:06
So my wife Anette and I are on our way home from Tim's wedding.

We flew British Airways both ways. In Copenhagen I told a lady at the BA check-in counter that I might have discovered a way for terrorists to put bombs on planes without being on the plane themselves.

You see, the last couple of times where I have had to change terminals in Heathrow and there's been approximately two hours or less between the flights my bags haven't made it.

So if the bags are onboard a plane but the passenger doesn't show up, they'll pull the bags. But if the bags are delayed they'll let the passenger fly without his luggage.

Everyone knows there have been huge problems with luggage in Heathrow. At one point there was more than 42,000 pieces stacked up. IBM had stopped a DW/BI project without having created even indexes on the Oracle database tables, so every piece of luggage required a full table scan of a rather large table, so it took a while to get over that one.

So I told the lady at the checkin in Kastrup airport, Copenhagen, that there might be a security risk in Heathrow and she said she would relay the information.

Well, apparently she did, because the checkin guy in Denver yesterday suddenly started behaving very strange, went into the backoffice to "do a security check" and marked our boarding cars with the dreaded "ssss" code highlighted in yellow which means "pay very special attention", and which meant that both Anette and I had to go through the new machine that will blow air on you so that it can smell traces of explosives, etc etc.

We're currently in Heathrow, about to board for Copenhagen. I wonder if our bags will make it.

So much for trying to warn the folks about a security problem :-))).


Index block split bug in 9i

Yasin Baskan - Wed, 2008-08-06 02:18
In his famous index internals presentation Richard Foote mentions a bug in 9i about index block splits when rows are inserted in the order of the index columns. Depending on when you commit your inserts the index size changes dramatically.

While I was trying to find out why a 3-column primary key index takes more space than its table I recalled that bug and it turned out that was the reason of the space issue. The related bug is 3196414 and it is fixed in 10G.

Here is the test case Richard presents in his paper.

SQL> create table t(id number,value varchar2(10));

Table created.

SQL> create index t_ind on t(id);

Index created.

SQL> @mystat split

------------------------------ ----------
leaf node splits 0
leaf node 90-10 splits 0
branch node splits 0

SQL> ed
Wrote file afiedt.buf

1 begin
2 for i in 1..10000 loop
3 insert into t values(i,'test');
4 commit;
5 end loop;
6* end;
SQL> r
1 begin
2 for i in 1..10000 loop
3 insert into t values(i,'test');
4 commit;
5 end loop;
6* end;

PL/SQL procedure successfully completed.

SQL> @mystat2 split

------------------------------ ---------- ----------
leaf node splits 35 35
leaf node 90-10 splits 0 0
branch node splits 0 0

SQL> analyze index t_ind validate structure;

Index analyzed.

SQL> select lf_blks, pct_used from index_stats;

---------- ----------
36 51

SQL> drop table t;

Table dropped.

I am trying to insert the rows in the order of the primary key column, so what I expect to see is that when an index block fills there will be a 90-10 split and the index will grow in size. But as the number of leaf block splits show there are 35 block splits and none of them are 90-10 splits meaning all are 50-50 block splits. I have 36 leaf blocks but half of each one is empty.

If we try the same inserts but commit after the loop the result changes.

SQL> create table t(id number,value varchar2(10));

Table created.

SQL> create index t_ind on t(id);

Index created.

SQL> @mystat split

------------------------------ ----------
leaf node splits 35
leaf node 90-10 splits 0
branch node splits 0

SQL> ed
Wrote file afiedt.buf

1 begin
2 for i in 1..10000 loop
3 insert into t values(i,'test');
4 end loop;
5 commit;
6* end;
SQL> r
1 begin
2 for i in 1..10000 loop
3 insert into t values(i,'test');
4 end loop;
5 commit;
6* end;

PL/SQL procedure successfully completed.

SQL> @mystat2 split

------------------------------ ---------- ----------
leaf node splits 53 53
leaf node 90-10 splits 18 18
branch node splits 0 0

SQL> analyze index t_ind validate structure;

Index analyzed.

SQL> select lf_blks, pct_used from index_stats;

---------- ----------
19 94

In this case we see that there have been 18 block splits and all were 90-10 splits as expected. We have 19 leaf blocks and all are nearly full. Depending on where the commit is we can get an index twice the size it has to be. When I ran the same test in 10G it did not matter where the commit was. I got 19 leaf blocks in both cases.

I did not test if this problem happens when several sessions insert a single row and commit just like in an OLTP system but I think it is likely because we have indexes showing this behavior in OLTP systems.

Lost Pictures - Recovered Pictures

Pawel Barut - Mon, 2008-08-04 16:09
Written by Paweł Barut
This weekend I've spend great time with family, and we make lot of pictures. But when I've arrived home, and plugged camera to PC, it looked like this:
All pictures seems gone ...
For us computer geeks data lost is not an option. I've started to look for tools that can help me recover those data. In such situation rule number one is: not allow to write anything on this storage. Any additional write could lead to total data lost. I've googl`ed for software that could help. I've found lot of software that does "FAT Recovery", "Recover Digital Camera", "Data Recovery" - most of them priced 49-200$. Well, not big money, but still quite many when you need to recover 50 pictures. And you never know, if this software is worth it, and will solve your problem.
Finally I've found really great software that I would like to recommend: PC INSPECTOR™ smart recovery. It run very fast, and recovered almost all pictures. When looking into Camera counter and number of files, only one picture is missing. This software is freeware, but authors wants us to "Pay What You Want". And I've did it, as this software did his job perfectly.
Hope it help someone else to recover lost pictures.


Related Articles on Paweł Barut blog:
    Categories: Development

    JavaFX Preview SDK

    Oracle EPM Smart Space - Mon, 2008-08-04 15:11

    In previous posts I have attempted to show how RIA’s (Rich Internet Applications) can be used for delivering EPM and BI data to end users.  Until now I have focused mostly on Microsoft’s Silverlight and have gotten some grief.  Well  now I have another option; JavaFX has launched their preview SDK and it is time to see how it compares and what the user experience will be like. 

    For this test I plan to see If I can create some Essbase ‘gadgets’ that can be used in the browser and then dragged to the desktop (cool feature shown in a JavaFX video).  Comment if you have ideas and stay tuned for posts on my progress.

    Categories: Development

    Tim is getting married...

    Moans Nogood - Sat, 2008-08-02 10:33
    Anette and I are in Denver, Colorado these days, because Tim Gorman is getting married to Lori tonight (Saturday). It's a hot wedding: This is, I think, the 20th day in a row with over 90 degrees Fahrenheit, which makes it the hottest period since 1857 or something like that.

    Tim is very well known in the Oracle community: He spent many years inside Oracle where I had the pleasure of communicating a lot with him on the wonderful HELPKERN list there.

    He also wrote a couple of books and contributed to the Tales of The OakTable book. Here's his website: http://www.evdbt.com/

    Good luck to Lori & Tim! (said the guy on his third marriage...)

    An unusual cause of ORA-12154

    Jared Still - Fri, 2008-08-01 17:30
    The ORA-12154 (and its cousin ORA-12514) have been the bane of many a novice DBA.

    This error is essentially telling you that you have messed up your database naming configuration, whether it be tnsnames, oracle names or OID. The fix is normally quite simple - just correct the naming.

    This is usually quite easily done with netmgr, a tool that is fairly good at its job. The syntax for setting up a tnsname is a little convoluted, and I've fallen back on netmgr a number of times when I can't seem to get it just right by typing the entries in the file.

    There's at least one other way to cause ORA-12154 to pop up and consume more time than you may like to admit. I won't tell how long I played with this...

    The cause is actually due to security policy. While the characters !@#$%^&*()_-=+~` are not normally allowed in Oracle passwords, it is actually quite simple to include them. Simply enclose the password in double quotes.
    alter user scott identified by "th!sat^st";

    This was done on a number of accounts on our databases, all in the name of security.
    These types of passwords have been used without issue for some time now.

    Today was a notable exception.

    After spending some time fiddling with tnsnames.ora files, I realized what the problem actually was.

    Suppose you were to have a password with an '@' in it? Yep, that was the problem.
    First let's see what it looks like from within sqlplus:

    15:41:52 SQL> alter user scott identified by "what@mistake";

    User altered.

    15:42:03 SQL> connect scott/"what@mistake";

    scott SQL> select user from dual;


    SQL> connect scott/what@mistake
    ORA-12154: TNS:could not resolve the connect identifier specified

    As long as the connect string was in double quotes, no problem.

    Now let's try it from a linux command line:

    Linux> sqlplus scott/'what@mistake'

    SQL*Plus: Release - Production on Fri Aug 1 15:42:20 2008

    ORA-12154: TNS:could not resolve the connect identifier specified

    Linux> sqlplus scott/'what\@mistake'

    SQL*Plus: Release - Production on Fri Aug 1 15:46:14 2008

    ORA-01017: invalid username/password; logon denied

    Linux> sqlplus scott/"what\@mistake"

    SQL*Plus: Release - Production on Fri Aug 1 15:46:21 2008

    Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

    ORA-01017: invalid username/password; logon denied

    Linux> sqlplus 'scott/what\@mistake'

    SQL*Plus: Release - Production on Fri Aug 1 15:47:23 2008

    Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

    ORA-01017: invalid username/password; logon denied

    Linux > sqlplus scott/what\@mistake

    SQL*Plus: Release - Production on Fri Aug 1 15:48:52 2008

    Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

    ORA-12154: TNS:could not resolve the connect identifier specified

    There doesn't seem to be any method to use a password with the @ character in it, at least not from sqlplus. The same password works fine when used on the command line with perl script that logs on to the same database and account:

    Linux> DBI_template.pl -database dv11 -username scott -password 'what@mistake'

    The 'X' is the correct output as this script simply selects * from dual.

    Lesson learned, do not ever, under any circumstances, use the @ character in the password for an Oracle database account.

    You might think that goes without saying, but it's actually pretty easy to get a password with '@' in it when the passwords are generated random characters.

    FWIW, this problem was manifested in Windows as well
    Categories: DBA Blogs

    TGI g_friday

    Oracle WTF - Fri, 2008-08-01 05:57

    Found in a package body:

    g_friday CONSTANT VARCHAR2(6) := 'Friday';

    ...then a couple of hundred lines later:

    if to_char(business_date,'fmDay') = g_friday then
        ...end-of-week processing...
        ...regular processing...
    end if;

    Now that's flexible. If end-of-week processing is ever moved to the weekend, all you have to do is set g_friday := 'Saturday'.

    OTN members, don't change your e-mail account!

    Yasin Baskan - Fri, 2008-08-01 03:44
    I am regular user of OTN and its forums. Last week I was trying to login to OTN from a public computer and I got the "invalid login" error everytime I tried. I was sure I was typing my password correct but I could not get in anyway. So, I tried to get my password reset and sent to my e-mail address. Then I remembered that the e-mail address I used to register for OTN was from my previous employer meaning I did not have access to it anymore. As OTN does not allow changing the registration e-mail address I was stuck. I send a request from OTN to get my password delivered to my current e-mail address. Here is the reply I got:
    Resolution: Oracle's membership management system does not currently support
    the editing of the email address or username in your membership profile.
    (It will support this capability in a future release.)
    Please create a new account with the new email address you wish to use. However,
    it is possible to change the email at which you receive
    Discussion Forum "watch" emails (see "Your Control Panel" when logged in).
    They tell me to create a new user and forget about my history, watch list, everything. What a user centric approach this is.

    If you are an OTN member do not lose your password and your e-mail account at the same time, you will not find anybody from OTN who is willing to solve your problem and help you to recover your password.

    I am used to bad behavior and unwillingness to solve problems in Metalink, now I get the same behavior in OTN. Whatever, just wanted to let you know about it.

    Metalink, SCM, the error and the good stuff

    Mihajlo Tekic - Thu, 2008-07-31 23:45
    Yesterday I got an e-mail from Oracle Support letting me know about the new Metalink interface which is supposed to go live this Fall:

    Dear MetaLink Customer,

    Oracle is committed to consistently improving your customer support experience. In the fall of 2008, MetaLink will have a new user interface. To help you prepare for the transition, you may now preview MetaLink's new user interface and provide valuable feedback about its features.


    I clicked on the link, that was supposed to redirect me to Metalink, and I got one very fancy schmancy login page. Well, actually, it is the Software Configuration Manager (SCM).

    I put in my login credentials and ... I got an error message "IO Error Error #2032"

    WOW ... :-)

    Well, OK the problem was fixed latter that day.

    Regardless of the problem I experienced, I must say that I've been using SCM for quite a while and I am pretty impressed with its functionality. It is really much easier to create and manage service requests using the configurations you I have registered with SCM. Oracle Support engineers have all the information they need about the configuration of the server and the database.

    Searching the Knowledge base looks improved as well. Now you can have your search results visible on the left panel of the screen, while, at the same time, you can read the content of the selected note. This makes navigation much easier.

    Service Requests part has new design too.

    All in all, improved functionality, better navigation, good design, some new features too ... Good job !!!

    Useful Links:


    Make it go away! (rman delete)

    Claudia Zeiler - Thu, 2008-07-31 20:11
    So much documentation tells you how to set up things for Oracle. It is harder to find information on how to get rid of it once you have it.

    I am doing perfectly lovely incremental compressed backupsets of my database. Then one day, playing with EM, I in experimented with the 'Oracle recommended backup stategy'. I may have missed something in my set up, but I ended up with an un needed image copy backup of my database. Help! This thing is devouring my flashback area. So what is the proper way to remove it.

    First step, of course, is to kill the Oracle job, so it doesn't do that again!

    Then in rman:
    RMAN> list copy;

    gives me a nice long list - one line of which was:
    List of Datafile Copies
    Key File S Completion Time Ckp SCN Ckp Time Name

    30428 4 A 29-JUL-08 224478 29-JUL-08 /bkup/flashbk/datafile/o1_mf_users_44y8x123_.dbf

    I double check on the file system:
    $> ls o1_mf_users_48y8x123_.dbf
    Yes, the copy is there.

    To delete it in RMAN I enter
    RMAN> delete datafilecopy 30428;

    and get:

    List of Datafile Copies
    Key File S Completion Time Ckp SCN Ckp Time Name

    30428 4 A 29-JUL-08 224478 29-JUL-08 /bkup/flashbk/datafile/o1_mf_users_44y8x123_.dbf

    Do you really want to delete the above objects (enter YES or NO)? y
    deleted datafile copy
    datafile copy filename=/bkup/flashbk//datafile/o1_mf_users_44y8x123_.dbf recid=30428 stamp=1334123
    Deleted 1 objects

    I check in the file system and this time my ls returns
    ls: 0653-341 The file
    o1_mf_users_44y8x123_.dbf does not exist.

    Good bye, nusance.


    Mary Ann Davidson - Wed, 2008-07-30 07:06

    This summer Idaho has had the loveliest profusion of wildflowers I’ve ever seen, the product of a healthy snow pack, full reservoirs and a late spring. Happily enough, many wildflowers have seeded themselves in my rock garden, which is far more diverse and healthy than is the case with whatever else is planted that is not coming up because I have a black thumb. (I’ve actually thought about planting weeds and hoping invasive flowers take over. A girl can dream.)

    I also have excellent early warning systems in my backyard in Idaho. Specifically, the critters I support on my property are all – individually and collectively – quite good at alerting me when Something Is Happening. Birds, pine squirrels (more on them later) and – last but not least – my dog Thunder are all very good alarm system proxies. It took me a couple of years living away from large urban enclaves to learn how to “read” nature’s cues. Now, my ears have been retrained to the point that I listen to the birds, squirrels and my dog when they are trying to tell me something. I claim no special nature skills but I like to think that family genes (my grandfather was and my father is a consummate woodsman after years of hunting) are asserting themselves.

    When I sit out in my backyard and hear the “chit-chit-chit-chit” of a pine squirrel, I know that it means “intruder at twelve o’clock.” Pine squirrels are really noisy, and thus very good at telling you when somebody or something is coming, at least 6 trees away from the action (and yes, I can tell the difference between pine squirrel alarms and pine squirrel pickup lines). The birds also get noisier, and in a different way, when there is a (fox, dog, cat, coyote, other) prowling through the sage brush that I can’t see, but I know is there because the birds have gone to Defcon 4. Thunder also has entirely different barks for “someone’s coming up the driveway I know,” “someone’s coming up the driveway I don’t know” and “a fox just ran across the porch and is hightailing it for the back yard.” The prize for alarm specificity goes to my sister’s miniature schnauzer Sneakers, whose bark (in increasing order of frenzy) refers to: a) a jogger b) a squirrel c) a fox d) the neighbor’s white dog e) deer or f) lots of deer.

    My other “tenants” (the family of white-throated swifts that nests under my peaked roof) don’t warn of “incoming,” but they keep pests out of my yard. Late afternoon, there are eight to ten of them in aerial dogfights with any flying insects that darken my airspace. Watching the sparrows turn, bank, and maneuver is just about as big a thrill as watching the Blue Angels. I like to grab a glass of wine at the end of the workday, go outside and watch the swifts on evening pest patrol. It’s very soothing and lends new meaning to the phrase, “running the debugger.”

    One of the things I have been doing some thinking and speaking about is the idea of synthesis. More specifically, the lessons we can learn in IT security from other disciplines, such as business, economics, history (especially military history and strategy) and biology. I confess that I felt a little nervous speaking on this topic at a university recently, because I figured any one of the professors or graduate students on the audience knew more than I did about IT security – certainly on the nerd level. On the other hand, they are all in the perfect environment to think differently about their profession via synthesis: all they have to do is walk across the quad to talk to another department. In fact, a professor of biology I met said that at her university, there was a tight synthesis between the computer science and biology departments. Each department had realized that they were kissin’ cousins, so to speak.

    Of course, we IT security weenies know this intuitively. We speak of computer “viruses” because they “infect” vulnerable hosts unless the host has been “inoculated” against them. Some of the research going on focuses on making hosts just different enough that viruses are not able to infect all of them. Mirroring the arms race that biological hosts and opportunistic germs engage in, virus makers try to find ways to defeat anti-virus defenses by disguising their nasty, germy little packages so they aren’t recognized by the defense systems – just like you can’t be inoculated against the common cold because there are so many slightly different rhinoviruses, as I know all too well because I have spent two weeks and then some getting rid of a particularly rotten summer cold. And, just as in biology, computer viruses do not want to kill the host, but to use it.

    A few years ago, there was an interesting paper positing that a software monoculture was a national security risk. That is, a lack of “biological diversity” in enterprises makes those enterprises more vulnerable to a cyber plague that affects the entire enterprise, not just a portion of it (just like the Irish potato famine wiped out millions of people because the strain of potatoes grown in Ireland was not resistant to the potato blight). Note that there is some happy medium here. If it is true that running only one kind of software may make the enterprise more susceptible to a cyber plague, it’s also true that running one of every type of application, database, operating system, and so on is neither economical nor easily secured, as one would have to be an expert in absolutely everything to manage such a system.

    We know that biological entities use trickery to survive, thrive and propagate. Moths disguise themselves as other, more toxic moths to fool predatory birds. (What is a honey pot but a technical equivalent of a biological system designed to attract predators?)

    I have read a couple of fascinating books on how companies are modifying plants to be resistant to some diseases. This is not without risk or without controversy. The University of Hawai’i, for example, just implemented a five-year ban on genetic modification of kalo (taro), in part, because for Hawaiians, kalo is not just a plant but part of their culture. I also note that genetic modification does not necessarily deliver all the promises claimed by the proponents (e.g., the so-called “golden rice,” genetically engineered to have Vitamin A in it, doesn’t have enough in it to do much good. More specifically, according to one book I read, you’d have to eat 12 pounds of the rice a day to get the minimum daily requirement and who eats 12 pounds of rice a day?)

    I’ve had the same discussions over products that claim “native protection” against classes of attacks (like SQL injection – which I believe is doable) and that do “virtual patching” (which I don’t believe all the claims for). For those who are not up on “virtual patching,” it is the idea that you can replicate in a gatekeeper/cyber-Doberman function the exact equivalent of what a patch does. You can’t. You can (in some cases) have a good workaround, or you can prevent a specific exploit or exploits, which may buy customers needed time to patch. That is very useful, I agree. Unfortunately, “virtual patch” as a term is indiscriminate: “preventing known exploits” is more accurate but doesn’t reel in the gullible, so we have “virtual patching” as an industry term and not “can’t replace patching but gives you some protection, maybe, so might be worth a shot.” To my point, shilling “virtual patching” as a replacement for patching is as irresponsible and potentially harmful to customers as parents skipping inoculations for DPT is to their children: someone, some time is going to get hit by something horrible.

    As I look at my backyard, I wonder what bright technoid will look at a white-throated swift and think, “I can build that. I can build a cyber patrolling predator so swift (no pun intended) and agile that it can dive bomb pests before they reach my cyberbackyard.” Instead of staying on the telephone wire and hoping a pest drives by (like static defenses people deploy now), the cyber swifts could circulate freely on perpetual pest patrol. I think about early warning systems as sophisticated, yet recognizable as my sister’s Schnauzer or the neighborhood pine squirrels. One frenzied bark or one “chit-chit-chit” and I have a pretty good idea what is out there and how worried I should be about it. I wish most of the cyber defenses we had now were as good, as recognizable, as accurate and descriptive. Of course, foxes, coyotes and cats aren’t constantly changing their guise to be unrecognizable to Neighborhood Crime Stopper Pine Squirrels, either.

    There are other disciplines that have applicability to the world of IT security, if we choose to explore them. For example, when I was in graduate business school, one of the financial market theories I learned pertained to whether companies should diversify given that investors can do it themselves. For example, conglomerates (companies that have a lot of diverse, not-necessarily-complementary lines of business), the theory goes, are not necessarily valued correctly by the marketplace. And in fact, since investors can diversify their own investments (by buying, say, automobile stock and pharmaceutical company stock separately, if that’s what they want to own), there is no reason – per se – for conglomerates to have multiple lines of disparate businesses. The big idea then (and now to a certain extent) is to focus on core competencies (we see this today in discussions about outsourcing or software as a service: if IT is not a core competency, why do it yourself?)

    A number of these business trends/theories, for better or worse and sometimes both, are extended to the global marketplace. For example, the idea that if they can produce sugar more cheaply in Foobaria, then the Snafu Republic should not subsidize their domestic sugar farmers but should happily import sugar from Foobaria. Over time, the Snafu Republic’s farmers will find something else to grow that they can grow better, cheaper or faster than Foobaria (or another country). (Note: You may be less enthused about this idea if you are a sugar farmer* in Foobaria than a policy wonk in Foobaria, because no policy wonk’s job has ever moved overseas that I know of.)

    Another argument, more along the lines of industrial policy, is that the people of the Snafu Republic – instead of being subsistence farmers, barely eking out enough food to feed their families – should go work in factories or someplace that will give them a higher wage so they can buy food (and more besides). In a happy dappy world, everyone (or every country) will focus on his or its core competencies and outsource everything else. Globalization facilitates everyone doing what he does best and the rising tide lifts all economies.

    I am not here to argue for or against globalization as a general policy or construct (it’s a lot more complicated than one can describe in a blog entry and I think it is dangerous to reduce complex ideas to sound bytes). But I do note that there are a number of interesting – if disturbing – discussions taking place recently about the limits of globalization as a result of spiraling food prices. Food prices, of course, are spiraling for a number of reasons: increased transportation costs, the “crowding out” effect of biofuels, higher demand for high quality food as a result of growing economies, crop failures in some key areas, and so on.

    Some countries have acted to ban exports of key staples (rice, for example), wanting to ensure that they can feed their own people. As a result, have-not countries are potentially rethinking that policy that said “get the subsistence farmers into higher wage jobs,” because at least a subsistence farmer might have been able to feed his own family. If you can no longer import food because exporters hoard it, you can’t always eat what the factory is producing unless they are refining sugar. You can eat potato chips but not microchips.

    In short, we’ve recently had a lesson that the theory of “everyone (read “every country”) does what it does best, and we all trade for what ever else we want” does not necessarily work when you have a shock to the system, like the transportation costs going through the roof, a result of which is that sugar schlepped from Foobaria is now really, really expensive to Snafuians. It also assumes that no country is ever going to use exports as a competitive weapon. Not only is that assumption a bigger stretch than most economists typically posit (“investors are rational” – they aren’t – otherwise how we do explain how breakfast cereal portal companies got funded in the DotCom days?), but we know from history it is not true. It’s never been true, in fact.

    The second mistake a lot of policy wonks make is assuming peace, love and happiness in perpetuity. That’s not true, either. Natural resources such as food water, minerals, spices (yes, spices – salt and cloves being two that immediately come to mind – the British empire enforced a monopoly on salt within their empire, and the Portuguese dominated the spice trade for years) are often used as competitive weapons and the fight over them causes wars. Japan (prior to World War II) felt that they could never be a great empire without controlling their own supply of key resources and a proximate trigger of the Pacific War was the US cutting off the supply of scrap metal to Japan. Japan did not go on a territory-acquiring binge just to have more places for rice paddies, but to acquire natural resources that went with the territory. (And ultimately they lost the war because the US destroyed so much of their merchant shipping that they could no longer ship oil to where they needed it – their ships and planes.)

    What’s the security issue? The security issue is that people need to think about their supply chain when formulating national security policies. Where are food, water, energy, spare parts, computer software and hardware coming from? Are any of those critical to national security, to the point where we need multiple suppliers or a “home grown” supplier because it is in one’s national security interests to do so? (For example, the Defense Science Board looked at this issue in relation to having a Trusted Foundry Program – domestic suppliers of integrated circuits for critical defense applications.) Do we actually trust non-domestic suppliers? (News flash: yes, other nation states would, too act to put malware or backdoors in software. A shock, I know, but some countries do act to advance their national interests at the expense of – gasp, horror – other nations. Been going on as long as recorded history.)

    We should assume that this is happening and deal with it instead of worrying about Hurting Other Country’s Feelings by calling them on it (the international relations equivalent of telling a country We Are On To You, Knock That &^^%$ Off Right This Minute). I recently participated in a meeting where the debate was whether the group should issue guidance on how to protect your electronics (e.g., cel phone, laptop) when you travel overseas from being co-opted by Bad Guys (bad guys here could be bad guys working for the foreign government). The guidance was all good guidance and not aimed at any country in particular, but the discussion devolved to topics as diverse as “shouldn’t the State Department be the one issuing this guidance?” and “what are the political issues around upsetting some country or another if this guidance goes out?”

    (It almost boggles the mind. We know this is happening, so why are people worried about making any country already engaged in industrial espionage, breaking into critical infrastructure and so on Feel Bad About It? It’s like wondering if the grizzly bear had a bad childhood as he is gnawing on your leg. Do I really care if you were an unwanted cub? Stop chewing on my leg!)

    In short, the theory of competitive advantage as applied to nation-states sounds great on paper, and may even work great to a point, but it does not take national security needs into account. A nation that is dependent upon others for key materials – like spare parts for their aircraft or microchips or food – can easily be at the mercy of others unless they have an alternate supply (and in fact, a secure supply).

    I am not advocating buying everything from inside one country or (getting back to a corporate example) avoiding outsourcing at all cost. Rather, the issue is that while you can outsource services and offshore production/services/sourcing, you can't outsource risk. Even financial markets tell us that you can diversify some kinds of risks, but not market risk – the risk that the entire market will tank. For example, I “outsource” medical care in that I go to see a doctor regularly since I am not an MD. However, I have a responsibility to take care of myself (e.g., to avoid high risk behaviors that are potentially damaging to my health like excessive drinking, using illegal drugs or abusing legal ones). I can’t outsource that risk and I can’t pass along 100% of my health responsibility to a doctor.

    Accordingly, whether you are a company looking at service or product providers, or a nation-state contemplating industrial policy, you need to consider risk with steely-eyed objectivity and act appropriately. You could even say that, while there is no one easy set of answers, a non-exhaustive list of potential solutions includes: thinking about country of origin in light of political, social and economic factors, as well as the state of law and law enforcement in the country, using proven suppliers; keeping better handles on your supply chain; keeping attuned to political and governmental actions in countries where you operate; and so on. Hoping geopolitical or business conditions never change, and that everyone you deal with in business has the ethics of the Boy Scouts is not risk management or even optimism, it’s fantasy.

    I have had many occasions recently to recount – as a cautionary tale – the story of Wake Island’s defenders in December 1941, one of many fine moments in the history of the US Marine Corps. The Marines managed to sink a Japanese ship from a shore battery (yes, really) but ultimately, the Japanese prevailed. Among other ironies, where did the metal come from for the armaments the Japanese used to shell the shore installations on Wake Island? Scrap metal the US had sold to Japan. If we need reminding, the lesson is that you should never, ever, ever arm your enemies.

    * Yes, I realize you don’t actually grow sugar but something sugar is refined from, like sugar beets, sugar cane, even corn (high fructose corn syrup).

    For more information:

    Book(s) of the Week:

    The Omnivore’s Dilemma
    is one of the most thoughtful and thought-provoking books about food, where it comes from and the implications of how your food is grown. It will change the way you look at what’s on your plate. It’s well researched and yet deeply personal. The second, The Botany of Desire, is really fascinating look at four plants and their impact on the world. The ethical implications of “licensing plants” alone are worth the read (yes, the potato is one of the four plants).

    You can find both of them and other works by Michael Pollan at:


    A great book on the defense of Wake Island is Given Up For Dead:


    A book on salt that includes a discussion of the British empire’s inter-empire monopoly on salt: Salt: A World History by Mark Kurlansky:


    More on the Salt Tax:


    A book about the history of the spice trade (who would think nations could be so combative over cloves?) is The Scents of Eden: A History of the Spice Trade:


    A web site on Idaho birds:


    And a picture of the white-throated swift:



    About the Trusted Foundry Program:


    The original paper on software monoculture that created such a stir


    A really (really, really) good book on issues around genetic modification of food (it mentions the hubbub over kalo (taro)) is Uncertain Peril: Genetic Engineering and the Future of Seeds by Claire Hope Cummings:


    More on the genetic modification of kalo (taro):


    Absolutely nothing to do with any of the above topics, but a great video of one of my favorite Hawaiian groups (‘Ike Pono) doing one of my favorite songs (Ua Noho Au A Kupa). It is just really happy music:


    If that doesn’t make you want to hula, there is no hope for you.

    OK, and Bobby Moderow, Jr. of Moanalua doing "Koke’e" (which I just love):


    Using EJBContext from an EJB 3 Interceptor

    Debu Panda - Wed, 2008-07-30 03:44

    I got a question from a customer that he wants to use EJBContext from an EJB 3 Interceptor.

    Yes, it’s very simple. Just inject the EJBContext into the interceptor using @Resource injection.

    See the example code that uses methods of SessionContext as follows:

    package actionbazaar.buslogic;

    import javax.annotation.Resource;
    import javax.interceptor.AroundInvoke;
    import javax.interceptor.InvocationContext;

    public class CheckPermissionInterceptor {
    private javax.ejb.SessionContext ctx;

    public Object checkUserRole(InvocationContext ic) throws Exception {
    System.out.println("*** CheckPermission Interceptor invoked for "
    + ic.getTarget() + " ***");
    if (!ctx.isCallerInRole("admin")) {
    throw new SecurityException("User: '"
    + ctx.getCallerPrincipal().getName()
    + "' does not have permissions for method "
    + ic.getMethod());
    return ic.proceed();

    If you want the run-able version of the example code, you can download from http://manning.com/panda and look at Chapter 5 example.

    11g SQLNet Client trace files

    Fairlie Rego - Wed, 2008-07-30 03:14
    I spent half an hour wondering why I was not able to generate a SQL* Net client trace file from a 11g client to a 10.2 database.
    Here are the contents of my client sqlnet.ora

    bart:ELEV:/opt/oracle/product/11.1.0/db_1/network/admin> cat sqlnet.ora
    TRACE_FILE_CLIENT= cli.trc

    It took me a while to realize that thanks to ADR the client trace files were actually going to $ORACLE_BASE/diag/clients/user_oracle/host_xxx/trace

    The only way to disable this is to add the following in the sqlnet.ora

    After setting the above line in the sqlnet.ora the client trace files were generated in /tmp

    How to enable FND_HELP search by creating an index on FND_LOBS efficiently

    Aviad Elbaz - Tue, 2008-07-29 03:57

    Have you ever tried to use the search option of Online Help in Oracle Applications?
    Our users did... and they got "The page cannot be found" message...

    I checked it on firefox, hope to get more accurate message, and I got this:
    "Not Found. The requested URL /pls/DEV/fnd_help.search was not found on this server".


    I checked fnd_help package and it's compiled and looks fine.

    These errors appeared in error_log_pls:

    [Mon Jul 28 10:34:54 2008] [warn] mod_plsql: Stale Connection due to Oracle error 20000
    [Mon Jul 28 10:34:54 2008] [error] mod_plsql: /pls/DEV/fnd_help.search ORA-20000
    ORA-20000: Oracle Text error:
    DRG-10599: column is not indexed
    ORA-06512: at "SYS.DBMS_SYS_SQL", line 1209
    ORA-06512: at "SYS.DBMS_SQL", line 328
    ORA-06512: at "APPS.FND_HELP", line 1043
    ORA-06512: at "APPS.FND_HELP", line 873
    ORA-06512: at line 20

    I found note 306239.1 - "Cannot Search Online Help After Fresh Install of 11.5.10" which suggest that Applications interMedia Text indexes are corrupt. In my instance it even wasn't exist...

    I followed this note which instructs to:

    1. drop index FND_LOBS_CTX;
    2. Rebuild the index using aflobbld.sql

    But aflobbld.sql had been running for more than 10 hours and the size of DR$FND_LOBS_CTX$I table has reached to 35 GB !
    I had been wondering how it can be that fnd_lobs table is less than 1GB and the index on it is 35 GB and counting.... ?!

    Note 396803.1 - "FND_LOBS_CTX is having huge size, how to reduce the sizeof the index?" suggests it's a bug, indexing all documents in FND_LOB table, also the binary files, while using wrong filter.

    So how can we make aflobbld.sql to index only FND_HELP documents?

    For each row in FND_LOBS table the file_format column is populated with one of the following values: IGNORE, BINARY, TEXT.
    aflobbld.sql will index only rows that have this column set to BINARY or TEXT.
    If we set all rows to IGNORE except FND_HELP rows, we could index them only.

    Note 397757.1 - "How to Speed Up Index Creation on FND_LOBS by indexing Only FND_HELP Data" suggests the steps to do it.

    These are the steps:

    1. Backup the fnd_lobs table before updating it, we will use it later:
      create table fnd_lobs_bk as select * from fnd_lobs;
      ** you can create a backup of this table and omit the file_date column to make this backup faster
    2. Drop index FND_LOBS_CTX if exists:
      drop index applsys.FND_LOBS_CTX;
    3. Update all rows to IGNORE except FND_HELP rows:
      update fnd_lobs
      set file_format = 'IGNORE'
      where nvl(program_name,'@') <> 'FND_HELP' ;
    4. Execute aflobbld.sql from OS terminal:
      sqlplus apps/sppas @$FND_TOP/sql/aflobbld.sql applsys apps;
    5. Since I'm not sure about the impact of leaving the FND_LOBS rows as IGNORE, I updated them back to the previous state:
      create unique index fnd_lobs_bk_u1 on fnd_lobs_bk (file_id);
      update (select fl.file_format ffo,flb.file_format ffb
              from fnd_lobs fl
                  ,fnd_lobs_bk flb
              where fl.file_id = flb.file_id)
      set ffo=ffb;
      drop table fnd_lobs_bk;
    6. Check the search option.... it should work now.

    You are welcome to leave a comment .


    Categories: APPS Blogs

    My Sessions at Oracle Open World 2008

    Debu Panda - Tue, 2008-07-29 01:31

    I’ve two sessions at Oracle Open World 2008, San Francisco. Here are my session details:

    S298520 : Best Practices for Managing Your Oracle WebLogic Environment with Oracle Enterprise Manager
    Track: Application Server and Transaction Processing
    Room: Rm 2006
    Date: 2008-09-22
    Start Time: 11:30

    S298522 : Top Five Things DBAs Must Know About Managing Middleware
    Track: Application Server and Transaction Processing
    Room: Rm 2003
    Date: 2008-09-25
    Start Time: 15:00

    See you at San Francisco!

    One off patches, conflicts and merges

    Fairlie Rego - Mon, 2008-07-28 21:23
    So you are applying one of the zillion patches on top of and you hit the following error


    ApplySession applying interim patch '6338357' to OH
    Interim patch 6338357 has File Conflict with patch(es) [ 5399670 ]
    in OH /opt/oracle/product/10.2.0/db_1


    Wouldn't it be nice if you could check for conflicts before you apply a patch
    Here is one way to do the same

    1. You can check which source code file is being modified by doing the following

    cd 6338357/etc/config
    cat actions | grep .o
    oracle.rdbms version="" opt_req="R"
    archive name="libserver10.a" path="%ORACLE_HOME%/lib" object_name="lib/libserver10.a/kelt.o"
    make change_dir="%ORACLE_HOME%/rdbms/lib" make_file="ins_rdbms.mk" make_target="ioracle"

    2. Then go to the $ORACLE_HOME/.patch_storage and see if any of the previous patches have modified the same file

    bart:TEST:/opt/oracle/product/10.2.0/db_1/.patch_storage> find . -name 'kelt.o' -print

    So this clearly shows that Patch 5399670 was previously applied which modified the same source code file and you need a merge patch for both bugs (5399670 and 6338357)


    Subscribe to Oracle FAQ aggregator