Oracle FAQ Your Portal to the Oracle Knowledge Grid

Home -> Community -> Mailing Lists -> Oracle-L -> RE : Recovery in a Replicated environment

RE : Recovery in a Replicated environment

From: Stephane Faroult <>
Date: Thu, 29 Jun 2000 11:16:55 +0200
Message-Id: <>


   One of my customers uses a home-made, near real-time master-to-master replication for security trading (just to say that when somebody goes wrong there is big money at stake). I have always find recovery in this type of configuration to be the hardest part. In case of crash, you switch to a replicated database which, by definition, is (more or less) up-to-date. Fine. Now the problem is that after you have repaired the crashed database, it is no longer in phase with the others and 'back-to-normal' is very difficult. In the initial design for my customer's replication, there used to be a transaction log table which had to be exported, re-imported into the recovered database and re-applied. There is now on this database an activity which is much much higher than what it was when it was designed, and the tests we have carried out have shown that the official recovery procedure was indeed (much) slower than rebuilding the database from scratch - just what we do when we add a new node to the network. As a result, for recovery what we do is that we wait until markets are closed at the site we want to restore plus the one we want to clone (increasingly difficult these days - there are still W/E, fortunately), download the really useful data (i.e. we forget about audit tables and the like which are not essential to business needs), 'tar' and 'gzip' it, transfer the file and reload everything at the other end with SQL*Loader and the direct path (the best way to minimize both file size - we have sites in America, Asia and Europe, it may have to travel a long way - and reloading times). We are lucky to have reasonably sized databases (a few Gigs) and if the whole system operates in 24x5 mode this is not true at the local level. An interesting side-effect of this strategy is that the most important sites for the business have a backup machine which is replicated as anything else, and that there is no other backup - in the worst case, we can get the data from another continent.   We also have a tool for resynchronizing 'live' data if need be, and well, all this has been in operation for three years now and we have survived the premature death of disk controllers in New-York and Zurich.


  Stephane Faroult
  Oriole Corporation
  Voice:  +44  (0) 7050-696-269 
  Fax:    +44  (0) 7050-696-449 
  Performance Tools & Free Scripts
------------------------------------------------------------------, designed by Oracle DBAs for Oracle DBAs

> Hi all.
> We are using multi-master replication using Oracle
> 8.1.6 running on Solaris 2.6. Although we are set up
> for multi-master replication, there will only be
> transaction activity at one site unless the primary
> site is lost, in which case we'll be switching to the
> secondary site.
> Does anyone have any experience performing recovery in
> this type of environment or know of any papers which
> outline what steps should be taken for performing
> recovery in a replicated environment using different
> recovery scenarios.
> I would appreciate any help I can get in this area as
> I have been having trouble finding any sources which
> address this with any level of detail.
Received on Thu Jun 29 2000 - 04:16:55 CDT

Original text of this message