Oracle-L: Re: AW: Tuning instance for batch processing

From: Mark Richard <mrichard_at_transurban.com.au>
Date: Sun, 16 Feb 2003 13:23:38 -0800
Message-ID: <F001.0054EF24.20030216132338@fatcity.com>

Stefan,

If you are doing a data conversion exercise then there may be some other things to consider:

Parallel loading into the staging tables - might not be significant if this is only a small portion of the conversion effort.
Consider options such as NOLOGGING - if you are creating temp tables, or even creating a final table then consider the NOLOGGING option. This can often half a queries execution time and if your recovery to that step failing would be "recreate the table from scratch" then you have nothing to lose.
If you are doing a lot of sorting / grouping then give yourself plenty of sort_area_size. If the machine has several GB of RAM then you don't want sorts going out to disk once they hit a few hundred MB. Since you'll have all of the ram to yourself there's no reason to not take all you can - as long as you stay in physical ram.
Determine when an index will help your conversion processing. If the index won't help your conversion, don't create it until you are finished. Sometimes you'll even want to create a temporary index to help your processing and then remove it again shortly after. Dropping indexes during data conversion and recreating them later has often saved me a considerable amount of time overall.

But to a degree you are right - there's no silver bullet, especially without understanding the data volumes, types of queries, etc. There are, however, a few settings like sort_area_size, nologging and parallel clauses which would be set differently for your environment than a multi-user system.

Regards,

Mark.

                                                                                                                   
                    Stefan Jahnke                                                                                  
                    <Stefan.Jahnke       To:     Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>       
                    @bov.de>             cc:                                                                       
                    Sent by:             Subject:     AW: Tuning instance for batch processing                     
                    root_at_fatcity.c                                                                                 
                    om                                                                                             
                                                                                                                   
                                                                                                                   
                    14/02/2003                                                                                     
                    20:43                                                                                          
                    Please respond                                                                                 
                    to ORACLE-L

Mark and Dennis

Thanks for the replies. I'm actually doing a data conversion (migrating mainframe data to Oracle). First step wil be loading all the files into a staging db using external tables. Then, a lot of insert/update jobs will run, and finally, we will do some queries / pl/sql stuff to check integrity (on a logical level). Additionally, the machine will be mine for a whole day, so I won't have to deal with side effects caused by other processes. Looks like I should go ahead, start a test with plain vanilla configuration,
run statspack, change some parameters or execute queries in parallel mode, run statspack and so on. And I think all your other points are also pretty important, like the use of bind variables, bulk inserts instead of loops commiting single rows, distribute i/o load etc.

But, it looks like it comes down to: "There is no silver bullet, just iterative testing" ?

Regards,

Stefan Jahnke
Consultant
BOV Aktiengesellschaft
Voice: +49 201 - 4513-298
Fax: +49 201 - 4513-149
mailto: stefan.jahnke_at_nospam.bov.de
Please remove nospam to contact me via email.

visit our website: http://www.bov.de
subscribe to our newsletter: http://www.bov.de/presse/newsletter.asp

Sicherheitsluecken mit IT-Security-Konzepten von BOV effizient schliessen! Weitere Informationen unter +49 201/45 13-240 oder E-Mail an mailto:andrea.palluck_at_bov.de.

Wie Sie wissen, koennen ueber das Internet versandte E-Mails leicht unter fremden Namen erstellt oder manipuliert werden. Aus diesem Grunde bitten wir um Verstaendnis dafuer, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erklaerungen und Aeusserungen ausschliessen.

As you are probably aware, e-mails sent via the Internet can easily be copied or manipulated by third parties. For this reason we would ask for your understanding that, for your own protection and ours, we must decline all legal responsibility for the validity of the statements and comments given above.

-----Ursprüngliche Nachricht-----

Von: Mark Richard [mailto:mrichard_at_transurban.com.au] Gesendet: Donnerstag, 13. Februar 2003 23:46 An: Multiple recipients of list ORACLE-L Betreff: Re: Tuning instance for batch processing

Stefan,

Are you looking at tuning in terms of Oracle parameters or rewriting SQL? I guess one of the first things to consider is: Will the server be doing anything else at the same time? If the batch jobs are the only thing running then you tuning should probably focus more on execution time than execution efficiency. For example, look at things like parallel queries to load the box and reduce overall execution time.

Other than that the usual rules apply:

Use trace and TKPROF to see what's happening.
Look at using bind variables in queries that are repeated often to reduce the parse count.
Look at processing a group of records in a single statement rather than using loops in program logic if at all possible.
Look at I/O - do you have the load balanced well across datafiles / disks.
If running on UNIX run sar or top or something similar to determine if the server is working hard. If not, perhaps you can run jobs in parallel.
Are you doing anything unnecessary (ie: executing a statement within a loop that will return the same value each time).

Even though #5 seems a little silly it can occaisionally work a treat. I was aked to tune a very ugly query once because a batch job was running for 2 hours. The query, whilst very ugly, ran in under 500ms. Looking at the program logic showed it was being called about 12,000 times though and was guaranteed to return the same value each time. Running it once and storing the result in a variable reduced the batch to under 5 minutes.

Regards,

Mark.

                    Stefan Jahnke

                    <Stefan.Jahnke       To:     Multiple recipients of

list
ORACLE-L <ORACLE-L_at_fatcity.com>

                    @bov.de>             cc:

                    Sent by:             Subject:     Tuning instance for
batch processing
                    root_at_fatcity.c

                    om





                    14/02/2003

                    01:49

                    Please respond

                    to ORACLE-L

Hi list

Does anyone know of any good white papers online (or maybe a Metalink Note ID, I couldn't really find the right thing) about tuning an Oracle instance for batch processing ?

Thanks,

Stefan Jahnke
Consultant
BOV Aktiengesellschaft
Voice: +49 201 - 4513-298
Fax: +49 201 - 4513-149
mailto: stefan.jahnke_at_nospam.bov.de
Please remove nospam to contact me via email.

visit our website: http://www.bov.de
subscribe to our newsletter: http://www.bov.de/presse/newsletter.asp

Sicherheitsluecken mit IT-Security-Konzepten von BOV effizient schliessen! Weitere Informationen unter +49 201/45 13-240 oder E-Mail an mailto:andrea.palluck_at_bov.de.

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: Stefan Jahnke
INET: Stefan.Jahnke_at_bov.de

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services


---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>

Privileged/Confidential information may be contained in this message.

          If you are not the addressee indicated in this message
       (or responsible for delivery of the message to such person),
            you may not copy or deliver this message to anyone.
In such case, you should destroy this message and kindly notify the sender
           by reply e-mail or by telephone on (61 3) 9612-6999.
   Please advise immediately if you or your employer does not consent to
                Internet e-mail for messages of this kind.
        Opinions, conclusions and other information in this message
              that do not relate to the official business of
                         Transurban City Link Ltd
         shall be understood as neither given nor endorsed by it.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: Mark Richard
INET: mrichard_at_transurban.com.au

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services


---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: Stefan Jahnke
INET: Stefan.Jahnke_at_bov.de

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services


---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
   Privileged/Confidential information may be contained in this message.
          If you are not the addressee indicated in this message
       (or responsible for delivery of the message to such person),
            you may not copy or deliver this message to anyone.
In such case, you should destroy this message and kindly notify the sender
           by reply e-mail or by telephone on (61 3) 9612-6999.
   Please advise immediately if you or your employer does not consent to
                Internet e-mail for messages of this kind.
        Opinions, conclusions and other information in this message
              that do not relate to the official business of
                         Transurban City Link Ltd
         shall be understood as neither given nor endorsed by it.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: Mark Richard
INET: mrichard_at_transurban.com.au

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services


---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message