Return-Path: Received: from ensim.rackshack.net (root@localhost) by orafaq.net (8.11.6/8.11.6) with ESMTP id h7UNfWJ19442 for ; Sat, 30 Aug 2003 18:41:32 -0500 X-ClientAddr: 66.27.56.210 Received: from ns3.fatcity.com (rrcs-west-66-27-56-210.biz.rr.com [66.27.56.210]) by ensim.rackshack.net (8.11.6/8.11.6) with ESMTP id h7UNfV319437 for ; Sat, 30 Aug 2003 18:41:31 -0500 Received: from ns3.fatcity.com (localhost.localdomain [127.0.0.1]) by ns3.fatcity.com (8.12.5/8.12.5) with ESMTP id h7ULA1GF012972 for ; Sat, 30 Aug 2003 14:10:15 -0700 Received: (from root@localhost) by ns3.fatcity.com (8.12.5/8.12.5/Submit) id h7UL0Sf2011291 for oracle-l@orafaq.net; Sat, 30 Aug 2003 14:00:29 -0700 Received: by fatcity.com (05-Jun-2003/v1.0g-b73/bab) via fatcity.com id 005CE3A6; Sat, 30 Aug 2003 13:59:25 -0800 Message-ID: Date: Sat, 30 Aug 2003 13:59:25 -0800 To: Multiple recipients of list ORACLE-L X-Comment: Oracle RDBMS Community Forum X-Sender: "Ryan" Sender: ml-errors@fatcity.com Reply-To: ORACLE-L@fatcity.com Errors-To: ML-ERRORS@fatcity.com From: "Ryan" Subject: Re: has anyone used threads to parallelize bulk loads? Organization: Fat City Network Services, San Diego, California X-ListServer: v1.0g, build 73; ListGuru (c) 1996-2003 Bruce A. Bergman Precedence: bulk Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_00E3_01C13174.B1CA0A20" ------=_NextPart_000_00E3_01C13174.B1CA0A20 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: has anyone used threads to parallelize bulk loads?we are still on 8i = at work. That stuff is useful in the future. what would you suggest for = parallel extraction to from a DB to files? I dont know perl and dont = have time to learn it right now. I can do it in C. yeah I know its easy, = but Im in school too and there are so many hours in a day.=20 how is unix redirection to a file for extraction? real slow? I always = get concerned about using C in a DB, since I have to consider whether = someone 2-3 years from now will be able to maintain it.=20 ----- Original Message -----=20 From: Tim Gorman=20 To: Multiple recipients of list ORACLE-L=20 Sent: Saturday, August 30, 2003 5:39 PM Subject: Re: has anyone used threads to parallelize bulk loads? Or forget about 3GL programming altogether and just query 9i external = tables in parallel and pass the results into an INSERT /*+ APPEND = PARALLEL */? Combine it all with PL/SQL pipelined table functions and = you have parallel ETL -- "extraction" (using 9i external tables), = "transformation" (using PL/SQL pipelined table functions), and "loading" = (using INSERT APPEND) -- in a single SQL statement. Also, don't forget = about 9i multi-table INSERT statements, conditional INSERT statements, = and of course the MERGE (a.k.a "up-sert") statements. There is a = tremendous amount of flexibility and new features around ETL built into = Oracle these days... I've been programming "C" for 20 years and there are very few reasons = to write another line of "C" code these days. I still find reasons, but = only rarely. And never related to ETL... on 8/30/03 1:54 PM, Ryan at rgaffuri@cox.net wrote: I find this useful when I have large loads with alot of small to = medium sized DML statements. Instead of running them serially I use = DBMS_JOB and break them up into pieces.=20 DBMS_JOB is limiting in that it can only do 36 at once per instance. = If I want to do more, Id have to do it outside the database. I have read = about unix 'pthreads'. These appear to have the same = wait,notify,synchronize interface that java has. Im also on Solaris. Has anyone done anything like this? Any features on Solaris that I = can use? I know solaris has some built-in low level C functions to = handle locking(such as a mutex script). Any idea of some sources I can = use to look this up?=20 I dont need this for work right now, but I may need it in the = future, so I want to play around with it. Or do I have to take this to a = unix sys admin forum? If so anyone know any good ones?=20 Threading interfaces seem to be very similiar across platforms. In = oracle you achieve the same thing with dbms_job,dbms_lock,and = dbms_alert.=20 anyone got any scripts or experience with this? This is not = essential. Im just playing around.=20 ------=_NextPart_000_00E3_01C13174.B1CA0A20 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Re: has anyone used threads to parallelize bulk = loads?
we are still on 8i at work. That stuff = is useful in=20 the future. what would you suggest for parallel extraction to from a DB = to=20 files? I dont know perl and dont have time to learn it right now. I can = do it in=20 C. yeah I know its easy, but Im in school too and there are so many = hours in a=20 day.
 
how is unix redirection to a file for = extraction?=20 real slow? I always get concerned about using C in a DB, since I have to = consider whether someone 2-3 years from now will be able to maintain it. =
----- Original Message -----
From:=20 Tim = Gorman=20
Sent: Saturday, August 30, 2003 = 5:39=20 PM
Subject: Re: has anyone used = threads to=20 parallelize bulk loads?

Or forget about 3GL programming = altogether and=20 just query 9i external tables in parallel and pass the results into an = INSERT=20 /*+ APPEND PARALLEL */?  Combine it all with PL/SQL pipelined = table=20 functions and you have parallel ETL --  =93extraction=94 (using = 9i external=20 tables), =93transformation=94 (using PL/SQL pipelined table = functions), and=20 =93loading=94 (using INSERT APPEND) -- in a single SQL statement. =  Also,=20 don=92t forget about 9i multi-table INSERT statements, conditional = INSERT=20 statements, and of course the MERGE (a.k.a =93up-sert=94) statements. =  There=20 is a tremendous amount of flexibility and new features around ETL = built into=20 Oracle these days...

I=92ve been programming =93C=94 for 20 = years and there=20 are very few reasons to write another line of =93C=94 code these days. =  I=20 still find reasons, but only rarely.  And never related to=20 ETL...



on 8/30/03 1:54 PM, Ryan at rgaffuri@cox.net=20 wrote:
I find this useful when = I have=20 large loads with alot of small to medium sized DML statements. = Instead of=20 running them serially I use DBMS_JOB and break them up into pieces.=20

DBMS_JOB is limiting in that it can = only do 36=20 at once per instance. If I want to do more, Id have to do it outside = the=20 database. I have read about unix 'pthreads'. These appear to have = the same=20 wait,notify,synchronize interface that java has. Im also on=20 Solaris.

Has anyone done anything like = this? Any=20 features on Solaris that I can use? I know solaris has some built-in = low=20 level C functions to handle locking(such as a mutex script). Any = idea of=20 some sources I can use to look this up?

I dont=20 need this for work right now, but I may need it in the future, so I = want to=20 play around with it. Or do I have to take this to a unix sys admin = forum? If=20 so anyone know any good ones?

Threading=20 interfaces seem to be very similiar across platforms. In oracle you = achieve=20 the same thing with dbms_job,dbms_lock,and dbms_alert. =

anyone got any scripts or experience with this? This is not = essential. Im just playing around. =


------=_NextPart_000_00E3_01C13174.B1CA0A20-- -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Ryan INET: rgaffuri@cox.net Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru@fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).