Re: 10g: chaining pipelined functions is slow
Date: Sun, 10 Oct 2010 07:19:30 -0700 (PDT)
Message-ID: <ece3716c-b8b9-4056-a144-381162acb748_at_a19g2000vbi.googlegroups.com>
Am 10.10.2010 14:30, schrieb Randolf Geist:
> On Oct 10, 10:36 am, Frank Bergemann<FBergem..._at_web.de> wrote:
>> However i am still wondering, if oracle sets up separate processing
>> instances (what i called 'worker threads') for each stage of the
>> chained pipelined functions? (i.e. for each pipelined function)
>
> Correct, as far as I know Oracle doesn't support this kind of
> processing model out of the box. You can either have the whole chain
> processed by a serial execution (single thread) or use parallel
> processing to have the whole chain processed in multiple "partitions"
> where each process takes care of a part of the whole set - see the
> images that can be found when following the provided links to the
> documentation.
I will. To be honest, i am a bit disappointed by this. It screw up my approach to scale up for a multi-stage processing chain, dealing with different resources (tables) in the different processing stages. So i have to change it the way you described before and set up parallism by data partitioning.
> However you need to understand that the model that you
> seem to have in mind potentially suffers from the required inter-
> process communication that needs to stream the data between the
> different "worker threads" - the same issue that Oracle has when using
> its parallel processing model where a similar "producer - consumer"
> model is possible, however it only supports one producer and consumer
> set at the same time.
Actually, i am not generally concerned about inter-process
communication here. The context data (records) would have to be
buffered in queues (kind of tables) again anyway. But it could even be
"in memory". And: for passing on a UNIFORM context structure to all
the stages 1) only one stage will access a concrete context record at
a time. So there is no need for access synchronization (e.g.
locking). Moreover 2) the inter-process messages could be cut down to
transfer only a tracking_id for a context record, not the entire
context record itself.
Isn't BEA MessageQ in oracle's portfolio? *smile*
best regards,
Frank
Received on Sun Oct 10 2010 - 09:19:30 CDT