Re: 10g: chaining pipelined functions is slow
Date: Mon, 11 Oct 2010 11:49:49 -0700 (PDT)
Message-ID: <1845f553-338c-49a7-a362-374c7be135a8_at_x42g2000yqx.googlegroups.com>
Am 11.10.2010 10:40, schrieb Randolf Geist:
> On Oct 10, 4:19 pm, Frank Bergemann<FBergem..._at_web.de> wrote:
>> I will. To be honest, i am a bit disappointed by this.
>> It screw up my approach to scale up for a multi-stage processing
>> chain, dealing with different resources (tables) in the different
>> processing stages. So i have to change it the way you described before
>> and set up parallism by data partitioning.
>
> Do you have evidence that Oracle's approach screwed up your processing
> chain meaning do you have traced your execution to understand where
> potentially excess work happens resp. where most of the time is spent?
> It is very fortunate that Oracle is instrumented that well so that you
> don't have to assume anything but can get hard facts where most of the
> time is spent.
Got it - i'll have a detailed look at the enterprise manager.
Your hints already helped me to change back implementation to make not
too much use of chaining (at least not in combination with this OPEN
LOOP FETCH CLOSE idiom)
- many thanks!!!
> If you use the approach outlined above (the cascaded table function
> model mentioned, not your original approach) then all the pipelined
> table functions are really executed in a cascaded way which looks to
> me like a quite efficient approach. You would potentially be CPU bound
> since only one process is working on the whole processing chain,
> however you need to be sure that CPU is really the problem here and
> not anything else like I/O operations.
>
>> the inter-process messages could be cut down to
>> transfer only a tracking_id for a context record, not the entire
>> context record itself.
>> Isn't BEA MessageQ in oracle's portfolio? *smile*
>
> Hm, and where is the actual data then stored that is pipelined between
> the different processing worker threads? I think this is much more
> complex than your simple description here. Either you have to pipeline
> the whole data stream or you need to keep that data separate which
> means that you have additional overhead for maintaining and accessing
> this data again at each processing stage.
Shared Memory?
But let me use, what's there now.
I'll have a deeper look at the links you mentioned and try to speed up
the implementation.
Most important for me was to get a _confirmation_, that there is no
assignment of chained pipelined functions to different CPUs, thread or
alike, but one "process" is executing the entire chain.
So i don't go further into wrong direction.
- thanks again!
> Regards,
> Randolf
>
> Oracle related stuff blog:
> http://oracle-randolf.blogspot.com/
>
> Co-author of the "OakTable Expert Oracle Practices" book:
> http://www.apress.com/book/view/1430226684
> http://www.amazon.com/Expert-Oracle-Practices-Database-Administration/dp/1430226684
regards,
Frank
Received on Mon Oct 11 2010 - 13:49:49 CDT