Re: 10g: chaining pipelined functions is slow
Date: Mon, 11 Oct 2010 01:40:28 -0700 (PDT)
Message-ID: <55a9d67b-34d5-4702-aa91-10570b76e364_at_q2g2000vbk.googlegroups.com>
On Oct 10, 4:19 pm, Frank Bergemann <FBergem..._at_web.de> wrote:
> I will. To be honest, i am a bit disappointed by this.
> It screw up my approach to scale up for a multi-stage processing
> chain, dealing with different resources (tables) in the different
> processing stages. So i have to change it the way you described before
> and set up parallism by data partitioning.
Do you have evidence that Oracle's approach screwed up your processing chain meaning do you have traced your execution to understand where potentially excess work happens resp. where most of the time is spent? It is very fortunate that Oracle is instrumented that well so that you don't have to assume anything but can get hard facts where most of the time is spent.
If you use the approach outlined above (the cascaded table function model mentioned, not your original approach) then all the pipelined table functions are really executed in a cascaded way which looks to me like a quite efficient approach. You would potentially be CPU bound since only one process is working on the whole processing chain, however you need to be sure that CPU is really the problem here and not anything else like I/O operations.
> the inter-process messages could be cut down to
> transfer only a tracking_id for a context record, not the entire
> context record itself.
> Isn't BEA MessageQ in oracle's portfolio? *smile*
Hm, and where is the actual data then stored that is pipelined between the different processing worker threads? I think this is much more complex than your simple description here. Either you have to pipeline the whole data stream or you need to keep that data separate which means that you have additional overhead for maintaining and accessing this data again at each processing stage.
Regards,
Randolf
Oracle related stuff blog:
http://oracle-randolf.blogspot.com/
Co-author of the "OakTable Expert Oracle Practices" book: http://www.apress.com/book/view/1430226684 http://www.amazon.com/Expert-Oracle-Practices-Database-Administration/dp/1430226684 Received on Mon Oct 11 2010 - 03:40:28 CDT