Re: [External] Remote database table cardinality estimation

Thank you so much.

I think as having a new function based index created for this query will still require hints/profile/SQL patch to force the indexed execution path so I am thinking why not just go for use_hash path (using prints/profile/SQL patch) if the response time is acceptable.

I am still trying to understand if by anyway tweaking/correcting any table/column stats optimizer can automatically pick the hash join for the remote table rather me forcing it to go for that path?

On Wed, 2 Dec 2020, 4:50 pm Laurentiu Oprea, <laurentiu.oprea06_at_gmail.com> wrote:

> Hello Lok,
>
> I was not able to find any reference that a function based index is not
> used for a remote table part of join so I tried to replicate the blog page
> referenced by Jonathan.
>
> Indeed by default on my pdb 19c test db behaved the same, but with proper
> hinting the index was used: /*+ GATHER_PLAN_STATISTICS index(t2 t2_f1)
> use_nl(t1 t2)*/
>
> This concludes that it is not a limitation but more of a plan selection
> issue.
>
> I would say the best approaches for your situation is :
> 1. maybe discuss with the business and possibly remove the upper
> function, it might be the case that is not needed
> 2. if 1 not possible then create a function based index
> 3. if 1 and 2 not possible then introduce USE_HASH(CT) hint either in the
> code or using a sql patch
>
> Hope this helps.
>
> În mar., 1 dec. 2020 la 16:46, Lok P <loknath.73_at_gmail.com> a scris:
>
>> The DISTINCT M_ID subquery estimation is deviating from ~2 estimated rows
>> VS ~15 actual rows, So I am thinking if by some way we can make the
>> estimation of that HASH JOIN at Line-25 accurate will help us here or the
>> only option would be to consider adding the remote table REMOTE_TAB alias
>> "CT" as HASH JOIN hint i.e. (USE_HASH(CT)) ?
>>
>> On Tue, Dec 1, 2020 at 2:24 PM Lok P <loknath.73_at_gmail.com> wrote:
>>
>>> Thank You very much.
>>>
>>> As you mentioned the function based index is not going to help(and also
>>> i verified we don't have any function based index rather a simple btree
>>> index on column NM_TXT), so basically the current "NESTED LOOP OUTER" path
>>> for scanning REMOTE_TAB is not going to help us here. And driving_site hint
>>> may not help as we have a lot of big tables that get scanned/fetched are
>>> part of the local database here in this query. So what would be the
>>> possible way to fix this query here?
>>>
>>> I am not able to fully confirm if its the estimation of the REMOTE_TAB
>>> is causing the issue OR the HASH join at plan_line_id - 25 is the cause
>>> of the bad path. As the remote_tab is holding ~400k rows and all of them
>>> have unique NM_TXT and the estimation shows as ~1 (which is the minimum
>>> possible values Oracle shows in the plan). But anyway, as i read your
>>> blogs, it says the optimizer won't be using column histogram info which
>>> going through DB link, (and i belive that means optimizer can use all other
>>> stats like num_rows, num_distinct, num_nulls etc from the remote
>>> database for evaluating execution plan) and in our case that column
>>> NM_TXT is all unique , so i believe that issue should not affect us here.
>>> Please correct me if I am wrong.
>>>
>>> Below is the second part of the UNION ALL from which the plan-line ~42
>>> is coming up. I have removed the exact object names and just kept the
>>> aliases, in the below query. it uses a bind variable :b2 which is
>>> coming as a string from Java and its coming as full string like '9999999'
>>> and no wild search is happening here (say like '%999999%'), so not sure
>>> why the 'LIKE' operator is used here, as it looks to me, it can be replaced
>>> by an equal operator. But then I was trying to see the estimation of
>>> that subquery part which is performing distinct M_ID as that is one of the
>>> inputs to the HASH join of plan line id-25.
>>>
>>> I see with the Like operator with bind variable, the estimation goes way
>>> higher, as i believe optimizer is doing it based on wild character search.
>>> But with an equal operator it's coming as '1' and with literal the
>>> estimation is coming as '2'. We do use bind variable here, but I think
>>> optimizer is peeking at it at the run time and deciding the estimation
>>> as "2". But I verified the actual count from this subquery is coming
>>> ~15. I am not sure how I can i fix this one , and if it's really going
>>> to put the table REMOTE_TAB in "HASH JOIn OUTER" irrespective of
>>> "NESTED LOOP OUTER"?
>>>
>>>
>>> SELECT ...
>>>
>>> FROM u, gc, ct, rol, mc, c, m
>>>
>>> WHERE gc.CT_ID = u.CT_ID(+)
>>>
>>> AND gc.AB_ID = u.AB_ID(+)
>>>
>>> AND u.NM_TXT = UPPER (ct.NM_TXT(+))
>>>
>>> AND u.RID = rol.RID(+)
>>>
>>> AND gc.M_CID = 'M'
>>>
>>> AND gc.C_TYP = 'X'
>>>
>>> AND gc.AB_ID = mc.AB_ID
>>>
>>> AND gc.CT_ID = mc.CT_ID
>>>
>>> AND mc.M_ID = m.M_ID
>>>
>>> AND m.CID = c.CID
>>>
>>> AND mc.M_ID IN
>>>
>>> (SELECT DISTINCT M_ID
>>>
>>> FROM M, C
>>>
>>> WHERE M.AB_ID = C.AB_ID
>>>
>>> AND M.CID = C.CID
>>>
>>> AND c.CID LIKE ( :B2)
>>>
>>> )
>>>
>>> ORDER BY 1;
>>>
>>>
>>> SQL> SELECT DISTINCT M_ID
>>>
>>> 2 FROM M, C
>>>
>>> 3 WHERE M.AB_ID = C.AB_ID
>>>
>>> 4 AND M.CID = C.CID
>>>
>>> 5 AND c.CID LIKE ( :B2);
>>>
>>> Elapsed: 00:00:00.04
>>>
>>>
>>> Execution Plan
>>>
>>> ----------------------------------------------------------
>>>
>>> Plan hash value: 2038584311
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------------------
>>>
>>> | Id | Operation | Name | Rows | Bytes |TempSpc| Cost
>>> (%CPU)| Time |
>>>
>>>
>>> -------------------------------------------------------------------------------------------
>>>
>>> | 0 | SELECT STATEMENT | | 33712
>>> | 1349K| | 6916 (1)| 00:01:23 |
>>>
>>> | 1 | HASH UNIQUE | | 33712 | 1349K| 1728K| 6916 (1)|
>>> 00:01:23 |
>>>
>>> |* 2 | HASH JOIN | | 33712 | 1349K| | 6555 (1)|
>>> 00:01:19 |
>>>
>>> |* 3 | INDEX FAST FULL SCAN| C_PK | 20096
>>> | 274K| | 817 (1)| 00:00:10 |
>>>
>>> | 4 | TABLE ACCESS FULL | MRC | 638K| 16M| | 5732 (1)|
>>> 00:01:09 |
>>>
>>>
>>> -------------------------------------------------------------------------------------------
>>>
>>>
>>> Predicate Information (identified by operation id):
>>>
>>> ---------------------------------------------------
>>>
>>>
>>> 2 - access("M"."AB_ID"="C"."AB_ID" AND
>>>
>>> "M"."CID"="C"."CID")
>>>
>>> 3 - filter("C"."CID" LIKE :B2)
>>>
>>>
>>>
>>> SQL> SELECT DISTINCT M_ID
>>>
>>> 2 FROM M, C
>>>
>>> 3 WHERE M.AB_ID = C.AB_ID
>>>
>>> 4 AND M.CID = C.CID
>>>
>>> 5 AND c.CID = ( :B2);
>>>
>>> Elapsed: 00:00:00.04
>>>
>>>
>>> Execution Plan
>>>
>>> ----------------------------------------------------------
>>>
>>> Plan hash value: 3581879523
>>>
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>> | Id | Operation | Name | Rows | Bytes | Cost
>>> (%CPU)| Time |
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>> | 0 | SELECT STATEMENT | | 1 | 41
>>> | 8 (13)| 00:00:01 |
>>>
>>> | 1 | HASH UNIQUE | | 1 | 41
>>> | 8 (13)| 00:00:01 |
>>>
>>> | 2 | NESTED LOOPS | | 1 | 41
>>> | 7 (0)| 00:00:01 |
>>>
>>> | 3 | NESTED LOOPS | | 1 | 41
>>> | 7 (0)| 00:00:01 |
>>>
>>> |* 4 | INDEX SKIP SCAN | C_PK | 1 | 14
>>> | 4 (0)| 00:00:01 |
>>>
>>> |* 5 | INDEX RANGE SCAN | MRC_IDX | 1
>>> | | 2 (0)| 00:00:01 |
>>>
>>> | 6 | TABLE ACCESS BY INDEX ROWID| MRC | 1 | 27
>>> | 3 (0)| 00:00:01 |
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>>
>>> Predicate Information (identified by operation id):
>>>
>>> ---------------------------------------------------
>>>
>>>
>>> 4 - access("C"."CID"=:B2)
>>>
>>> filter("C"."CID"=:B2)
>>>
>>> 5 - access("M"."AB_ID"="C"."AB_ID" AND "M"."CID"=:B2)
>>>
>>>
>>> SQL> SELECT DISTINCT M_ID
>>>
>>> 2 FROM M, C
>>>
>>> 3 WHERE M.AB_ID = C.AB_ID
>>>
>>> 4 AND M.CID = C.CID
>>>
>>> 5 AND c.CID like '99999' ;
>>>
>>> Elapsed: 00:00:00.04
>>>
>>>
>>> Execution Plan
>>>
>>> ----------------------------------------------------------
>>>
>>> Plan hash value: 3581879523
>>>
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>> | Id | Operation | Name | Rows | Bytes | Cost
>>> (%CPU)| Time |
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>> | 0 | SELECT STATEMENT | | 2 | 82
>>> | 8 (13)| 00:00:01 |
>>>
>>> | 1 | HASH UNIQUE | | 2 | 82
>>> | 8 (13)| 00:00:01 |
>>>
>>> | 2 | NESTED LOOPS | | 2 | 82
>>> | 7 (0)| 00:00:01 |
>>>
>>> | 3 | NESTED LOOPS | | 2 | 82
>>> | 7 (0)| 00:00:01 |
>>>
>>> |* 4 | INDEX SKIP SCAN | C_PK | 1 | 14
>>> | 4 (0)| 00:00:01 |
>>>
>>> |* 5 | INDEX RANGE SCAN | MRC_IDX | 1
>>> | | 2 (0)| 00:00:01 |
>>>
>>> | 6 | TABLE ACCESS BY INDEX ROWID| MRC | 2 | 54
>>> | 3 (0)| 00:00:01 |
>>>
>>>
>>> ----------------------------------------------------------------------------------------------
>>>
>>>
>>> Predicate Information (identified by operation id):
>>>
>>> ---------------------------------------------------
>>>
>>>
>>> 4 - access("C"."CID"='99999')
>>>
>>> filter("C"."CID"='99999')
>>>
>>> 5 - access("M"."AB_ID"="C"."AB_ID" AND
>>>
>>> "M"."CID"="C"."CID")
>>>
>>> On Tue, Dec 1, 2020 at 3:40 AM Jonathan Lewis <jlewisoracle_at_gmail.com>
>>> wrote:
>>>
>>>>
>>>> Function-based index wouldn't help. (In fact there may already be such
>>>> an index in place).
>>>> The optimizer ignored function-based indexes on remote tables in
>>>> distributed joins.
>>>> https://jonathanlewis.wordpress.com/2018/05/04/fbi-limitation/
>>>> The cardinality estimate for this operation might be one of the basic
>>>> guesses (like 1% for "I haven't a clue but it's an equality")
>>>>
>>>>
>>>> Regards
>>>> Jonathan Lewis
>>>>
>>>>
>>>> On Mon, 30 Nov 2020 at 20:46, Jeffrey Beckstrom <jbeckstrom_at_gcrta.org>
>>>> wrote:
>>>>
>>>>> Your step 42 states:
>>>>>
>>>>> 42 - SELECT "NM_TXT","F_NM","L_NM","EM","A_LCK" FROM
>>>>> "USER1"."REMOTE_TAB" "CT" WHERE :1=UPPER("NM_TXT") (accessing 'XXXXX.DBLNK'
>>>>> )
>>>>>
>>>>> Since you are doing a "upper" on the column, do you have a have a
>>>>> function index on that column?
>>>>>
>>>>> Jeffrey Beckstrom
>>>>> Lead Database Administrator
>>>>> Information Technology Department
>>>>> Greater Cleveland Regional Transit Authority
>>>>> 1240 W. 6th Street
>>>>> Cleveland, Ohio 44113
>>>>>
>>>>> >>> Lok P <loknath.73_at_gmail.com> 11/30/20 3:40 PM >>>
>>>>> Basically I was thinking if oracle make the cardinality estimation by
>>>>> reading the statistics information from the remote database objects during
>>>>> parsing of the query only? Or it apply some percentage logic to have the
>>>>> cardinality estimation of the remote object which then can deviate from
>>>>> actuals and thus may resulted into bad plan?
>>>>>
>>>>> On Tue, 1 Dec 2020, 1:36 am Lok P, <loknath.73_at_gmail.com> wrote:
>>>>>
>>>>>> Its version 11.2.0.4 of oracle database. We have below query spending
>>>>>> quite a lot of time on "SQL*Net message from dblink" on step-42. And it
>>>>>> seems it's because it's scanning that table in FULL, hundreds of times
>>>>>> which seems to be because of some bad estimation. My thought was if line
>>>>>> number -25 in the plan i.e the HASH JOIN is causing the overall estimation
>>>>>> wrong and thus the REMOTE_TAB is getting joined as "Nested loop outer" but
>>>>>> not "Hash Join outer". And possibly REMOTE_TAB would have joined as "hash
>>>>>> join outer" making it finish in quick time. Is my understanding correct
>>>>>> here? OR The estimation of remote object cardinality is posing a problem
>>>>>> for optimizers here and is the cause?
>>>>>> The query is a UNION ALL query, i have shared the predicate section.
>>>>>> And another thing is column NM_TXT of REMOTE_TAB is having an index on it,
>>>>>> but because of the UPPER function it's not getting used. So probably
>>>>>> creating a new function based index on REMOTE_TAB.UPPER(NM_TXT) will make
>>>>>> the query faster even with NESTED LOOP OUTER path. But I want to see if we
>>>>>> can fix any table/column stats , so that the optimizer can by default
>>>>>> follow the HASh JOIN OUTER path? Also that table REMOTE_TAB has ~427K rows
>>>>>> and almost all the rows have distinct NM_TXT values.
>>>>>> Below is the sql monitor but its not formatted well so i also
>>>>>> Attached the same sql monitor of the query , i have replaced the exact
>>>>>> object names with dummy ones.
>>>>>>
>>>>>> Global Information
>>>>>> ------------------------------
>>>>>> Status : EXECUTING
>>>>>> Instance ID : 1
>>>>>> SQL Execution ID : 16777456
>>>>>> Execution Started : 11/23/2020 00:40:07
>>>>>> First Refresh Time : 11/23/2020 04:13:17
>>>>>> Last Refresh Time : 11/23/2020 04:38:00
>>>>>> Duration : 14327s
>>>>>> Program : JDBC Thin Client
>>>>>>
>>>>>> Global Stats
>>>>>>
>>>>>> =================================================================================
>>>>>> | Elapsed | Cpu | IO | Concurrency | Other | Buffer | Read | Read |
>>>>>> | Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs |
>>>>>> Bytes |
>>>>>>
>>>>>> =================================================================================
>>>>>> | 14274 | 5.21 | 0.00 | 0.00 | 14268 | 67307 | 1 | 8192 |
>>>>>>
>>>>>> =================================================================================
>>>>>>
>>>>>> SQL Plan Monitoring Details (Plan Hash Value=1724585072)
>>>>>>
>>>>>> =============================================================================================================================================================================================
>>>>>> | Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows |
>>>>>> Mem | Activity | Activity Detail |
>>>>>> | | | | (Estim) | | Active(s) | Active | | (Actual) | | (%) | (#
>>>>>> samples) |
>>>>>>
>>>>>> =============================================================================================================================================================================================
>>>>>> | 0 | SELECT STATEMENT | | | | | | 1 | | | | |
>>>>>> | -> 1 | SORT ORDER BY | | 2 | 20772 | 1484 | +12790 | 1 | 0 | 2M | |
>>>>>> |
>>>>>> | -> 2 | UNION-ALL | | | | 1484 | +12790 | 1 | 6887 | | | |
>>>>>> | 3 | NESTED LOOPS OUTER | | 1 | 2272 | 1 | +12790 | 1 | 152 | | | |
>>>>>> | 4 | NESTED LOOPS OUTER | | 1 | 11 | 1 | +12790 | 1 | 152 | | | |
>>>>>> | 5 | NESTED LOOPS OUTER | | 1 | 10 | 1 | +12790 | 1 | 152 | | | |
>>>>>> | 6 | NESTED LOOPS | | 1 | 8 | 1 | +12790 | 1 | 152 | | | |
>>>>>> | 7 | NESTED LOOPS | | 1 | 6 | 1 | +12790 | 1 | 152 | | | |
>>>>>> | 8 | TABLE ACCESS BY INDEX ROWID | TAB1 | 1 | 5 | 1 | +12790 | 1 | 1
>>>>>> | | | |
>>>>>> | 9 | INDEX SKIP SCAN | TAB1_PK | 1 | 4 | 1 | +12790 | 1 | 1 | | | |
>>>>>> | 10 | INDEX RANGE SCAN | G_CTCT_PK | 1 | 1 | 1 | +12790 | 1 | 152 |
>>>>>> | | |
>>>>>> | 11 | TABLE ACCESS BY INDEX ROWID | G_CTCT | 1 | 2 | 1 | +12790 |
>>>>>> 152 | 152 | | | |
>>>>>> | 12 | INDEX UNIQUE SCAN | G_CTCT_PK | 1 | 1 | 1 | +12790 | 152 | 152
>>>>>> | | | |
>>>>>> | 13 | TABLE ACCESS BY INDEX ROWID | PU_PRFL | 1 | 2 | 1 | +12790 |
>>>>>> 152 | 141 | | | |
>>>>>> | 14 | INDEX UNIQUE SCAN | PU_PRFL_IDX | 1 | 1 | 1 | +12790 | 152 |
>>>>>> 141 | | | |
>>>>>> | 15 | TABLE ACCESS BY INDEX ROWID | P_RL | 1 | 1 | 1 | +12790 | 152
>>>>>> | 141 | | | |
>>>>>> | 16 | INDEX UNIQUE SCAN | P_RL_PK | 1 | | 1 | +12790 | 152 | 141 | |
>>>>>> | |
>>>>>> | 17 | REMOTE | REMOTE_TAB | 1 | 2261 | 1 | +12790 | 152 | 73 | | | |
>>>>>> | -> 18 | NESTED LOOPS OUTER | | 1 | 18500 | 1484 | +12790 | 1 | 6735
>>>>>> | | | |
>>>>>> | -> 19 | HASH JOIN | | 1 | 16239 | 1484 | +12790 | 1 | 6736 | 2M | |
>>>>>> |
>>>>>> | 20 | NESTED LOOPS OUTER | | 2 | 14795 | 1 | +12790 | 1 | 8356 | | |
>>>>>> |
>>>>>> | 21 | NESTED LOOPS OUTER | | 2 | 14793 | 1 | +12790 | 1 | 8356 | | |
>>>>>> |
>>>>>> | 22 | HASH JOIN | | 2 | 14789 | 1 | +12790 | 1 | 8356 | | | |
>>>>>> | 23 | NESTED LOOPS | | 2 | 8903 | 1 | +12790 | 1 | 8356 | | | |
>>>>>> | 24 | NESTED LOOPS | | 4 | 8903 | 1 | +12790 | 1 | 8356 | | | |
>>>>>> | 25 | HASH JOIN | | 4 | 8895 | 1 | +12790 | 1 | 8356 | | | |
>>>>>> | 26 | VIEW | VW_NSO_1 | 2 | 7 | 1 | +12790 | 1 | 15 | | | |
>>>>>> | 27 | HASH UNIQUE | | 2 | | 1 | +12790 | 1 | 15 | | | |
>>>>>> | 28 | NESTED LOOPS | | 2 | 7 | 1 | +12790 | 1 | 15 | | | |
>>>>>> | 29 | NESTED LOOPS | | 2 | 7 | 1 | +12790 | 1 | 15 | | | |
>>>>>> | 30 | INDEX SKIP SCAN | TAB1_PK | 1 | 4 | 1 | +12790 | 1 | 1 | | | |
>>>>>> | 31 | INDEX RANGE SCAN | MRC_IDX | 1 | 2 | 1 | +12790 | 1 | 15 | | |
>>>>>> |
>>>>>> | 32 | TABLE ACCESS BY INDEX ROWID | MRC | 2 | 3 | 1 | +12790 | 15 |
>>>>>> 15 | | | |
>>>>>> | 33 | INDEX FULL SCAN | M_CTCT | 1M | 8862 | 1 | +12790 | 1 | 1M | |
>>>>>> | |
>>>>>> | -> 34 | INDEX UNIQUE SCAN | G_CTCT_PK | 1 | 1 | 1484 | +12790 |
>>>>>> 8356 | 8356 | | | |
>>>>>> | 35 | TABLE ACCESS BY INDEX ROWID | G_CTCT | 1 | 2 | 1 | +12790 |
>>>>>> 8356 | 8356 | | | |
>>>>>> | 36 | TABLE ACCESS FULL | MRC | 639K | 5874 | 1 | +12790 | 1 | 641K
>>>>>> | | | |
>>>>>> | 37 | TABLE ACCESS BY INDEX ROWID | PU_PRFL | 1 | 2 | 1 | +12790 |
>>>>>> 8356 | 8356 | | | |
>>>>>> | 38 | INDEX UNIQUE SCAN | PU_PRFL_IDX | 1 | 1 | 1 | +12790 | 8356 |
>>>>>> 8356 | | | |
>>>>>> | 39 | TABLE ACCESS BY INDEX ROWID | P_RL | 1 | 1 | 1 | +12790 | 8356
>>>>>> | 8352 | | | |
>>>>>> | 40 | INDEX UNIQUE SCAN | P_RL_PK | 1 | | 1 | +12790 | 8356 | 8352 |
>>>>>> | | |
>>>>>> | -> 41 | TABLE ACCESS FULL | TAB1 | 394K | 1437 | 1484 | +12790 | 1
>>>>>> | 9033 | | | |
>>>>>> | -> 42 | REMOTE | REMOTE_TAB | 1 | 2261 | 3302 | +11027 | 6736 | 379
>>>>>> | | 100.00 | SQL*Net message from dblink (3248) |
>>>>>>
>>>>>> =============================================================================================================================================================================================
>>>>>>
>>>>>>
>>>>>> Predicate Information (identified by operation id):
>>>>>> ---------------------------------------------------
>>>>>> 9 - access("C"."CID" LIKE :1) filter("C"."CID" LIKE :1)
>>>>>> 10 - access("CC"."AB_ID"="C"."AB_ID" AND "CC"."CID"="C"."CID")
>>>>>> 11 - filter(("GC"."M_CID"='C' AND "GC"."AB_ID" IS NOT NULL AND
>>>>>> "GC"."C_TYP"='X' AND "GC"."AB_ID"="CC"."AB_ID"))
>>>>>> 12 - access("GC"."CT_ID"="CC"."CT_ID")
>>>>>> 13 - filter("GC"."AB_ID"="U"."AB_ID")
>>>>>> 14 - access("GC"."CT_ID"="U"."CT_ID")
>>>>>> 16 - access("U"."RID"="ROL"."RID")
>>>>>> 19 - access("M"."CID"="C"."CID")
>>>>>> 22 - access("MC"."M_ID"="M"."M_ID")
>>>>>> 25 - access("MC"."M_ID"="M_ID")
>>>>>> 30 - access("C"."CID" LIKE :2) filter("C"."CID" LIKE :2)
>>>>>> 31 - access("M"."AB_ID"="C"."AB_ID" AND "M"."CID"="C"."CID")
>>>>>> 34 - access("GC"."CT_ID"="MC"."CT_ID")
>>>>>> 35 - filter(("GC"."AB_ID" IS NOT NULL AND "GC"."M_CID"='M' AND
>>>>>> "GC"."C_TYP"='X' AND "GC"."AB_ID"="MC"."AB_ID"))
>>>>>> 37 - filter("GC"."AB_ID"="U"."AB_ID")
>>>>>> 38 - access("GC"."CT_ID"="U"."CT_ID")
>>>>>> 40 - access("U"."RID"="ROL"."RID")
>>>>>>
>>>>>> Remote SQL Information (identified by operation id):
>>>>>> ----------------------------------------------------
>>>>>> 17 - SELECT "NM_TXT","F_NM","L_NM","EM","A_LCK" FROM
>>>>>> "USER1"."REMOTE_TAB" "CT" WHERE :1=UPPER("NM_TXT") (accessing 'XXXXX.DBLNK'
>>>>>> )
>>>>>>
>>>>>> 42 - SELECT "NM_TXT","F_NM","L_NM","EM","A_LCK" FROM
>>>>>> "USER1"."REMOTE_TAB" "CT" WHERE :1=UPPER("NM_TXT") (accessing 'XXXXX.DBLNK'
>>>>>> )
>>>>>>
>>>>>>

--
http://www.freelists.org/webpage/oracle-l