Home » SQL & PL/SQL » SQL & PL/SQL » Cursor limit.. how??
Cursor limit.. how?? [message #158328] |
Thu, 09 February 2006 23:01  |
a_developer
Messages: 194 Registered: January 2006
|
Senior Member |
|
|
If I put say 50M of records in a CURSOR, and fetch them using bulk collect limit 5000, will I have out of memory error???
I have to delete from 11 tables with approx 1000 records from each. We have 500MB SGA..
It's actually like this:
I have to use the CURSOR with ORDER BY. This will fetch approx 50M. Then process using BULK COLLECT limit of 5000. How does this happen? I mean will the sorted 50M be put to memory OR only evrytime 5000 records are fetched they're put in memory???
If it only puts 5000 records in the memory, once it finishes processing those, will it again execute the SELECT stmt of the cursor to get the next 5000??? Or will it get that from another memory location where the sorted 50M records are???
Please help.. thanks.
[Updated on: Thu, 09 February 2006 23:58] Report message to a moderator
|
|
|
|
Re: Cursor limit.. how?? [message #158388 is a reply to message #158343] |
Fri, 10 February 2006 06:41   |
a_developer
Messages: 194 Registered: January 2006
|
Senior Member |
|
|
Which one is better?
APPROACH 1:
CURSOR c IS
SELECT id FROM parent
WHERE status = 'delete'
ORDER BY id;
OPEN c
LOOP
FETCH c BULK COLLECT INTO rec_tab LIMIT 5000
FOR i IN 1..rectab.COUNT
LOOP
< delete child records here and eventually the parent record >
END LOOP;
EXIT WHEN C%NOTFOUND;
END LOOP;
CLOSE c;
APPROACH 2:
CURSOR c IS
SELECT * FROM temp_t
ORDER BY id;
EXECUTE IMMEDIATE 'DROP TABLE temp_T';
EXECUTE IMMEDIATE 'CREATE GLOBAL TEMPORARY TABLE temp_t
AS SELECT id FROM parent
WHERE status = 'delete';
SELECT COUNT(*)
INTO rec_ctr
FROM temp_t;
WHILE rec_ctr > 0
LOOP
-- note that this does not loop in the cursor
-- it opens the cursor, gets 5000 recs and closes it
--
OPEN c
FETCH c BULK COLLECT INTO rec_tab LIMIT 5000
FOR i IN 1..rectab.COUNT
LOOP
< delete child records here and eventually the parent record >
DELETE FROM temp_t WHERE id = rectab(i)
cOMMIT;
END LOOP;
CLOSE c;
SELECT COUNT(*)
INTO rec_ctr
FROM temp_t;
END LOOP;
My colleague is afraid of 'out of memory' error so he suggests APPROACH 2.
He says that in APPROACH 1, for example, there are 15M of records retrieved by the cursor which are sorted since it has an ORDER BY clause. So what Oracle does is stores these sorted 15M cursor records in a memory. And from there, we may fetch in bulk collect with limit of 5000 records - and again puts this in another memory location. He says that we may run out of memory because apart from this, there are also processing going on (deletion) with the child records..
While in APPROACH 2, he saves the 15M to a TEMP table (which is not sorted). Create a cursor selecting from TEMP_t with ORDER BY. OPEN that cursor, then fetch from this cursor 5000 records, processed that and then DELETE those 5000 records from TEMP_T. Closes the cursor immediately -- no looping. So by the next time we AGAIN open the cursor, when we select from TEMP_T, it has 5000-less records. So he argues that the memory occupied by TEMP_T gets reduced by 5000 records all the time. The sequence is like:
-- FIRST ITERATION
-- CURSOR c has 15M records retrieved from TEMP_T
OPEN c
FETCH 5000 in BULK
DELETE THE 5000 fetched records from TEMP_T
CLOSE;
-- 2nd ITERATIOn
-- CURSOR c has 15M-5000 records retrieved from TEMP_T
OPEN c
FETCH 5000 in BULK
DELETE THE 5000 fetched records from TEMP_T
CLOSE;
-- 3nd ITERATIOn
-- CURSOR c has (15M-5000)-5000 records retrieved from TEMP_T
OPEN c
FETCH 5000 in BULK
DELETE THE 5000 fetched records from TEMP_T
CLOSE;
.. and so on UNTIL there are no more records in TEMP_T.
I know APPROACH 2 is very crude and takes more time.. BUT he's argument is to save MEMORY. Is there really more memory consumed by APPROACH 1 than APPROACH 2??
Please help.. or refer any link on CURSOR regarding how it is stored in memory, etc...
THANKS..
|
|
|
Re: Cursor limit.. how?? [message #158415 is a reply to message #158388] |
Fri, 10 February 2006 13:05   |
scottwmackey
Messages: 515 Registered: March 2005
|
Senior Member |
|
|
Hmm, first I am going to make a suggestion that may make the argument moot. Why can't you just:
DELETE child_table
WHERE child_id IN (SELECT child_id
FROM parent_table
WHERE status = 'delete');
--
DELETE parent_table
WHERE status = 'delete'
If you don't have a reason why not, I cannot recommend strongly enough that you do it this way. If the answer is, We have to do other stuff in between and absolutely have to have a cursor, then I would still ask what the ORDER BY is buying you. You're deleting. Why does the order matter?
With regard to you specific question and your argument with your colleague. The table, ft_trns_input, in the PL/SQL block below contains about 117 million records. The block executes in 1.328 seconds. Ask your colleague if he really believes that our machine can load and sort 117 million records in that amount of time.
SQL> DECLARE
2 CURSOR c IS
3 SELECT *
4 FROM ft_trns_input
5 ORDER BY trns_input_id DESC;
6 TYPE r_tab_type IS TABLE OF c%ROWTYPE;
7 r_tab r_tab_type;
8 BEGIN
9 OPEN c;
10 FETCH c BULK COLLECT INTO r_tab LIMIT 5000;
11 CLOSE c;
12 END;
13 /
PL/SQL procedure successfully completed
Executed in 1.328 seconds
SQL> As I said previously, if the ORDER BY is on an indexed column, then Oracle does not have to do any work to sort. It just uses the index. And no, it does not retrieve all of the records.
Finally, one of the nice things about our line of work is that we can easily test our hypothoses. I think Approach 2 is a very poor design. I would think the burden of proof would lie with your colleague to demonstrate that Approach 1 (which I also think is not a very good design) does indeed create memory problems.
|
|
|
Re: Cursor limit.. how?? [message #158443 is a reply to message #158415] |
Sat, 11 February 2006 00:06   |
a_developer
Messages: 194 Registered: January 2006
|
Senior Member |
|
|
Thank you very much, Scott.
I think the code you suggested should be like:
DELETE child_table
WHERE parent_id IN (SELECT parent_id
FROM parent_table
WHERE status = 'delete');
--
DELETE parent_table
WHERE status = 'delete'
That's really the fastest way. However, I have like 8 child tables, so instead of having 8 like:
DELETE FROM child_table_1 WHERE parent_id IN (SELECT parent_id FROM parent WHERE status = 'delete';
DELETE FROM child_table_2 WHERE parent_id IN (SELECT parent_id FROM parent WHERE status = 'delete';
...
DELETE FROM child_table_8 WHERE parent_id IN (SELECT parent_id FROM parent WHERE status = 'delete';
isn't it more efficient to not have the subquery SELECT parent_id .. always execute by passing a value from a cursor?? Like:
CURSOR c IS SELECT parent_id FROM parent WHERE status = 'delete';
-- then have
DELETE FROM child_table_1 WHERE parent_id IN (c_rec.parent_id);
DELETE FROM child_table_2 WHERE parent_id IN (c_rec.parent_id);
...
DELETE FROM child_table_8 WHERE parent_id IN (c_rec.parent_id);
And also, if I do it in one shot (like what we experimented before) using FORALL, we ran out rollback segments. Thus, I decided to use a cursor and COMMIT after every 5000 parent records fetched.
Regarding the ORDER BY, the users want it that way (yup!). They wanna make sure that the older data get deleted first.
And yes, the ORDER BY column is 'indexed' - so, from your reply, can I say that: 'Only the INDEXES are kept sorted in MEMORY' - not the entire 15M cursor records (unlike in APPROACH 2)???
I was also thinking how can APPROACH1 have memory problems... Perhaps, I should ask some proof 
Thanks again.. I was surfing the net on how a cursor's memory is managed by Oracle, but I couldn't find anything complete and satisfying info. I appreciate your input.
|
|
|
Re: Cursor limit.. how?? [message #158525 is a reply to message #158443] |
Sun, 12 February 2006 23:05   |
scottwmackey
Messages: 515 Registered: March 2005
|
Senior Member |
|
|
Now that you mention it, I did give you bad advice. With the way I suggested, there is no way to guarantee that somebody doesn't set another row in your parent table to 'delete' while the delete on the child table is executing. You would therefore attempt to delete a parent record in the second delete that still had child records. Oops.
If you knew that could never happen, then it might be interesting to see which is faster. If you had an index on parent_table(status, id), then the subquery would never have to hit parent_table. It could do all the work from the index itself and, I would imagine, be quite fast. It is something you might want to test. On the other hand, I know the FORALL is very fast also, and if you have 8 child tables, I would probably go with it. If you are running into rollback segment problems, tell your DBA to increase the rollback segments. If he/she gives you some lame excuse about why it can't/won't be done, point him to http://asktom.oracle.com/pls/ask/f?p=4950:8:15167342725977524415::NO::F4950_P8_DISPLAYID,F4950_P8_CRITERIA:4951966319022 and tell him to look at the 5th and 7th sections.
As for the ORDER BY, it makes no sense to say the older data gets deleted first. It all gets deleted at the same time, when you commit at the end of the transaction. There is no first, no last, and your job is to make sure it all gets deleted. The ORDER BY is useless.
|
|
|
Re: Cursor limit.. how?? [message #158563 is a reply to message #158525] |
Mon, 13 February 2006 03:53  |
a_developer
Messages: 194 Registered: January 2006
|
Senior Member |
|
|
scottwmackey wrote on Sun, 12 February 2006 23:05 |
As for the ORDER BY, it makes no sense to say the older data gets deleted first. It all gets deleted at the same time, when you commit at the end of the transaction. There is no first, no last, and your job is to make sure it all gets deleted. The ORDER BY is useless.
|
You got a point there thanks again!
|
|
|
Goto Forum:
Current Time: Sun Aug 10 09:16:57 CDT 2025
|