BULK COLLECTIONS and CURRENT OF [message #521829] |
Fri, 02 September 2011 05:26  |
mjm22
Messages: 54 Registered: January 2010 Location: Singapore
|
Member |
|
|
I have a driver table from which I need to update another table while, at the same time, record the fact that I have processed each record on the driver table.
The driver table will contain around 3.5 million records, therefore I intended to handle this using some bulk collections, with a LIMIT option so that I don't hit any memory problems.
I would also prefer to commit in batches, or at least handle exceptions using the SAVE EXCEPTION clause. The problem is I seem to be running into an error when trying to make the update to the driver table (the commented out code). With this in, I get the error:
ORA-01410: invalid ROWID
ORA-06512: at "CUST_MAIL_UPDATE", line 217
ORA-06512: at line 38
Can the CURRENT OF not work with the FORALL? What is my best approach here? If I use a FOR LOOP I lose my SAVE EXCEPTIONS exception handling.
The Procedure is as follows:
-- declare some object structures to hold the retrieved data
TYPE driver_rec IS RECORD (
account_no ext_driver.account_no%TYPE,
update_action ext_driver.update_action%TYPE,
customers_rowid ext_driver.customers_rowid%TYPE);
TYPE driver_recs_tt IS TABLE OF driver_rec;
-- cursor to get the records from the driver table
CURSOR cur_get_driver_recs IS
SELECT account_no,
update_action,
customers_rowid
FROM ext_driver
WHERE processed = 'N'
FOR UPDATE OF processed;
driver_records driver_recs_tt;
error_count NUMBER;
CUST_BULK_UPDATE_FAIL EXCEPTION;
PRAGMA EXCEPTION_INIT(CUST_BULK_UPDATE_FAIL, -24381);
BEGIN
-- get the records from the driver table
OPEN cur_get_driver_recs;
LOOP
-- process the records using Bulk Collections.
FETCH cur_get_driver_recs BULK COLLECT INTO driver_records LIMIT 1000;
FORALL i IN 1..driver_records.COUNT SAVE EXCEPTIONS
UPDATE customers c
SET mailing_option = driver_records(i).update_action
WHERE c.rowid = driver_records(i).customers_rowid;
-- UPDATE ext_driver
-- SET processed = 'Y'
-- WHERE CURRENT OF cur_get_driver_recs;
EXIT WHEN cur_get_driver_recs%NOTFOUND;
END LOOP;
COMMIT;
EXCEPTION
WHEN CUST_BULK_UPDATE_FAIL THEN
error_count := SQL%BULK_EXCEPTIONS.COUNT;
DBMS_OUTPUT.PUT_LINE('Number of updates that failed: ' || error_count);
FOR i IN 1..error_count LOOP
DBMS_OUTPUT.PUT_LINE('Error #' || i || ' occurred during '||
'iteration #' || SQL%BULK_EXCEPTIONS(i).ERROR_INDEX);
DBMS_OUTPUT.PUT_LINE('Error message is ' ||
SQLERRM(-SQL%BULK_EXCEPTIONS(i).ERROR_CODE));
END LOOP;
END;
|
|
|
|
Re: BULK COLLECTIONS and CURRENT OF [message #521841 is a reply to message #521836] |
Fri, 02 September 2011 06:11   |
mjm22
Messages: 54 Registered: January 2010 Location: Singapore
|
Member |
|
|
I need to update the driver table for each successful customers table update. In this case I am now actually not sure if the SAVE EXCEPTIONS gives me what I need. If the first update fails (on Customers table), the update to the driver table will not execute right?
|
|
|
|
|
Re: BULK COLLECTIONS and CURRENT OF [message #531266 is a reply to message #521829] |
Mon, 14 November 2011 17:21   |
mjm22
Messages: 54 Registered: January 2010 Location: Singapore
|
Member |
|
|
Nelsond,
Yes, I realised my own errors!
Firstly... Originally I wanted to BULK COLLECT and employ a LIMIT of 1000 with a FORALL (see the code in my original post). The problem with this approach though is that I make two updates, one to the target table but I also need to update the driver table showing that I have processed the record.
Re-reading the Oracle documentation I realised that I cannot have two separate DMLS in the same FORALL loop if you are BULK COLLECTING.
To fix this isssue I simply altered the code:
OPEN cur_get_driver_recs;
LOOP
-- process the records using Bulk Collections.
FETCH cur_get_driver_recs BULK COLLECT INTO driver_records LIMIT 1000;
FORALL i IN 1..driver_records.COUNT SAVE EXCEPTIONS
to
OPEN cur_get_driver_recs;
LOOP
FETCH cur_get_driver_recs BULK COLLECT INTO driver_records LIMIT 1000;
EXIT WHEN driver_records.COUNT = 0;
-- we have multiple DML statements so cannot use FORALL, instead use FOR LOOP
FOR i IN 1..driver_records.COUNT
LOOP
However, this then gave me another problem... I got a fetch out of sequence error after processing the first 1000 rows. This was due to the cursor definition, cur_get_driver_recs , which I had declared 'FOR UPDATE of processed' on.
When I thought about it, this clause obviously opens a transaction in order to lock the processed table. The COMMIT I was using to attempt to commit every 1000 records was ending the transaction opened by the FOR UPDATE (As this transaction was outside of the inner loop of records I wanted to commit I was getting the fetch out of sequence error). Since I was the only process updating the PROCESSED table anyway, I simply removed the FOR UPDATE clause of the CURSOR.
In short, the new code looked like this (apologies for any small errors, I did this from memory using the original code in this thread so there may be some small typo... but you can see the FOR loop in operation with the BULK COLLECT.
TYPE driver_rec IS RECORD (
account_no ext_driver.account_no%TYPE,
update_action ext_driver.update_action%TYPE,
customers_rowid ext_driver.customers_rowid%TYPE,
driver_rowid ext_driver.customers_rowid%TYPE);
TYPE driver_recs_tt IS TABLE OF driver_rec;
-- cursor to get the records from the driver table
CURSOR cur_get_driver_recs IS
SELECT account_no,
update_action,
customers_rowid,
ROWID driver_rowid
FROM ext_driver
WHERE processed = 'N'
driver_records driver_recs_tt;
BEGIN
OPEN cur_get_driver_recs;
LOOP
FETCH cur_get_driver_recs BULK COLLECT INTO driver_records LIMIT 1000;
EXIT WHEN driver_records.COUNT = 0;
-- we have multiple DML statements so cannot use FORALL, instead use FOR LOOP
FOR i IN 1..driver_records.COUNT
LOOP
BEGIN
UPDATE customers c
SET mailing_option = driver_records(i).update_action
WHERE a.rowid = driver_records(i).customers_rowid;
-- now update the driver table
UPDATE driver_table
SET processed = 'Y'
WHERE ROWID = driver_records(i).driver_rowid;
EXCEPTION
-- if either update fails then we get here. Result is no update to driver
-- table and record will be processed the next time.
WHEN OTHERS THEN
dbms_output.put_line('Update failed for Acct: ' || driver_records(i).account_num ||
' with Error: ' || SQLCODE || ' (' || SQLERRM || ')');
END;
END LOOP;
dbms_output.put_line('Commiting ' || driver_records.COUNT);
-- commit the batch
COMMIT;
END LOOP;
CLOSE cur_get_driver_recs;
END;
|
|
|
|
|
|
Re: BULK COLLECTIONS and CURRENT OF [message #531311 is a reply to message #531266] |
Tue, 15 November 2011 03:19   |
cookiemonster
Messages: 13967 Registered: September 2008 Location: Rainy Manchester
|
Senior Member |
|
|
mjm22 wrote on Mon, 14 November 2011 23:21Re-reading the Oracle documentation I realised that I cannot have two separate DMLS in the same FORALL loop if you are BULK COLLECTING.
While that is true you can do two seperate forall loops on the same array. Might be worth trying both approaches and seeing which is more efficient.
|
|
|
Re: BULK COLLECTIONS and CURRENT OF [message #531314 is a reply to message #531310] |
Tue, 15 November 2011 03:52  |
 |
Michel Cadot
Messages: 68767 Registered: March 2007 Location: Saint-Maur, France, https...
|
Senior Member Account Moderator |
|
|
Quote: I do not really expect anything to go wrong!
Your assumption is wrong.
What about a memory shortage? A network, OS, memory error? And a thousand of other possible reasons your statement can fail.
Quote:The reason for the WHEN OTHERS exception handling is more to allow the process to continue to update records after an error is encountered
And how will you know which ones have already been done and which have not? (maybe no output will be displayed or it can't be displayed or no one will read it or...)
Quote:The procedure is called from a UNIX script which handles output to a log file in the file system, so anything that enters into the exception handling will be captured in the log.
And if the file system is full?
Regards
Michel
|
|
|