| Transferring data from one DB to another DB [message #572664] |
Sat, 15 December 2012 00:43  |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
Hi Experts,
In production database 30 tables are there in different schemas.
In some tables data is existed for 5 years
in some tables data is existed 8 years
in some tables data is existed for 6 years.
And all these tables are having billions and millions of records.
Let us assume the tables in the production database are
Table1 -- 8 years data -- No. Of Records 3538969000
Table2 -- 6 years data -- No. Of Records 592844435
Table3 -- 3 years data -- No. Of Records 33224993
Table4 -- 4 years data -- No. Of Records 52361756
Table5 -- 5 years data -- No. Of Records 8948567
.
.
.
Table15 -- 6 years data -- No. Of Records 308476987
Now I want to trnasfer these 15 tables data to test database.
Based on the following conditions.
For Table1 I want to transfer 6 years data to test database, keeep 2 years data in production
and delete all 6 years data from production databse.
For Table2 I want to transfer 4 years data to test database, keeep 2 years data in production
and delete all 4 years data from production databse.
For Table3 I want to transfer 2 years data to test database keeep 1 years data in production
and delete all 2 years data from production databse.
This will be done periodically.
I.e. suppose if we run the script now and transfer the requested data to test database,
again after one year the data got increased in production then again we have to run the
script and transfer data to test database based on some conditions,so it should work for
long time.
Please help me what is the good and fast process to transfer the data.
Thanks in advance.
[EDITED by LF: fixed topic title typo]
[Updated on: Sat, 15 December 2012 09:45] by Moderator Report message to a moderator
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #572741 is a reply to message #572668] |
Mon, 17 December 2012 02:45   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
The client is not agreeing for partitioning.
Client want to go for inserting data in Archive DB and deletepurge) data from Production DB
using DB links.And this process will be done using DBMS_JOB.
Based on the following client's requirement
For Table1 needs to keep 2 years worth of data in production DB,older than 2 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
For Table1 needs to keep 2 years worth of data in production DB,older than 2 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
For Table1 needs to keep 1 years worth of data in production DB,older than 1 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
For Table1 needs to keep 3 years worth of data in production DB,older than 3 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
I have prepared the following script.
CREATE OR REPLACE PROCEDURE Archive_tables
AS
COUNT NUMBER := 0;
total NUMBER := 0;
CURSOR del_rec_Table2
IS
SELECT *
FROM @production_db.Table2
WHERE event_time <= ADD_MONTHS (SYSDATE, -48);
CURSOR del_rec_Table3
IS
SELECT *
FROM @production_db.Table3
WHERE event_time <= ADD_MONTHS (SYSDATE, -24);
BEGIN
BEGIN
FOR rec IN del_rec_Table2
LOOP
INSERT INTO Archive_db.Table2_ARCHIVE
VALUES (rec.COLUMN1,rec.COLUMN2,rec.COLUMN3,rec.COLUMN4);
COMMIT;
DELETE FROM @production_db. Table2
WHERE ROWID = rec.ROWID;
total := total + 1;
COUNT := COUNT + 1;
IF (COUNT >= 5000)
THEN
COMMIT;
COUNT := 0;
END IF;
END LOOP;
COMMIT;
DBMS_OUTPUT.PUT_LINE ('Deleted ' || total || ' records from Table2');
END;
/
BEGIN
FOR rec IN del_record_cur
LOOP
INSERT INTO Archive_db.Table3_ARCHIVE
VALUES (rec.COLUMN1,rec.COLUMN2,rec.COLUMN3,rec.COLUMN4);
COMMIT;
DELETE FROM @production_db.Table3
WHERE ROWID = rec.ROWID;
total := total + 1;
COUNT := COUNT + 1;
IF (COUNT >= 5000)
THEN
COMMIT;
COUNT := 0;
END IF;
END LOOP;
COMMIT;
DBMS_OUTPUT.PUT_LINE ('Deleted ' || total || ' records from Table3');
END;
END Archive_tables;
/
We have 30 tables and all the tables are having millions of records and all tables to be archived and purged.
As per client requirement it will done using DBMS_JOB scheduled jobs and the jobs
will be scheduled in ARCHIVE DB.
i.e. The job should execute every one year and number of jobs should not be grater than 5.
If I write the procedure for all thirty tables it will take long time to the job.
So I am thinking to implement this using only one SELECT,INSERT and DELTE statement \passing table_names and column_names dynamically
using 4 dbms_jobs.
By using any control table to pass the table_names,Where conditions and all.
Please help me how to implement this.
Thanks in advance.
[Updated on: Mon, 17 December 2012 02:54] by Moderator Report message to a moderator
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #572822 is a reply to message #572755] |
Mon, 17 December 2012 23:53   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
Hi Experts,
To make the process dynamic why we want to make it dynamic in future any tables needs to e archived
only one script should execute.
I hav ecreate dthe below control table to pass ID, TABLE_NAME, ARCHIVE_TABLE_NAME, WHERE_CONDITION dynamically.
ID TABLE_NAME ARCHIVE_TABLE_NAME WHERE_CONDITION
1 Table1 archive_table1 created_date< MONTHS_BETWEEN(SYSDATE,-24)
1 Table2 archive_table2 Accepted_date< MONTHS_BETWEEN(SYSDATE,-24)
1 Table3 archive_table3 last_proposal_date< MONTHS_BETWEEN(SYSDATE,-24)
1 Table4 archive_table4 lat_modified_date< MONTHS_BETWEEN(SYSDATE,-24)
2 Table5 archive_table5 changed_date< MONTHS_BETWEEN(SYSDATE,-24)
2 Table6 archive_table6 received_date< MONTHS_BETWEEN(SYSDATE,-24)
3 Table7 archive_table7 send_date< MONTHS_BETWEEN(SYSDATE,-24)
3 Table8 archive_table8 exact_date< MONTHS_BETWEEN(SYSDATE,-24)
I want to pass ID from the JOB then the procedure should execute for all the tables with respective ID.
this is my job
DECLARE
X NUMBER;
BEGIN
SYS.DBMS_JOB.SUBMIT
( job => X
,what => 'APPS.myprocedure(1);'
,next_date => to_date('24/12/2012 00:00:00','dd/mm/yyyy hh24:mi:ss')
,interval => 'TRUNC(SYSDATE+1)'
,no_parse => FALSE
);
SYS.DBMS_OUTPUT.PUT_LINE('Job Number is: ' || to_char(x));
COMMIT;
END;
/
For example if I pass the ID as '1' through job,then the procedure should execute for
1 Table1 archive_table1 created_date< MONTHS_BETWEEN(SYSDATE,-24)
1 Table2 archive_table2 Accepted_date< MONTHS_BETWEEN(SYSDATE,-24)
1 Table3 archive_table3 last_proposal_date< MONTHS_BETWEEN(SYSDATE,-24)
INSERT INTO archive_table1
(SELECT * FROM Table1 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
DELETE FROM Table1 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
INSERT INTO archive_table2
(SELECT * FROM Table2 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
DELETE FROM Table2 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
INSERT INTO archive_table3
(SELECT * FROM Table3 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
DELETE FROM Table3 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
If I pass ID as 2 through job,then the procedure should execute for respective table names and where conditions.
I have written the following code but it's not giving expected result.
CREATE OR REPLACE PROCEDURE myprocedure (P_ID IN NUMBER)
IS
CURSOR C IS
SELECT id, table_name,archive_table_name,where_condition FROM apps_global.control_ram WHERE id=p_id
ORDER BY id,table_name;
BEGIN
FOR I IN C
LOOP
EXECUTE IMMEDIATE 'INSERT INTO '|| I.ARCHIVE_TABLE_NAME || '
(SELECT * FROM '|| I.TABLE_NAME ||' WHERE '||I.WHERE_CONDITION||)';
EXECUTE IMMEDIATE 'DELETE FROM ' ||I.TABLE_NAME|| ' WHERE '||I.WHERE_CONDITION||' ;
END LOOP;
END;
END myprocedure;
Please help me.
Thanks in advance.
|
|
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #572838 is a reply to message #572836] |
Tue, 18 December 2012 01:48   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
As you said if I write the code like this for all thirty tables.
CREATE OR REPLACE PROCEDURE arch_proc
IS
BEGIN
INSERT INTO archive_table1
(SELECT * FROM Table1 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
DELETE FROM Table1 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
INSERT INTO archive_table2
(SELECT * FROM Table2 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
DELETE FROM Table2 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
INSERT INTO archive_table3
(SELECT * FROM Table3 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
DELETE FROM Table3 WHERE created_date< MONTHS_BETWEEN(SYSDATE,-24));
commit;
END arch_proc;
/
In feature one table Table31 has to be archived and purged with the following condition.
Quote:For Table31 needs to keep 1 years worth of data in production DB,older than 1 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
then how will this script works.
Please provide the code how to handle it.
Thanks
[Updated on: Tue, 18 December 2012 01:50] by Moderator Report message to a moderator
|
|
|
|
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #572857 is a reply to message #572838] |
Tue, 18 December 2012 03:53   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
Thanks a lot for your response.
But could you please help me on
In feature one table Table31 has to be archived and purged with the following condition.
Quote:For Table31 needs to keep 1 years worth of data in production DB,older than 1 years data gets purged from production DB and
ARCHIVE table yearly in Archive DB.
then how will this script works.
Please provide the code how to handle it.
Thanks.
[Updated on: Tue, 18 December 2012 04:05] by Moderator Report message to a moderator
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #573157 is a reply to message #572860] |
Sat, 22 December 2012 04:37   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
Apology for violating the rules.
Finally I have done my PROCEDURE coding
Since I am new to scheduling jobs.
Please help me on my below requirement.
I have scheduled a job using DBMS_JOB Package; in this job I am calling a procedure.
How can we send an e-mail(alert) to the user if the job is successful (or) job fails.
If the job is successfully completed, then we have to send mail as "Job is completed successfully along with job name".
If the job fails we have to send email as "error message of the job along with job name"(why the job is failed).
This alert should be sending automatically no manual intervention.
Please help me.
CREATE OR REPLACE PROCEDURE APPS_GLOBAL.arc_procedure (P_ID IN NUMBER)
IS
CURSOR C IS SELECT id,table_name,archive_table_name,where_condition
FROM apps_global.control_ram
WHERE id = p_id
ORDER BY id, table_name;
BEGIN
FOR I IN C
LOOP
EXECUTE IMMEDIATE
'INSERT INTO '
|| I.ARCHIVE_TABLE_NAME
|| '
(SELECT * FROM '
|| I.TABLE_NAME
|| ' WHERE '
|| I.WHERE_CONDITION
|| ')';
EXECUTE IMMEDIATE
'DELETE FROM '
|| I.TABLE_NAME
|| ' WHERE '
|| I.WHERE_CONDITION
|| '';
COMMIT;
END LOOP;
EXCEPTION
WHEN OTHERS
THEN
ROLLBACK;
DBMS_OUTPUT.PUT_LINE (
'An error was encountered - ' || SQLCODE || ' -ERROR- ' || SQLERRM);
END arc_procedure;
/
This is my job.
DECLARE
X NUMBER;
BEGIN
SYS.DBMS_JOB.SUBMIT
( job => X
,what => 'APPS.arc_procedure(1);'
,next_date => to_date('05/01/2013 00:00:00','dd/mm/yyyy hh24:mi:ss')
,interval => 'TRUNC(SYSDATE+1)'
,no_parse => FALSE
);
SYS.DBMS_OUTPUT.PUT_LINE('Job Number is: ' || to_char(x));
COMMIT;
END;
/
Thanks in advance.
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #573173 is a reply to message #573168] |
Sat, 22 December 2012 23:49   |
ajaykumarkona
Messages: 264 Registered: August 2010
|
Senior Member |
|
|
Thanks for your reply.
I have gone through the link however I am not able to understand
how to send mail automatically without manual intervention
whenever the job is completed or job is failed.
Could you please provide me the code as per my requirement it will be very helpful to me.
Thanks.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Trnasfering data from one DB to another DB [message #573218 is a reply to message #573214] |
Mon, 24 December 2012 05:22  |
 |
Michel Cadot
Messages: 54252 Registered: March 2007 Location: Nanterre, France, http://...
|
Senior Member Account Moderator |
|
|
And partition is not the best way for this?
So, you, each year you will move ALL data when 90% are already there and you don't care to move them just the last year ones.
You think of a one shot move; it is not, read again: "again after one year...".
You have to think of a permanent method. If it is one shot then why not a simple backup and restore of the database (whole or just some tablespaces), you then even not have to create the target database, it will be done by RMAN?
Regards
Michel
Regards
Michel
|
|
|
|