Home » RDBMS Server » Server Utilities » SQL Loader & lookup tables issue
SQL Loader & lookup tables issue [message #74616] Fri, 21 January 2005 07:43 Go to next message
OptionTrader
Messages: 20
Registered: January 2005
Location: Praha, Czech Republic
Junior Member
I need to load about 30 million rows into one table of my DWH from a txt files. Every text files takes about 300 000 records.

Then (or perhaps during the loading?) I need to find (for 2 columns) a key for column in a small lookup dimension table and to replace a real value with its foreign key. I have 2 dim tables.

Currently I am doing it perhaps a bit clumsy this way:
I have working and target tables.
First I load a part of my data into ORACLE working table wih SQLLoader.
I then launch a procedure which open a cursor for all current records in the working table, then for every row it searches for that 2 foreign keys in 2 dim tables and then inserts a record into the target table.

SQL Loader works quite swiftly.
But the next step with that procedure is CPU and time consuming.

My question is : Is there any way to load my data and replace 2 values with theirs foreign keys in one ORACLE process step, i.e loading only.

All tables are nollogging and no redo log writing for tablespace. Still a pretty long time to get them in.

Thank you for help.

Svatopluk Solc
Re: SQL Loader & lookup tables issue [message #74620 is a reply to message #74616] Fri, 21 January 2005 14:03 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9094
Registered: November 2002
Location: California, USA
Senior Member
You can create a function that retrieves the data from your lookup table, based on another column that is loaded from your text file, like so:

create or replace function get_key 
  (p_col in lookup_table.col1%type)
  return    lookup_table.col2%type
as
  v_result  lookup_table.col2%type;
begin
  select col2
  into   v_result
  from   lookup_table 
  where  col1 = p_col;
  return v_result;
end get_key;
/


Then you can use that function in your SQL*Loader control file, like so:

load data
infile 'data.txt'
into table dwh_table
fields terminated by ','
trailing nullcols
(col1,
col2 "get_key (:col1)")
Re: SQL Loader & lookup tables issue [message #74632 is a reply to message #74616] Mon, 24 January 2005 11:21 Go to previous messageGo to next message
OptionTrader
Messages: 20
Registered: January 2005
Location: Praha, Czech Republic
Junior Member
Thank you very much Barbara.

Your help with SQL Loader will save me pretty much my time and my PC resources, as well.

Just another question on SQL Loader.

I need to join two fields from source text file and then load into one table column only.

I tried it with filler declaration but got an error message.

Thank you for help once again

Svatopluk
Re: SQL Loader & lookup tables issue [message #74633 is a reply to message #74620] Mon, 24 January 2005 11:23 Go to previous messageGo to next message
OptionTrader
Messages: 20
Registered: January 2005
Location: Praha, Czech Republic
Junior Member
Thank you very much Barbara.

Your help with SQL Loader will save me pretty much my time and my PC resources, as well.

Just another question on SQL Loader.

I need to join two fields from source text file and then load into one table column only.

I tried it with filler declaration but got an error message.

Thank you for help once again

Svatopluk
Re: SQL Loader & lookup tables issue [message #74636 is a reply to message #74632] Mon, 24 January 2005 18:55 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9094
Registered: November 2002
Location: California, USA
Senior Member
I believe the two columns to be concatenated would actually have to be columns in the table. You could add these columns, load your data, then drop the columns. Please see the following example:

-- contents of abc.txt:
col1;col2;col3;col4
value1;value2;value3;value4


-- contents of test.ctl:
LOAD DATA INFILE 'abc.txt' INTO TABLE ABC
APPEND
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
COLUMN1,
column2,
column3,
COLUMN4,
COLUMN5 ":column2 || :column3"
)


-- table for testing:
scott@ORA92> create table abc
  2    (column1 varchar2(10),
  3  	column4 varchar2(10),
  4  	column5 varchar2(20))
  5  /

Table created.


-- load data:
scott@ORA92> alter table abc add (column2 varchar2(10), column3 varchar2(10))
  2  /

Table altered.

scott@ORA92> host sqlldr scott/tiger control=test.ctl log=test.log

scott@ORA92> alter table abc drop column column2
  2  /

Table altered.

scott@ORA92> alter table abc drop column column3
  2  /

Table altered.


-- check results:
scott@ORA92> select * from abc
  2  /

COLUMN1    COLUMN4    COLUMN5
---------- ---------- --------------------
col1       col4       col2col3
value1     value4     value2value3

scott@ORA92> 
Re: SQL Loader & lookup tables issue [message #74642 is a reply to message #74636] Tue, 25 January 2005 02:12 Go to previous message
OptionTrader
Messages: 20
Registered: January 2005
Location: Praha, Czech Republic
Junior Member
Hi Barbara,

thank you very much once again for your snap help, now for this workaround.

If there is this solution to my issue only then I think that SQL Loader designers have simplified theirs job and skipped a direct concatenation Loader feature.

I find it like constructing an Indy winning car with a rung steering wheel.

Thanks and let OraGod (if EXISTS) bless you ...

Svatopluk Solc
Prague
Czech Republic
Previous Topic: internal error code
Next Topic: facing problem while exporting
Goto Forum:
  


Current Time: Fri May 17 05:13:09 CDT 2024