Home » SQL & PL/SQL » SQL & PL/SQL » External table and blank strings (Oracle 10g XE)
External table and blank strings [message #524402] |
Fri, 23 September 2011 08:37 |
|
wakproductions_fdb
Messages: 15 Registered: September 2011 Location: Indiana
|
Junior Member |
|
|
Hi All,
I am having a problem with the following code. I am importing some data using an external table, but the file on which the external table is built has some rows where a certain column is populated with two empty space characters.
CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'C:/...';
CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
GCN_SEQNO NUMBER(6),
PDM_MNAGE NUMBER(4),
PDM_MXAGE NUMBER(4),
PDM_MND NUMBER(18,6),
PDM_MNDU VARCHAR(2),
PDM_MNU NUMBER(18,6),
PDM_MNUF VARCHAR(2),
PDM_MXD NUMBER(18,6),
PDM_MXDU VARCHAR(2),
PDM_MXU NUMBER(18,6),
PDM_MXUF VARCHAR(2),
PDM_NTED NUMBER(18,6),
PDM_NTEDU VARCHAR(2),
PDM_NTEU NUMBER(18,6),
PDM_NTEUF VARCHAR(2),
DOSING_AGE_SOURCE_ID NUMBER(4)
)
ORGANIZATION EXTERNAL (
default directory xtern_data_dir
ACCESS PARAMETERS
(
records delimited by newline
badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
fields terminated by '|'
MISSING FIELD VALUES ARE NULL
)
location ('RPDMMA1_TEST.TXT')
);
INSERT INTO RPDMMA1_PEDI_MSTR
(GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID)
(SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID FROM ET_RPDMMA1_PEDI_MSTR);
This is an except of what's in the external text file. The full text file has been attached.
000011|0030|....|000000.000000| |000000.000000| |0002
000011|0365|....|000000.000000| |000000.000000| |0002
000011|0730|....|000000.000000| |000000.000000| |0002
^
blank spaces may be |
causing error----------
Here is the error message I am receiving. I believe this is caused by the blank fields in the data.
INSERT INTO RPDMMA1_PEDI_MSTR (GCN_SEQNO,....
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-30653: reject limit reached
ORA-06512: at "SYS.ORACLE_LOADER", line 52
This is difficult to work with because the external table function does not appear to be even reading the file so it's not like I can convert the data as I'm loading into the internal database table. What are some approaches I can use to get Oracle to accept these blank columns and either populate them with blank spaces or set them to null?
Thanks!
-Winston Kotzan
|
|
|
|
|
|
|
|
Re: External table and blank strings [message #524451 is a reply to message #524423] |
Fri, 23 September 2011 15:50 |
|
Barbara Boehmer
Messages: 9090 Registered: November 2002 Location: California, USA
|
Senior Member |
|
|
Your code works with the provided data file on my 11g Enterprise Edition, as demonstrated below. Your problem may be due to some peculiarity with 10g Express Edition. Just because the table creation executes without error does not mean that any rows were loaded. The purpose of setting reject limit unlimited is so that you can complete the creation, so that you can check the log file to see why some rows were not loaded. You need to select count(*) or select * to confirm that the rows are actually loaded. If you can select them, then attempt to insert them. You need to test these things one at a time to determine where the error is. I suspect that you will find that no rows have loaded and the error that tells you why will be in your log file, so check that first. I don't know what makes you suspect that the spaces are the cause of the problem, but you can put some test data in those spaces to see if it loads without spaces to test your theory.
SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'C:\my_oracle_files';
Directory created.
SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_log_dir AS 'C:\my_oracle_files';
Directory created.
SCOTT@orcl_11gR2>
SCOTT@orcl_11gR2> CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
2 GCN_SEQNO NUMBER(6),
3 PDM_MNAGE NUMBER(4),
4 PDM_MXAGE NUMBER(4),
5 PDM_MND NUMBER(18,6),
6 PDM_MNDU VARCHAR(2),
7 PDM_MNU NUMBER(18,6),
8 PDM_MNUF VARCHAR(2),
9 PDM_MXD NUMBER(18,6),
10 PDM_MXDU VARCHAR(2),
11 PDM_MXU NUMBER(18,6),
12 PDM_MXUF VARCHAR(2),
13 PDM_NTED NUMBER(18,6),
14 PDM_NTEDU VARCHAR(2),
15 PDM_NTEU NUMBER(18,6),
16 PDM_NTEUF VARCHAR(2),
17 DOSING_AGE_SOURCE_ID NUMBER(4)
18 )
19 ORGANIZATION EXTERNAL (
20 default directory xtern_data_dir
21 ACCESS PARAMETERS
22 (
23 records delimited by newline
24 badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
25 logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
26 discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
27 fields terminated by '|'
28 MISSING FIELD VALUES ARE NULL
29 )
30 location ('RPDMMA1_TEST.TXT')
31 )
32 REJECT LIMIT UNLIMITED
33 /
Table created.
SCOTT@orcl_11gR2> SELECT * FROM ET_RPDMMA1_PEDI_MSTR
2 /
GCN_SEQNO PDM_MNAGE PDM_MXAGE PDM_MND PD PDM_MNU PD PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
PDM_MXU PD PDM_NTED PD PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
11 30 364 10 07 .2 01 12 07
.24 01 0 0 2
11 365 729 7 07 .14 01 12 07
.24 01 0 0 2
11 730 1824 6.25 07 .125 01 12.25 07
.245 01 0 0 2
3 rows selected.
SCOTT@orcl_11gR2>
|
|
|
Re: External table and blank strings [message #524777 is a reply to message #524451] |
Mon, 26 September 2011 13:04 |
|
wakproductions_fdb
Messages: 15 Registered: September 2011 Location: Indiana
|
Junior Member |
|
|
Hi Barbara,
Thanks for checking the script on Oracle 11g. If it is just a bug in 10g XE I'm not so concerned. I did some more extensive testing and determined that it is the blank spaces in the NUMBER field, DOSING_AGE_SOURCE_ID NUMBER(4), that is causing the problem. Could you please tell me what happens when you reproduce the following test scenario?
1) I created a new test data file RPDMMA1_CRLF_TEST.TXT (see attachment)
2) Case 1: I ran the following script (without a reject unlimited). Here is the script and results:
CREATE OR REPLACE DIRECTORY xtern_log_dir AS '[PATH]/logs/';
CREATE OR REPLACE DIRECTORY xtern_data_dir AS '[PATH]/';
CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
GCN_SEQNO NUMBER(6),
PDM_MNAGE NUMBER(4),
PDM_MXAGE NUMBER(4),
PDM_MND NUMBER(18,6),
PDM_MNDU VARCHAR(2),
PDM_MNU NUMBER(18,6),
PDM_MNUF VARCHAR(2),
PDM_MXD NUMBER(18,6),
PDM_MXDU VARCHAR(2),
PDM_MXU NUMBER(18,6),
PDM_MXUF VARCHAR(2),
PDM_NTED NUMBER(18,6),
PDM_NTEDU VARCHAR(2),
PDM_NTEU NUMBER(18,6),
PDM_NTEUF VARCHAR(2),
DOSING_AGE_SOURCE_ID NUMBER(4)
)
ORGANIZATION EXTERNAL (
default directory xtern_data_dir
ACCESS PARAMETERS
(
records delimited by newline
badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
fields terminated by '|'
MISSING FIELD VALUES ARE NULL
)
location ('RPDMMA1_CRLF_TEST.TXT')
);
INSERT INTO RPDMMA1_PEDI_MSTR (GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID) (SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID FROM ET_RPDMMA1_PEDI_MSTR);
I received the following error message when running the script in SQL*Plus:
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-30653: reject limit reached
ORA-06512: at "SYS.ORACLE_LOADER", line 52
Log file says:
error processing column DOSING_AGE_SOURCE_ID in row 2 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number
In the .bad file, line "000027|0365|6569..." was rejected. Note that this was the first line with a blank DOSING_AGE_SOURCE.
3) Case 2: I ran the same script as above with the reject unlimited instruction. Here are my results:
SQL*Plus reports "4 rows created" (note that the original data file is 6 rows)
Log file says:
error processing column DOSING_AGE_SOURCE_ID in row 2 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number
error processing column DOSING_AGE_SOURCE_ID in row 5 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number
In the bad file, the following lines were rejected:
000027|0365|6569...
000032|0365|4744...
I'm not sure how to prevent that blank number field from tripping up the loader. But I'm also not sure if this is a bug or a feature in Oracle 10g XE. Would appreciate if anyone can weigh in.
Thanks so much for your help!
-Winston Kotzan
|
|
|
Re: External table and blank strings [message #524781 is a reply to message #524777] |
Mon, 26 September 2011 14:01 |
|
Barbara Boehmer
Messages: 9090 Registered: November 2002 Location: California, USA
|
Senior Member |
|
|
Your script works fine as is in 11g. Based on what you have posted I agree that the blanks in a numeric field are causing the problem in your version. I suggest that you specify your data type in the lower portion of your external table creation and use nullif and blanks, so that any blanks in the numeric field are converted to null, like so:
DOSING_AGE_SOURCE_ID CHAR(255) NULLIF (DOSING_AGE_SOURCE_ID=BLANKS)
Please see the demonstration below. I have used my own directories and assumed that the table that you are trying to insert into has the same structure as your internal table. You should try selecting from the external table first, without inserting, in order to determine if the problem occurs when selecting or only when inserting, which might indicate a different problem.
SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_log_dir AS 'c:\my_oracle_files'
2 /
Directory created.
SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'c:\my_oracle_files'
2 /
Directory created.
SCOTT@orcl_11gR2> CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
2 GCN_SEQNO NUMBER(6),
3 PDM_MNAGE NUMBER(4),
4 PDM_MXAGE NUMBER(4),
5 PDM_MND NUMBER(18,6),
6 PDM_MNDU VARCHAR(2),
7 PDM_MNU NUMBER(18,6),
8 PDM_MNUF VARCHAR(2),
9 PDM_MXD NUMBER(18,6),
10 PDM_MXDU VARCHAR(2),
11 PDM_MXU NUMBER(18,6),
12 PDM_MXUF VARCHAR(2),
13 PDM_NTED NUMBER(18,6),
14 PDM_NTEDU VARCHAR(2),
15 PDM_NTEU NUMBER(18,6),
16 PDM_NTEUF VARCHAR(2),
17 DOSING_AGE_SOURCE_ID NUMBER(4)
18 )
19 ORGANIZATION EXTERNAL (
20 default directory xtern_data_dir
21 ACCESS PARAMETERS
22 (
23 records delimited by newline
24 badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
25 logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
26 discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
27 fields terminated by '|'
28 MISSING FIELD VALUES ARE NULL
29 (GCN_SEQNO,
30 PDM_MNAGE,
31 PDM_MXAGE,
32 PDM_MND ,
33 PDM_MNDU ,
34 PDM_MNU ,
35 PDM_MNUF ,
36 PDM_MXD ,
37 PDM_MXDU ,
38 PDM_MXU ,
39 PDM_MXUF ,
40 PDM_NTED ,
41 PDM_NTEDU,
42 PDM_NTEU ,
43 PDM_NTEUF,
44 DOSING_AGE_SOURCE_ID CHAR(255) NULLIF (DOSING_AGE_SOURCE_ID=BLANKS))
45 )
46 location ('RPDMMA1_CRLF_TEST.TXT')
47 )
48 reject limit unlimited
49 /
Table created.
SCOTT@orcl_11gR2> SELECT COUNT (*) FROM ET_RPDMMA1_PEDI_MSTR
2 /
COUNT(*)
----------
6
1 row selected.
SCOTT@orcl_11gR2> SELECT * FROM ET_RPDMMA1_PEDI_MSTR
2 /
GCN_SEQNO PDM_MNAGE PDM_MXAGE PDM_MND PD PDM_MNU PD PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
PDM_MXU PD PDM_NTED PD PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
27 49 364 5 10 .076923 01 10 10
.153846 01 0 0 2
27 365 6569 1000000 1000000 1000000
1000000 1000000 1000000
32 30 48 2.5 10 .025 01 5 10
.05 01 0 0 3
32 49 364 2.5 10 .025 01 10 10
.1 01 0 0 3
32 365 4744 1000000 1000000 1000000
1000000 1000000 1000000
32 4745 6569 50 11 .5 02 800 11
8 02 0 0 3
6 rows selected.
SCOTT@orcl_11gR2> CREATE TABLE RPDMMA1_PEDI_MSTR (
2 GCN_SEQNO NUMBER(6),
3 PDM_MNAGE NUMBER(4),
4 PDM_MXAGE NUMBER(4),
5 PDM_MND NUMBER(18,6),
6 PDM_MNDU VARCHAR(2),
7 PDM_MNU NUMBER(18,6),
8 PDM_MNUF VARCHAR(2),
9 PDM_MXD NUMBER(18,6),
10 PDM_MXDU VARCHAR(2),
11 PDM_MXU NUMBER(18,6),
12 PDM_MXUF VARCHAR(2),
13 PDM_NTED NUMBER(18,6),
14 PDM_NTEDU VARCHAR(2),
15 PDM_NTEU NUMBER(18,6),
16 PDM_NTEUF VARCHAR(2),
17 DOSING_AGE_SOURCE_ID NUMBER(4)
18 )
19 /
Table created.
SCOTT@orcl_11gR2> INSERT INTO RPDMMA1_PEDI_MSTR
2 (GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD,
3 PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF,
4 DOSING_AGE_SOURCE_ID)
5 (SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF,
6 PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU,
7 PDM_NTEUF, DOSING_AGE_SOURCE_ID
8 FROM ET_RPDMMA1_PEDI_MSTR)
9 /
6 rows created.
SCOTT@orcl_11gR2> SELECT * FROM RPDMMA1_PEDI_MSTR
2 /
GCN_SEQNO PDM_MNAGE PDM_MXAGE PDM_MND PD PDM_MNU PD PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
PDM_MXU PD PDM_NTED PD PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
27 49 364 5 10 .076923 01 10 10
.153846 01 0 0 2
27 365 6569 1000000 1000000 1000000
1000000 1000000 1000000
32 30 48 2.5 10 .025 01 5 10
.05 01 0 0 3
32 49 364 2.5 10 .025 01 10 10
.1 01 0 0 3
32 365 4744 1000000 1000000 1000000
1000000 1000000 1000000
32 4745 6569 50 11 .5 02 800 11
8 02 0 0 3
6 rows selected.
SCOTT@orcl_11gR2>
|
|
|
|
|
|
|
Goto Forum:
Current Time: Thu Apr 25 17:16:35 CDT 2024
|