Home » SQL & PL/SQL » SQL & PL/SQL » External table and blank strings (Oracle 10g XE)
External table and blank strings [message #524402] Fri, 23 September 2011 08:37 Go to next message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
Hi All,

I am having a problem with the following code. I am importing some data using an external table, but the file on which the external table is built has some rows where a certain column is populated with two empty space characters.

CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'C:/...';

CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
	GCN_SEQNO NUMBER(6),
	PDM_MNAGE NUMBER(4),
	PDM_MXAGE NUMBER(4),
	PDM_MND NUMBER(18,6),
	PDM_MNDU VARCHAR(2),
	PDM_MNU NUMBER(18,6),
	PDM_MNUF VARCHAR(2),
	PDM_MXD NUMBER(18,6),
	PDM_MXDU VARCHAR(2),
	PDM_MXU NUMBER(18,6),
	PDM_MXUF VARCHAR(2),
	PDM_NTED NUMBER(18,6),
	PDM_NTEDU VARCHAR(2),
	PDM_NTEU NUMBER(18,6),
	PDM_NTEUF VARCHAR(2),
	DOSING_AGE_SOURCE_ID NUMBER(4)
)
ORGANIZATION EXTERNAL (
	default directory xtern_data_dir
	ACCESS PARAMETERS
	(
		records delimited by newline
		badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
		logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
		discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
		fields terminated by '|'
	MISSING FIELD VALUES ARE NULL
	)
	location ('RPDMMA1_TEST.TXT')
);

INSERT INTO RPDMMA1_PEDI_MSTR 
(GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID) 
(SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID FROM ET_RPDMMA1_PEDI_MSTR);


This is an except of what's in the external text file. The full text file has been attached.

000011|0030|....|000000.000000|  |000000.000000|  |0002
000011|0365|....|000000.000000|  |000000.000000|  |0002
000011|0730|....|000000.000000|  |000000.000000|  |0002
                               ^
         blank spaces may be   |
         causing error----------


Here is the error message I am receiving. I believe this is caused by the blank fields in the data.
INSERT INTO RPDMMA1_PEDI_MSTR (GCN_SEQNO,....
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout 
ORA-30653: reject limit reached 
ORA-06512: at "SYS.ORACLE_LOADER", line 52 


This is difficult to work with because the external table function does not appear to be even reading the file so it's not like I can convert the data as I'm loading into the internal database table. What are some approaches I can use to get Oracle to accept these blank columns and either populate them with blank spaces or set them to null?


Thanks!

-Winston Kotzan
Re: External table and blank strings [message #524403 is a reply to message #524402] Fri, 23 September 2011 08:44 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
Consider testing with single VARCHAR2 column external table, while you perfect the required syntax.
Re: External table and blank strings [message #524405 is a reply to message #524403] Fri, 23 September 2011 08:56 Go to previous messageGo to next message
muralikri
Messages: 638
Registered: August 2011
Location: chennai
Senior Member

Add the "reject limit unlimited" end of the script as like.

	)
	location ('RPDMMA1_TEST.TXT')
)reject limit unlimited;
Re: External table and blank strings [message #524406 is a reply to message #524405] Fri, 23 September 2011 09:05 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
SELECT COUNT(*) FROM ET_RPDMMA1_PEDI_MSTR;
what results from SQL above?
Re: External table and blank strings [message #524421 is a reply to message #524406] Fri, 23 September 2011 10:18 Go to previous messageGo to next message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
BlackSwan - here's the result of my count(*)
SQL> select count(*) from et_rpdmma1_pedi_mstr;
select count(*) from et_rpdmma1_pedi_mstr
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout 
ORA-30653: reject limit reached 
ORA-06512: at "SYS.ORACLE_LOADER", line 52 


I used muralikri's advice to set the reject limit to unlimited and that seems to work and load the data properly. Could you please explain why it works in this case? What is causing the "rejection" in the first place and are there adverse side effects to this method?

The problem is that I am writing a program to automatically generate similar code for hundreds of tables, but if there are odd exceptions like this it causes an issue.

Thanks!
Re: External table and blank strings [message #524423 is a reply to message #524421] Fri, 23 September 2011 10:29 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>I used muralikri's advice to set the reject limit to unlimited
then why does COUNT(*) throw error "ORA-30653: reject limit reached"?

>and that seems to work
so you claim without providing any proof that above is valid.

>and load the data properly.
if above is true, then nothing more needs to be done

How many records get loaded?
How many records get rejected?
Re: External table and blank strings [message #524451 is a reply to message #524423] Fri, 23 September 2011 15:50 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9090
Registered: November 2002
Location: California, USA
Senior Member
Your code works with the provided data file on my 11g Enterprise Edition, as demonstrated below. Your problem may be due to some peculiarity with 10g Express Edition. Just because the table creation executes without error does not mean that any rows were loaded. The purpose of setting reject limit unlimited is so that you can complete the creation, so that you can check the log file to see why some rows were not loaded. You need to select count(*) or select * to confirm that the rows are actually loaded. If you can select them, then attempt to insert them. You need to test these things one at a time to determine where the error is. I suspect that you will find that no rows have loaded and the error that tells you why will be in your log file, so check that first. I don't know what makes you suspect that the spaces are the cause of the problem, but you can put some test data in those spaces to see if it loads without spaces to test your theory.

SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'C:\my_oracle_files';

Directory created.

SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_log_dir AS 'C:\my_oracle_files';

Directory created.

SCOTT@orcl_11gR2> 
SCOTT@orcl_11gR2> CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
  2  	     GCN_SEQNO NUMBER(6),
  3  	     PDM_MNAGE NUMBER(4),
  4  	     PDM_MXAGE NUMBER(4),
  5  	     PDM_MND NUMBER(18,6),
  6  	     PDM_MNDU VARCHAR(2),
  7  	     PDM_MNU NUMBER(18,6),
  8  	     PDM_MNUF VARCHAR(2),
  9  	     PDM_MXD NUMBER(18,6),
 10  	     PDM_MXDU VARCHAR(2),
 11  	     PDM_MXU NUMBER(18,6),
 12  	     PDM_MXUF VARCHAR(2),
 13  	     PDM_NTED NUMBER(18,6),
 14  	     PDM_NTEDU VARCHAR(2),
 15  	     PDM_NTEU NUMBER(18,6),
 16  	     PDM_NTEUF VARCHAR(2),
 17  	     DOSING_AGE_SOURCE_ID NUMBER(4)
 18  )
 19  ORGANIZATION EXTERNAL (
 20  	     default directory xtern_data_dir
 21  	     ACCESS PARAMETERS
 22  	     (
 23  		     records delimited by newline
 24  		     badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
 25  		     logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
 26  		     discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
 27  		     fields terminated by '|'
 28  	     MISSING FIELD VALUES ARE NULL
 29  	     )
 30  	     location ('RPDMMA1_TEST.TXT')
 31  )
 32  REJECT LIMIT UNLIMITED
 33  /

Table created.

SCOTT@orcl_11gR2> SELECT * FROM ET_RPDMMA1_PEDI_MSTR
  2  /

 GCN_SEQNO  PDM_MNAGE  PDM_MXAGE    PDM_MND PD    PDM_MNU PD    PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
   PDM_MXU PD   PDM_NTED PD   PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
        11         30        364         10 07         .2 01         12 07
       .24 01          0             0                       2

        11        365        729          7 07        .14 01         12 07
       .24 01          0             0                       2

        11        730       1824       6.25 07       .125 01      12.25 07
      .245 01          0             0                       2


3 rows selected.

SCOTT@orcl_11gR2>








Re: External table and blank strings [message #524777 is a reply to message #524451] Mon, 26 September 2011 13:04 Go to previous messageGo to next message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
Hi Barbara,

Thanks for checking the script on Oracle 11g. If it is just a bug in 10g XE I'm not so concerned. I did some more extensive testing and determined that it is the blank spaces in the NUMBER field, DOSING_AGE_SOURCE_ID NUMBER(4), that is causing the problem. Could you please tell me what happens when you reproduce the following test scenario?


1) I created a new test data file RPDMMA1_CRLF_TEST.TXT (see attachment)

2) Case 1: I ran the following script (without a reject unlimited). Here is the script and results:

CREATE OR REPLACE DIRECTORY xtern_log_dir AS '[PATH]/logs/';

CREATE OR REPLACE DIRECTORY xtern_data_dir AS '[PATH]/';

CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
	GCN_SEQNO NUMBER(6),
	PDM_MNAGE NUMBER(4),
	PDM_MXAGE NUMBER(4),
	PDM_MND NUMBER(18,6),
	PDM_MNDU VARCHAR(2),
	PDM_MNU NUMBER(18,6),
	PDM_MNUF VARCHAR(2),
	PDM_MXD NUMBER(18,6),
	PDM_MXDU VARCHAR(2),
	PDM_MXU NUMBER(18,6),
	PDM_MXUF VARCHAR(2),
	PDM_NTED NUMBER(18,6),
	PDM_NTEDU VARCHAR(2),
	PDM_NTEU NUMBER(18,6),
	PDM_NTEUF VARCHAR(2),
	DOSING_AGE_SOURCE_ID NUMBER(4)
)
ORGANIZATION EXTERNAL (
	default directory xtern_data_dir
	ACCESS PARAMETERS
	(
		records delimited by newline
		badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
		logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
		discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
		fields terminated by '|'
		MISSING FIELD VALUES ARE NULL
	)
	location ('RPDMMA1_CRLF_TEST.TXT')
);

INSERT INTO RPDMMA1_PEDI_MSTR (GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID) (SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF, DOSING_AGE_SOURCE_ID FROM ET_RPDMMA1_PEDI_MSTR);


I received the following error message when running the script in SQL*Plus:
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout 
ORA-30653: reject limit reached 
ORA-06512: at "SYS.ORACLE_LOADER", line 52 


Log file says:
error processing column DOSING_AGE_SOURCE_ID in row 2 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number


In the .bad file, line "000027|0365|6569..." was rejected. Note that this was the first line with a blank DOSING_AGE_SOURCE.

3) Case 2: I ran the same script as above with the reject unlimited instruction. Here are my results:

SQL*Plus reports "4 rows created" (note that the original data file is 6 rows)

Log file says:
error processing column DOSING_AGE_SOURCE_ID in row 2 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number
error processing column DOSING_AGE_SOURCE_ID in row 5 for datafile C:/.../\RPDMMA1_CRLF_TEST.TXT
ORA-01722: invalid number


In the bad file, the following lines were rejected:
000027|0365|6569...
000032|0365|4744...


I'm not sure how to prevent that blank number field from tripping up the loader. But I'm also not sure if this is a bug or a feature in Oracle 10g XE. Would appreciate if anyone can weigh in.

Thanks so much for your help!

-Winston Kotzan
Re: External table and blank strings [message #524781 is a reply to message #524777] Mon, 26 September 2011 14:01 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9090
Registered: November 2002
Location: California, USA
Senior Member
Your script works fine as is in 11g. Based on what you have posted I agree that the blanks in a numeric field are causing the problem in your version. I suggest that you specify your data type in the lower portion of your external table creation and use nullif and blanks, so that any blanks in the numeric field are converted to null, like so:

DOSING_AGE_SOURCE_ID CHAR(255) NULLIF (DOSING_AGE_SOURCE_ID=BLANKS)

Please see the demonstration below. I have used my own directories and assumed that the table that you are trying to insert into has the same structure as your internal table. You should try selecting from the external table first, without inserting, in order to determine if the problem occurs when selecting or only when inserting, which might indicate a different problem.

SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_log_dir AS 'c:\my_oracle_files'
  2  /

Directory created.

SCOTT@orcl_11gR2> CREATE OR REPLACE DIRECTORY xtern_data_dir AS 'c:\my_oracle_files'
  2  /

Directory created.

SCOTT@orcl_11gR2> CREATE TABLE ET_RPDMMA1_PEDI_MSTR (
  2  	     GCN_SEQNO		   NUMBER(6),
  3  	     PDM_MNAGE		   NUMBER(4),
  4  	     PDM_MXAGE		   NUMBER(4),
  5  	     PDM_MND		   NUMBER(18,6),
  6  	     PDM_MNDU		   VARCHAR(2),
  7  	     PDM_MNU		   NUMBER(18,6),
  8  	     PDM_MNUF		   VARCHAR(2),
  9  	     PDM_MXD		   NUMBER(18,6),
 10  	     PDM_MXDU		   VARCHAR(2),
 11  	     PDM_MXU		   NUMBER(18,6),
 12  	     PDM_MXUF		   VARCHAR(2),
 13  	     PDM_NTED		   NUMBER(18,6),
 14  	     PDM_NTEDU		   VARCHAR(2),
 15  	     PDM_NTEU		   NUMBER(18,6),
 16  	     PDM_NTEUF		   VARCHAR(2),
 17  	     DOSING_AGE_SOURCE_ID  NUMBER(4)
 18  )
 19  ORGANIZATION EXTERNAL (
 20  	     default directory xtern_data_dir
 21  	     ACCESS PARAMETERS
 22  	     (
 23  		     records delimited by newline
 24  		     badfile xtern_log_dir:'RPDMMA1_PEDI_MSTR.bad'
 25  		     logfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
 26  		     discardfile xtern_log_dir:'RPDMMA1_PEDI_MSTR'
 27  		     fields terminated by '|'
 28  		     MISSING FIELD VALUES ARE NULL
 29  		       (GCN_SEQNO,
 30  			PDM_MNAGE,
 31  			PDM_MXAGE,
 32  			PDM_MND  ,
 33  			PDM_MNDU ,
 34  			PDM_MNU  ,
 35  			PDM_MNUF ,
 36  			PDM_MXD  ,
 37  			PDM_MXDU ,
 38  			PDM_MXU  ,
 39  			PDM_MXUF ,
 40  			PDM_NTED ,
 41  			PDM_NTEDU,
 42  			PDM_NTEU ,
 43  			PDM_NTEUF,
 44  			DOSING_AGE_SOURCE_ID CHAR(255) NULLIF (DOSING_AGE_SOURCE_ID=BLANKS))
 45  	     )
 46  	     location ('RPDMMA1_CRLF_TEST.TXT')
 47  )
 48  reject limit unlimited
 49  /

Table created.

SCOTT@orcl_11gR2> SELECT COUNT (*) FROM ET_RPDMMA1_PEDI_MSTR
  2  /

  COUNT(*)
----------
         6

1 row selected.

SCOTT@orcl_11gR2> SELECT * FROM ET_RPDMMA1_PEDI_MSTR
  2  /

 GCN_SEQNO  PDM_MNAGE  PDM_MXAGE    PDM_MND PD    PDM_MNU PD    PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
   PDM_MXU PD   PDM_NTED PD   PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
        27         49        364          5 10    .076923 01         10 10
   .153846 01          0             0                       2

        27        365       6569    1000000       1000000       1000000
   1000000       1000000       1000000

        32         30         48        2.5 10       .025 01          5 10
       .05 01          0             0                       3

        32         49        364        2.5 10       .025 01         10 10
        .1 01          0             0                       3

        32        365       4744    1000000       1000000       1000000
   1000000       1000000       1000000

        32       4745       6569         50 11         .5 02        800 11
         8 02          0             0                       3


6 rows selected.

SCOTT@orcl_11gR2> CREATE TABLE RPDMMA1_PEDI_MSTR (
  2  	     GCN_SEQNO		   NUMBER(6),
  3  	     PDM_MNAGE		   NUMBER(4),
  4  	     PDM_MXAGE		   NUMBER(4),
  5  	     PDM_MND		   NUMBER(18,6),
  6  	     PDM_MNDU		   VARCHAR(2),
  7  	     PDM_MNU		   NUMBER(18,6),
  8  	     PDM_MNUF		   VARCHAR(2),
  9  	     PDM_MXD		   NUMBER(18,6),
 10  	     PDM_MXDU		   VARCHAR(2),
 11  	     PDM_MXU		   NUMBER(18,6),
 12  	     PDM_MXUF		   VARCHAR(2),
 13  	     PDM_NTED		   NUMBER(18,6),
 14  	     PDM_NTEDU		   VARCHAR(2),
 15  	     PDM_NTEU		   NUMBER(18,6),
 16  	     PDM_NTEUF		   VARCHAR(2),
 17  	     DOSING_AGE_SOURCE_ID  NUMBER(4)
 18  )
 19  /

Table created.

SCOTT@orcl_11gR2> INSERT INTO RPDMMA1_PEDI_MSTR
  2    (GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF, PDM_MXD,
  3  	PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU, PDM_NTEUF,
  4  	DOSING_AGE_SOURCE_ID)
  5  (SELECT GCN_SEQNO, PDM_MNAGE, PDM_MXAGE, PDM_MND, PDM_MNDU, PDM_MNU, PDM_MNUF,
  6  	     PDM_MXD, PDM_MXDU, PDM_MXU, PDM_MXUF, PDM_NTED, PDM_NTEDU, PDM_NTEU,
  7  	     PDM_NTEUF, DOSING_AGE_SOURCE_ID
  8   FROM   ET_RPDMMA1_PEDI_MSTR)
  9  /

6 rows created.

SCOTT@orcl_11gR2> SELECT * FROM RPDMMA1_PEDI_MSTR
  2  /

 GCN_SEQNO  PDM_MNAGE  PDM_MXAGE    PDM_MND PD    PDM_MNU PD    PDM_MXD PD
---------- ---------- ---------- ---------- -- ---------- -- ---------- --
   PDM_MXU PD   PDM_NTED PD   PDM_NTEU PD DOSING_AGE_SOURCE_ID
---------- -- ---------- -- ---------- -- --------------------
        27         49        364          5 10    .076923 01         10 10
   .153846 01          0             0                       2

        27        365       6569    1000000       1000000       1000000
   1000000       1000000       1000000

        32         30         48        2.5 10       .025 01          5 10
       .05 01          0             0                       3

        32         49        364        2.5 10       .025 01         10 10
        .1 01          0             0                       3

        32        365       4744    1000000       1000000       1000000
   1000000       1000000       1000000

        32       4745       6569         50 11         .5 02        800 11
         8 02          0             0                       3


6 rows selected.

SCOTT@orcl_11gR2>




Re: External table and blank strings [message #524783 is a reply to message #524781] Mon, 26 September 2011 14:29 Go to previous messageGo to next message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
Hi Barbara,

That seems to have resolved the problem. There must be a quirk in 10g XE that causes the error with the spaces.

Thanks!
Re: External table and blank strings [message #524937 is a reply to message #524783] Tue, 27 September 2011 12:04 Go to previous messageGo to next message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
FYI, I tried the above code in message #524451 that you said worked in 11g on my Oracle 11g XE (I upgraded to the newer version). I received the same errors, but adding the "NULLIF (DOSING_AGE_SOURCE_ID=BLANKS)" code fixes it. Do you think this is this likely a bug in the XE editions or Oracle, or may it have something to do with a default setting in my XE installation?
Re: External table and blank strings [message #524940 is a reply to message #524937] Tue, 27 September 2011 13:02 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9090
Registered: November 2002
Location: California, USA
Senior Member
I don't know. I don't currently have XE installed for comparison. The default trim_spec values may be different or some such thing. You miight try:

fields terminated by ... optionally enclosed by ... LRTRIM

where lrtrim is the value for the trim_spec clause, which would trim leading and trailing spaces. Please see the following section of the online documentation for explanation.

http://download.oracle.com/docs/cd/E11882_01/server.112/e22490/et_params.htm#SUTIL1411
Re: External table and blank strings [message #524941 is a reply to message #524402] Tue, 27 September 2011 14:09 Go to previous message
wakproductions_fdb
Messages: 15
Registered: September 2011
Location: Indiana
Junior Member
Thanks! LTRIM works great! That simple fix would make a lot of efficiency for my project as I am using a script to generate a lot of this repetitive code for a database of a few hundred tables.
Previous Topic: Need to run the Job (2 Merged)
Next Topic: UNPIVOT in Oracle 11g
Goto Forum:
  


Current Time: Thu Apr 25 17:16:35 CDT 2024