| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
 |
 | |||
| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
|
| |||
Home -> Community -> Mailing Lists -> Oracle-L -> RE: Flat file generation integrity ideas...
Chris,
Our favourite footer (at a previous project) was:
#EOF#<count_of_detail_records><sum_of_numerical_column1><sum_of_numerical_column2>...
Sometimes there were no numerical columns, sometimes 1 and sometimes 12 or more. In most cases we were the recipients of the files, so we wrote a utility that created the footer based on an input specification. When we received a file we would trim the header and footer into separate files, then run our footer generation script against the detail records. We then used diff to compare our footer with the supplied footer. If we didn't get the same result as the source system then something was wrong. Since we didn't know and didn't care about how the source system created the original footer it also meant that effectively the data was verified twice.
For our situation the utility was worthwhile - we ended up with over 100 files using this validation technique from all sorts of source systems and in each case we just created a 1 line text file detailing the location of the numerical columns and some info or whether the file was fixed width or csv.
cheers,
Mark
"Grabowy,
Chris" To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
<cgrabowy_at_fcg. cc:
com> Subject: RE: Flat file generation integrity ideas...
Sent by:
root_at_fatcity.c
om
25/10/2002
05:01
Please respond
to ORACLE-L
I will have to keep those in mind, if I ever get back onto a UNIX platform.
Right now, I'm sticking to Tom's suggestion because I religiously follow KISS. Thanks!!
-----Original Message-----
Sent: Thursday, October 24, 2002 2:19 PM
To: Multiple recipients of list ORACLE-L
Chris - I would consider checksums. Many systems have built-in checksum functions. On Unix these are cksum and sum. You could probably write a simple one that would work on your various systems. Search for checksum and somewhere on the Internet is probably sample code for one in a language that will work for you.
Dennis Williams
DBA, 40%OCP
Lifetouch, Inc.
dwilliams_at_lifetouch.com
-----Original Message-----
Sent: Thursday, October 24, 2002 12:35 PM
To: Multiple recipients of list ORACLE-L
I have to create packages that will generate several flat files of data from tables that will be sent to other systems to be processed.
I am looking for ideas on how to ensure data integrity in the flat files.
For example, the expected record count is stored on the first line of the file to ensure that the correct amount of records was received.
The systems group is chartered to ensure the flat files are correctly FTPed between systems, so that's covered.
I just worry that if "somehow" a flat file is scrambled then the scrambled data is loaded into the database, therefore corrupting it.
At this phase, XML is not an option
I keep thinking that some sort of CRC should be stored with each line in the flat file. And then before the line is loaded into the database, the CRC is compared against the generated CRC of the just read line. Has anyone done anything like this? Any examples out there?
Many TIA!!
-- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: DENNIS WILLIAMS INET: DWILLIAMS_at_LIFETOUCH.COM Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Grabowy, Chris INET: cgrabowy_at_fcg.com Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).Received on Thu Oct 24 2002 - 19:43:50 CDT
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Privileged/Confidential information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply e-mail or by telephone on (61 3) 9612-6999. Please advise immediately if you or your employer does not consent to Internet e-mail for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of Transurban City Link Ltd shall be understood as neither given nor endorsed by it. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Mark Richard INET: mrichard_at_transurban.com.au Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
 |
 |