Home » SQL & PL/SQL » SQL & PL/SQL » Updating CLOB (1-g R2)
Updating CLOB [message #504100] Thu, 21 April 2011 09:54 Go to next message
amerar
Messages: 18
Registered: April 2011
Junior Member

Ok, CLOB columns are such a hassle.

I have a variable in my script: v_field1 VARCHAR2(32000);

This is part of a cursor record:

v_mf_table IS TABLE OF mf_detail%ROWTYPE INDEX BY BINARY_INTEGER;
v_mf_record v_mf_table;

I use a FORALL to insert the data into a table:

FORALL x IN v_mf_record.FIRST .. v_mf_record.COUNT
INSERT INTO monthly_mf_snapshot VALUES v_mf_record(x);

BUT! v_field1 is > 4000 characters. Does this trash my changes of using FORALL? Do I need to deal with 4k chunks in an UPDATE instead?

Help Please.



Re: Updating CLOB [message #504104 is a reply to message #504100] Thu, 21 April 2011 10:04 Go to previous messageGo to next message
cookiemonster
Messages: 13975
Registered: September 2008
Location: Rainy Manchester
Senior Member
You make absolutely no mention in your post of how you are using v_field1 in the code. Telling us that would be a necessary starting point.
Re: Updating CLOB [message #504105 is a reply to message #504104] Thu, 21 April 2011 10:12 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
My bad, sorry:

FOR x_rec IN mf_monthly(v_mf_record(v_rec).ticker) LOOP
v_monthly_data := v_monthly_data || x_rec.monthly_data || '+++';
END LOOP;

Basically concatenating a long string, making it delimited for the application to parse. If it is > 4k, does that trash my FORALL statement? The table I am inserting to is 100+ columns. I'd hate to have to list them out.
Re: Updating CLOB [message #504106 is a reply to message #504100] Thu, 21 April 2011 10:15 Go to previous messageGo to next message
Michel Cadot
Messages: 68773
Registered: March 2007
Location: Saint-Maur, France, https...
Senior Member
Account Moderator
desc monthly_mf_snapshot

Please read OraFAQ Forum Guide, especially "How to format your post?" section.
Make sure that lines of code do not exceed 80 characters when you format.
Indent the code, use code tags and align the columns in result.
Use the "Preview Message" button to verify.
Also always post your Oracle version, with 4 decimals.

Regards
Michel
Re: Updating CLOB [message #504109 is a reply to message #504106] Thu, 21 April 2011 10:23 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
You really ready for this?



Name Null? Type
----------------------------------------- -------- ----------------------------
TICKER VARCHAR2(6)
FUND_NAME VARCHAR2(100)
FAMILY_NAME VARCHAR2(100)
OBJ_DESCR VARCHAR2(100)
OBJ_PEER VARCHAR2(100)
BETA_3YR NUMBER
ALPHA_3YR NUMBER
R_SQUARE_3YR NUMBER
STD_DEV_3YR NUMBER
SHARPE_3YR NUMBER
BETA_5YR NUMBER
ALPHA_5YR NUMBER
R_SQUARE_5YR NUMBER
STD_DEV_5YR NUMBER
SHARPE_5YR NUMBER
BETA_10YR NUMBER
ALPHA_10YR NUMBER
R_SQUARE_10YR NUMBER
STD_DEV_10YR NUMBER
SHARPE_10YR NUMBER
BETA_VS_SP500 NUMBER
BETA_VS_LEHMAN NUMBER
BETA_VS_EAFE NUMBER
MATURITY_AVG NUMBER
MODIFIED_DURATION NUMBER
OVERALL_RANK NUMBER
RISK_RANK NUMBER
GROWTH_PERS_1YR NUMBER
GROWTH_PERS_5YR NUMBER
LARGE_GROWTH VARCHAR2(10)
LARGE_VALUE VARCHAR2(10)
SMALL_GROWTH VARCHAR2(10)
SMALL_VALUE VARCHAR2(10)
FOREIGN_STOCK VARCHAR2(10)
EMERGING_MARKET VARCHAR2(10)
PREC_METALS VARCHAR2(10)
INTER_BOND VARCHAR2(10)
FOREIGN_BOND VARCHAR2(10)
HIGH_YLD_BOND VARCHAR2(10)
MAX_LOAD_PERC VARCHAR2(10)
MIN_LOAD_PERC VARCHAR2(10)
DEFERRED_LOAD_PERC VARCHAR2(10)
FEE_12B1_PERC VARCHAR2(10)
MAX_MGMT_FEE_PERC VARCHAR2(10)
REDEMPTION_FEE_PERC VARCHAR2(10)
EXPENSE_RATIO VARCHAR2(10)
AS_OF_DATE DATE
STOCKS_PERC NUMBER
CASH_PERC NUMBER
BONDS_PERC NUMBER
PREFERRED_PERC NUMBER
CONVERTIBLE_PERC NUMBER
FOREIGN_STOCK_PERC NUMBER
FOREIGN_BOND_PERC NUMBER
FOREIGN_HEDGED NUMBER
OTHER_PERC NUMBER
TOTAL_ISSUES NUMBER
PRICE_EARNINGS_AVG NUMBER
PRICE_BOOK_AVG NUMBER
EPS_GROWTH_AVG NUMBER
MARKET_VALUE_AVG NUMBER
A_PERCENTAGE NUMBER
AA_PERCENTAGE NUMBER
AAA_PERCENTAGE NUMBER
BBB_PERCENTAGE NUMBER
BB_PERCENTAGE NUMBER
BELOW_B_PERC NUMBER
NO_RATING_PERC NUMBER
COUPON_AVG NUMBER
NAV NUMBER
EQUITY VARCHAR2(1)
BOND VARCHAR2(1)
NET_ASSETS VARCHAR2(10)
FY NUMBER
FY_1 NUMBER
FY_2 NUMBER
FY_3 NUMBER
FY_4 NUMBER
FY_5 NUMBER
FY_6 NUMBER
FY_7 NUMBER
FY_8 NUMBER
FY_9 NUMBER
FY_10 NUMBER
PORTFOLIO_TURNOVER NUMBER
ADDRESS VARCHAR2(200)
TOLL_FREE_PHONE VARCHAR2(15)
LP_AS_OF_DATE VARCHAR2(10)
TOTAL_RETURN_YTD NUMBER
TOTAL_RETURN_1MT NUMBER
TOTAL_RETURN_3MT NUMBER
TOTAL_RETURN_6MT NUMBER
TOTAL_RETURN_1YR NUMBER
TOTAL_RETURN_3YR NUMBER
TOTAL_RETURN_5YR NUMBER
TOTAL_RETURN_10YR NUMBER
TOTAL_RETURN_15YR NUMBER
TOTAL_RETURN_20YR NUMBER
RANK_VS_OBJ_YTD NUMBER
RANK_VS_OBJ_1MT NUMBER
RANK_VS_OBJ_3MT NUMBER
RANK_VS_OBJ_6MT NUMBER
RANK_VS_OBJ_1YR NUMBER
RANK_VS_OBJ_3YR NUMBER
RANK_VS_OBJ_5YR NUMBER
RANK_VS_OBJ_10YR NUMBER
RANK_VS_OBJ_15YR NUMBER
RANK_VS_OBJ_20YR NUMBER
BULL_MARKET NUMBER
PREVIOUS_BEAR NUMBER
PREVIOUS_BULL NUMBER
DATE_ENTERED VARCHAR2(10)
CUR_NAV_DATE VARCHAR2(10)
CUR_NAV NUMBER
NAV_CHANGE NUMBER
YIELD_PERCENT NUMBER
SEC_YIELD_PERCENT NUMBER
LOAD_ADJ_RETURN_1YR NUMBER
LOAD_ADJ_RETURN_5YR NUMBER
LOAD_ADJ_RETURN_10YR NUMBER
LOAD_ADJ_RETURN_INCEPT NUMBER
MANAGER VARCHAR2(50)
MGR_START_DATE VARCHAR2(10)
MGR_RATING NUMBER
INCEPTION_DATE VARCHAR2(10)
SHAREHOLDERS NUMBER
DIVIDEND_FREQ VARCHAR2(1)
MIN_INIT_INVESTMENT VARCHAR2(10)
MIN_SUBS_INVESTMENT VARCHAR2(10)
IRA_AVAILABLE VARCHAR2(1)
MIN_INIT_IRA NUMBER
CHECK_WRITING VARCHAR2(10)
INVEST_PHONE VARCHAR2(10)
EXCHANGE_PHONE VARCHAR2(10)
REDEEM_PHONE VARCHAR2(10)
AUTO_INVEST VARCHAR2(10)
REDEEM_WIRE VARCHAR2(10)
WIRE_FEE VARCHAR2(10)
WIRE_TRANSFER VARCHAR2(10)
CLASS_RANK NUMBER
HOLDINGS_DATA CLOB
SECTOR_DATA CLOB
QUARTERLY_DATA CLOB
YEARLY_DATA CLOB
MONTHLY_DATA CLOB
FUND_DESCRIPTION VARCHAR2(4000)
Re: Updating CLOB [message #504110 is a reply to message #504109] Thu, 21 April 2011 10:26 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
I guess I could check the length of each time through the loop, and if the length of the variable is > 4k, then insert the record, then update the CLOB column using a separate loop....cheap resolution, but it will do.
Re: Updating CLOB [message #504113 is a reply to message #504109] Thu, 21 April 2011 10:29 Go to previous messageGo to next message
Michel Cadot
Messages: 68773
Registered: March 2007
Location: Saint-Maur, France, https...
Senior Member
Account Moderator
Split the value into 4000 character values that you concatenate.
Or better insert an empty clob and then update it with the variable.

Regards
Michel
Re: Updating CLOB [message #504118 is a reply to message #504113] Thu, 21 April 2011 10:34 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Can I do a single update even though it is greater than 4k, or do I need to split to regardless and I assume use WRITEAPPEND of DBMS_LOB?

[Updated on: Thu, 21 April 2011 10:35]

Report message to a moderator

Re: Updating CLOB [message #504122 is a reply to message #504118] Thu, 21 April 2011 10:46 Go to previous messageGo to next message
Michel Cadot
Messages: 68773
Registered: March 2007
Location: Saint-Maur, France, https...
Senior Member
Account Moderator
Yes, yes, this is what I should try.

Regards
Michel
Re: Updating CLOB [message #504158 is a reply to message #504122] Thu, 21 April 2011 13:05 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Been playing around. This is messed up and confusing. See my table definition above, I have CLOB columns in the table. I can issue a direct insert into that table of a value = 32000K.

However, if in my PL/SQL code I have this: v_mf_record(v_rec).monthly_data := RPAD('A',32000,'A'), where v_mf_record(v_rec).monthly_data is a ROWTYPE based on the table above, I get an error.

So, why will a direct insert statement to the table work, and assigning a variable the same length value, which is based on a CLOB column, not work? I'm a bit lost there.
Re: Updating CLOB [message #504161 is a reply to message #504158] Thu, 21 April 2011 13:08 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>So, why will a direct insert statement to the table work, and assigning a variable the same length value, which is based on a CLOB column, not work? >I'm a bit lost there.

so are we because we don't know what you really did do or exactly how Oracle responded.

It is really, Really, REALLY difficult to fix a problem that can not be seen.
use COPY & PASTE so we can see what you do & how Oracle responds.

It would be helpful if you followed Posting Guidelines - http://www.orafaq.com/forum/t/88153/0/
Re: Updating CLOB [message #504163 is a reply to message #504161] Thu, 21 April 2011 13:15 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
There is the type based on the cursor / table from above:

TYPE v_mf_table IS TABLE OF mf_detail%ROWTYPE INDEX BY BINARY_INTEGER;
v_mf_record v_mf_table;


Then there is the loop:

FOR x_rec IN mf_monthly(v_mf_record(v_rec).ticker) LOOP
v_monthly_data := v_monthly_data || x_rec.monthly_data || '+++';
END LOOP;


Then there is the record assignment:

v_mf_record(v_rec).monthly_data := v_monthly_data;

ORA-06502: PL/SQL: numeric or value error: character string buffer too small

Now that record element is based on a CLOB column. A direct insert into the table works for a value up to 32k, but storing it into that variable first gives me the error.

Just strange that the insert works, and the variable assignment has some type of limitation when both values are basically 32k, not greater than.

Re: Updating CLOB [message #504166 is a reply to message #504163] Thu, 21 April 2011 13:31 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Only thing I can guess is that it is based on a CLOB column, which is somehow causing issues because the value is > 4k. Very strange if you ask me.
Re: Updating CLOB [message #504168 is a reply to message #504166] Thu, 21 April 2011 13:44 Go to previous messageGo to next message
Michel Cadot
Messages: 68773
Registered: March 2007
Location: Saint-Maur, France, https...
Senior Member
Account Moderator
If you post a test case we can reproduce maybe we can address your problem otherwise we can't.

Please read OraFAQ Forum Guide, especially "How to format your post?" section.
Make sure that lines of code do not exceed 80 characters when you format.
Indent the code, use code tags and align the columns in result.
Use the "Preview Message" button to verify.
Also always post your Oracle version, with 4 decimals.

Regards
Michel
Re: Updating CLOB [message #504173 is a reply to message #504168] Thu, 21 April 2011 14:06 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Try this script......

CREATE TABLE data (
key1 VARCHAR2(100),
col1 VARCHAR2(100),);

CREATE TABLE test (
key1 VARCHAR2(100),
col1 VARCHAR2(100),
c1 CLOB);

DECLARE
CURSOR data_crs IS
SELECT key1, col1, NULL data_string
FROM data;

CURSOR mf_monthly (p_key VARCHAR2) IS
SELECT mr.m_fund_id ||'|'|| mr.year ||'|'|| mr.month ||'|'|| mr.total_return ||'|'||
mr.unit_value ||'|'|| mr.nav monthly_data
FROM mutual_fund mf, month_result mr, fund_master fm
WHERE mf.m_fund_id = mr.m_fund_id
AND mf.m_fund_id = fm.fund_id
AND TO_DATE(mr.year, 'YYYY') >= ADD_MONTHS(SYSDATE, -132)
AND mf.ticker = p_key
ORDER BY mr.year, mr.month;

v_data CLOB;

TYPE v_data_table IS TABLE OF data_crs%ROWTYPE INDEX BY BINARY_INTEGER;
v_data_record v_data_table;

BEGIN
OPEN data_crs;
FETCH data_crs BULK COLLECT INTO v_data_record;
CLOSE data_crs;

FOR v_rec IN v_data_record.FIRST .. v_data_record.COUNT LOOP
FOR x_rec IN mf_monthly(v_data_record(v_rec).key1) LOOP
v_data := v_data || x_rec.monthly_data || '+++';
END LOOP;

v_data := NULL;
END LOOP;

/*** Error should happen here if v_data > 32k ***/
v_data_record(v_rec) := v_data;

FORALL x IN v_data_record.FIRST .. v_data_record.COUNT
INSERT INTO test VALUES v_data_record(x);
END;
/
Re: Updating CLOB [message #504176 is a reply to message #504173] Thu, 21 April 2011 14:14 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
SQL> CREATE TABLE data (
key1 VARCHAR2(100),
col1 VARCHAR2(100),);  2    3  
col1 VARCHAR2(100),)
                   *
ERROR at line 3:
ORA-00904: : invalid identifier


please only post TESTED & valid SQL
Re: Updating CLOB [message #504178 is a reply to message #504176] Thu, 21 April 2011 14:26 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
I'm just so frustrated.

CREATE TABLE data (
key1 VARCHAR2(100),
col1 VARCHAR2(100));

CREATE TABLE test1 (
key1 VARCHAR2(100),
col1 VARCHAR2(100),
c1 CLOB);

INSERT INTO data VALUES ('X','XXX');

DECLARE
CURSOR data_crs IS
SELECT key1, col1, NULL data_string
FROM data;

CURSOR mf_monthly IS
SELECT RPAD('A',32000,'A') monthly_data FROM dual;

v_data CLOB;

TYPE v_data_table IS TABLE OF data_crs%ROWTYPE INDEX BY BINARY_INTEGER;
v_data_record v_data_table;

BEGIN
OPEN data_crs;
FETCH data_crs BULK COLLECT INTO v_data_record;
CLOSE data_crs;

FOR v_rec IN v_data_record.FIRST .. v_data_record.COUNT LOOP
FOR x_rec IN mf_monthly LOOP
v_data := v_data || x_rec.monthly_data || '+++';
END LOOP;

/*** Error should happen here if v_data > 32k ***/
v_data_record(v_rec).data_string := v_data;

v_data := NULL;
END LOOP;

FORALL x IN v_data_record.FIRST .. v_data_record.COUNT
INSERT INTO test1 VALUES v_data_record(x);
END;
/

Re: Updating CLOB [message #504199 is a reply to message #504178] Thu, 21 April 2011 23:32 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9106
Registered: November 2002
Location: California, USA
Senior Member
You need to make the data_string in your data_crs an empty_clob() as shown below.

SCOTT@orcl_11gR2> CREATE TABLE data
  2    (key1  VARCHAR2 (100),
  3  	col1  VARCHAR2 (100))
  4  /

Table created.

SCOTT@orcl_11gR2> CREATE TABLE test1
  2    (key1  VARCHAR2(100),
  3  	col1  VARCHAR2(100),
  4  	c1    CLOB)
  5  /

Table created.

SCOTT@orcl_11gR2> INSERT INTO data VALUES ('X', 'XXX')
  2  /

1 row created.

SCOTT@orcl_11gR2> DECLARE
  2    CURSOR data_crs
  3    IS
  4    SELECT key1, col1,
  5  	      EMPTY_CLOB() AS data_string
  6    FROM   data;
  7  
  8    CURSOR mf_monthly
  9    IS
 10    SELECT RPAD ('A', 32000, 'A') AS monthly_data
 11    FROM   dual;
 12  
 13    v_data CLOB;
 14  
 15    TYPE v_data_table IS TABLE OF data_crs%ROWTYPE INDEX BY BINARY_INTEGER;
 16    v_data_record v_data_table;
 17  BEGIN
 18    OPEN data_crs;
 19    FETCH data_crs BULK COLLECT INTO v_data_record;
 20    CLOSE data_crs;
 21  
 22    FOR v_rec IN v_data_record.FIRST .. v_data_record.COUNT LOOP
 23  	 FOR x_rec IN mf_monthly LOOP
 24  	   v_data := v_data || x_rec.monthly_data || '+++';
 25  	 END LOOP;
 26  
 27  	 v_data_record(v_rec).data_string := v_data;
 28  
 29  	 v_data := NULL;
 30    END LOOP;
 31  
 32    FORALL x IN v_data_record.FIRST .. v_data_record.COUNT
 33    INSERT INTO test1 VALUES v_data_record(x);
 34  END;
 35  /

PL/SQL procedure successfully completed.

SCOTT@orcl_11gR2> SELECT * FROM test1
  2  /

KEY1
--------------------------------------------------------------------------------
COL1
--------------------------------------------------------------------------------
C1
--------------------------------------------------------------------------------
X
XXX
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
... -- truncated to save space
AAAAAAAAAAAAAAA
+++


1 row selected.

SCOTT@orcl_11gR2>

Re: Updating CLOB [message #504250 is a reply to message #504199] Fri, 22 April 2011 17:34 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
So, the difference was you defining the empty column as an EMPTY_CLOB vs a NULL like I did. When I ran it, it worked fine. That is good news.

However, when I tried that and ran it with my entire record set. It has increased the processing time of the stored procedure like 500%. Are objects that slow to process?
Re: Updating CLOB [message #504263 is a reply to message #504250] Fri, 22 April 2011 19:07 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>Are objects that slow to process?
YES
OBJECT datatype is NOT native to any RDBMS
TYPE datatype were added to SQL to placate Object Oriented proponents
Re: Updating CLOB [message #504267 is a reply to message #504263] Fri, 22 April 2011 19:17 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Hmmm.....kind of a catch 22. The code executes quickly, but when the FORALL goes to perform the insert, might as well take a 2 year vacation. Inserting these CLOB's into the table is taking way too long. Would using DBMS_LOB and cutting it into 4k pieces work any better? I was hoping to avoid that. Your solution works. But when processing 10,000 records, each record with 5 CLOB columns, things seem to slow to a crawl.

Maybe look through several times, issuing the FORALL and then clearing out the array. If you have any other ideas, I'm all ears.
Re: Updating CLOB [message #504268 is a reply to message #504267] Fri, 22 April 2011 19:28 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
If you reached into your tool chest & grabbed a screwdriver; it may be sub-optimal for driving a nail into the board.

I request you step back & provide us some background information regarding the business problem that needs to be solved.
Or ignore this request & continue to bang away at the nail with your screwdriver.
It is your choice.
Re: Updating CLOB [message #504269 is a reply to message #504267] Fri, 22 April 2011 19:31 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9106
Registered: November 2002
Location: California, USA
Senior Member
I am assuming that this is a somewhat simplified version of the actual problem and that there is a relation between the two actual cursors. Otherwise, you could just calculate the value of v_data once. I would think that the forall would be fast and that the nested for loops would be the slow part. Have you timed those parts separately? If it weren't for the size of your clobs, you could just use a simple insert into ... select ... from ... I would at least try to eliminate as many intermediary steps as possible, such as eliminating the first cursor and the v_data variable as shown below.

SCOTT@orcl_11gR2> CREATE TABLE data
  2    (key1  VARCHAR2 (100),
  3  	col1  VARCHAR2 (100))
  4  /

Table created.

SCOTT@orcl_11gR2> CREATE TABLE test1
  2    (key1  VARCHAR2(100),
  3  	col1  VARCHAR2(100),
  4  	c1    CLOB)
  5  /

Table created.

SCOTT@orcl_11gR2> INSERT INTO data VALUES ('X', 'XXX')
  2  /

1 row created.

SCOTT@orcl_11gR2> DECLARE
  2    TYPE v_data_table IS TABLE OF test1%ROWTYPE INDEX BY BINARY_INTEGER;
  3    v_data_record v_data_table;
  4  
  5    CURSOR mf_monthly
  6    IS
  7    SELECT RPAD ('A', 32000, 'A') AS monthly_data
  8    FROM   dual;
  9  BEGIN
 10    SELECT key1, col1,
 11  	      EMPTY_CLOB() AS c1
 12    BULK   COLLECT
 13    INTO   v_data_record
 14    FROM   data;
 15  
 16    FOR v_rec IN v_data_record.FIRST .. v_data_record.COUNT LOOP
 17  	 FOR x_rec IN mf_monthly LOOP
 18  	   v_data_record(v_rec).c1 :=
 19  	     v_data_record(v_rec).c1 || x_rec.monthly_data || '+++';
 20  	 END LOOP;
 21    END LOOP;
 22  
 23    FORALL x IN v_data_record.FIRST .. v_data_record.COUNT
 24    INSERT INTO test1 VALUES v_data_record(x);
 25  END;
 26  /

PL/SQL procedure successfully completed.

SCOTT@orcl_11gR2> SELECT * FROM test1
  2  /

KEY1
--------------------------------------------------------------------------------
COL1
--------------------------------------------------------------------------------
C1
--------------------------------------------------------------------------------
X
XXX
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
... -- truncated to save space
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+++


1 row selected.

SCOTT@orcl_11gR2>

Re: Updating CLOB [message #504270 is a reply to message #504269] Fri, 22 April 2011 19:40 Go to previous messageGo to next message
ThomasG
Messages: 3212
Registered: April 2005
Location: Heilbronn, Germany
Senior Member
What I really feel here is that the CLOBs are not the right way to hold the data. Anything in a CLOB (or LOB for that matter) should not be anything that needs to be touched and updated thousands of times, since that will of course slow everything to a crawl.

Plus, having a CLOB named "MONTHLY_DATA" where records get appended sounds pretty much like having a *.CSV file stored in a CLOB that you then read to get to the data, and append to store more data, sounds pretty much like a complete design failure from the start.

At first glance child tables that hold the records would be the more RDBMS fitting solution. Than using a RDBMS to store "files" and then re-implement all the RDBMS features that would already be there for data in tables yourself.

Re: Updating CLOB [message #504271 is a reply to message #504270] Fri, 22 April 2011 20:00 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
Actually the table is truncated and re-populated once once a month. The reason for the CLOB columns is this: There is a 1 to many relationship between some tables. What is being done is to select the records from the 'many' tables and concatenate them like this: ABC+++DEF+++GHI+++JKL and so on. The application will take care of parsing the string.

Some of those strings get longer than 4000 bytes. Therefore, they would need to be stored in CLOB columns. This has slowed everything down. If I comment out the FORALL and run the script it is lightening fast. If I only select a small handful of records it is also pretty fast. But when I use the entire record set in the outermost FOR loop, which is about 10000, then it just crawls.

Maybe I need to do a LIKE, and do each letter by itself, emptying the array each time to have a small set inserted.

What else can I provide to help?
Re: Updating CLOB [message #504273 is a reply to message #504271] Fri, 22 April 2011 20:10 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9106
Registered: November 2002
Location: California, USA
Senior Member
I agree with Thomas. Why concatenate the values? Why not store them in separate rows in a child table? What is ultimately done with this data? How are you ultimately trying to display it? If you store it in child tables, then you can select using a simple join and display the columns individually without concatenation.

Re: Updating CLOB [message #504274 is a reply to message #504271] Fri, 22 April 2011 20:14 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>Actually the table is truncated and re-populated once once a month.
What business goal does this achieve?

>select the records from the 'many' tables and concatenate them like this: ABC+++DEF+++GHI+++JKL and so on
WHY "must" concatenation occur?

Why is data being moved from one table to another; when it still resides in the same database?
What business goal/requirement is being satisfied?
It almost sounds like to you are rearranging the deck chairs on the Titanic.
Re: Updating CLOB [message #504277 is a reply to message #504274] Fri, 22 April 2011 22:33 Go to previous messageGo to next message
amerar
Messages: 18
Registered: April 2011
Junior Member
The general idea what they are trying to do is minimize the number of database calls PHP must make. So, the thought was to create a table that basically contains all the data needed to display on the different web pages. The application can parse out the data and it would be faster than multiple database calls, or a more complex query with a bunch of joins, etc.

So, that is why they wanted to take this approach. As to why truncate it monthly, because new data comes in once a month, and this table is supposed to be the result of a bunch of queries, ie: it avoids all the table joins ans it makes only 1 database call.

They've done that in many other places. Just the whole CLOB thing and how long they take to process......
Re: Updating CLOB [message #504283 is a reply to message #504277] Fri, 22 April 2011 22:45 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
>They've done that in many other places.
It worked when data set was small, but does not scale.

one SQL call can bring back MANY rows such that each row could be saved in local data structure

>it avoids all the table joins ans it makes only 1 database call.
many folks do this with VIEW

You are simply treating CLOB as stored flat file; which complicates the logic of the application to parse the data.

The design is flawed & does not scale.
You can continue to use the existing "design" or start over with a design that can scale well.
Re: Updating CLOB [message #504309 is a reply to message #504283] Sat, 23 April 2011 05:39 Go to previous messageGo to next message
ThomasG
Messages: 3212
Registered: April 2005
Location: Heilbronn, Germany
Senior Member
One option which might be the easiest to implement without changing too much of the other code:

Don't store and then select the CLOBs from the table, crate a function that returns a CLOB and call that function in the select from PHP instead of the CLOB column.

Then put the logic that you now use to store the CLOB in the table in the function that returns the CLOB.

I have use that approach a few times, where I built complete HTML documents inside the database to pass to a front end as CLOBs.

[Updated on: Sat, 23 April 2011 05:39]

Report message to a moderator

Re: Updating CLOB [message #504335 is a reply to message #504309] Sat, 23 April 2011 14:30 Go to previous message
amerar
Messages: 18
Registered: April 2011
Junior Member
So, are you suggesting maybe an inline function to collect the data, and the function itself will get the data? The table would have 5 CLOB columns, so you are suggesting to use individual functions to get the data instead?
Previous Topic: error from dbms_metadata.get_ddl
Next Topic: Need to Query for the top 10 records
Goto Forum:
  


Current Time: Fri Dec 26 22:14:10 CST 2025