BI & Warehousing

Want to learn Exadata ?

Amardeep Sidhu - Fri, 2015-01-02 03:19

Many people have asked me this question that how they can learn Exadata ? It starts sounding even more difficult as a lot of people don’t have access to Exadata environments. So thought about writing a small post on the same.

It actually is not as difficult as it sounds. There are a lot of really good resources available from where you can learn about Exadata architecture and the things that work differently from any non-Exadata platform. You might be able to do lot more RnD if you have got access to an Exadata environment but don’t worry if you haven’t. Without that also there is a lot that you can explore. So here we go:

  1. I think the best reference that one can start with is Expert Oracle Exadata book by Tanel Poder, Kerry Osborne and Randy Johnson. As a traditional book covers the subject topic by topic from ground up so it makes a fun read. This book is also no different. It will teach you a lot. They are already working on the second edition. (See here).
  2. Next you can jump to whitepapers on Oracle website Exadata page, blog posts (keep an eye on OraNA.info) and whitepapers written by other folks. There is a lot of useful material out there. You just need to Google a bit.
  3. Exadata documentation (not public yet) should be your next stop if you have got access to it. Patch 10386736 on MOS if you have got the access.
  4. Try to attend an Oracle Users Group conference if there is one happening in your area. Most likely someone would be presenting on Exadata so you can use that opportunity to learn about it. Also you will get a chance to ask him questions.
  5. Lastly if you have an Exadata machine available do all the RnD you can.

Happy New Year and Happy Learning !

Categories: BI & Warehousing

Paginated HTML is here and has been for some time ... I think!

Tim Dexter - Fri, 2014-12-12 18:03

We have a demo environment in my team and of course things get a little beaten up in there. Our go to, 'here's Publisher' report was looking really bad. Data was not returning or being rendered correctly on the five templates we have for it.
So, I spent about a half hour cleaning up the report; getting things working again; clearing out the rubbish. I noticed that one of the layouts when rendered in HTML was repeatedly showing a header down the screen. Oh, I know where to get rid of that and off I click to the report properties to fix it. But what is this I see? Is it? Can it be? Are my tired old eyes deceiving me?

Yes, Dexter, you see that right, 'View Paginated'! I nervously changed the value to 'true' and went back to the HTML output.
Holy Amaze Balls Batman, paginated HTML, the holy grail of HTML rendered reports, the Mount Everest of ... no, thats too easy, the K2 of html output ... its fan-bloody-tastic! Can you tell Im excited? I was immediately on messenger to Leslie (doc writer extraordinaire) 


Obviously not quite as big a deal in the sane, real world outside of my head. 'Oh yeah, we have that now ...' Leslie is so calm and collected, however, she does like Maroon 5 but, we overlook that :)

I command you 11.1.1.6+'ers to go find the property and turn it on right now and bask in the glory that is, 'paginated html.!'
I cannot stop clicking back and forth and then to the end and then all the way back to the beginning. Its fantastic!

Just look at those icons, just click em, you know you want to!

Categories: BI & Warehousing

File Encoding in the Next Generation Outline Extractor

Tim Tow - Tue, 2014-12-09 20:11
We had a couple of issues reported with the output of the Next Generation Outline Extractor where the exported file did not work properly as a load file. After some investigation, we found that the file encoding was incorrect. We were encoding the files using the Unicode/UTF-8 format. We chose this encoding so that we could write all characters in all languages, but we did not consider that UTF-8 is only valid for loading Unicode databases in Essbase.

To resolve this issue, we decided to add a configuration to the Next Generation Outline Extractor to allow users to select the file encoding. Here is a screenshot showing the new configuration setting.



As of yesterday, December 8, 2014, the updated Next Generation Outline Extractor is available on our website. The first version to feature this functionality is version 2.0.2.692. Version 2.0.2.692 is available for all Essbase versions from Essbase 9.3.1 forward. We are also happy to announce that his version of the Next Generation Outline Extractor is the first version to support the recently released Essbase 11.1.2.3.505.

If you encounter any issues with the Next Generation Outline Extractor, please don't hesitate to contact our support team at support@appliedolap.com.

Categories: BI & Warehousing

VirtualBox and Windows driver verifier

Amardeep Sidhu - Wed, 2014-12-03 04:34

I was troubleshooting some Windows hangs on my Desktop system running Windows 8 and enabled driver verifier. Today when I tried to start VirtualBox it failed with this error message.

Failed to load VMMR0.r0 (VERR_LDR_MISMATCH_NATIVE)

Most of the online forums were asking to reinstall VirtualBox to fix the issue. But one of the thread mentioned that it was being caused by Windows Driver Verifier. I disabled it, restarted Windows and VirtualBox worked like a charm. Didn’t have time to do more research as i quickly wanted to test something. May be we can skip some particular stuff from Driver Verifier and VirutalBox can then work.

Categories: BI & Warehousing

Bordering Text

Tim Dexter - Tue, 2014-11-18 16:08

A tough little question appeared on one of our internal mailing lists today that piqued my interest. A customer wanted to place a border around all data fields in their BIP output. Something like this:


Naturally you think of using a table, embedding the field inside a cell and turning the cell border on. That will work but will need some finessing to get the cells to stretch or shrink depending on the width of the runtime text. Then things might get a bit squirly (technical term) if the text is wide enough to force a new line at the page edge. Anyway, it will get messy. So I took a look at the problem to see if the fields can be isolated in the page as far as the XSLFO code is concerned. If the field can be siolated in its own XSL block then we can change attribute values to get the borders to show just around the field. Sadly not.

This is an embedded field YEARPARAM in a sentence.

translates to

 <fo:inline height="0.0pt" style-name="Normal" font-size="11.0pt" style-id="s0" white-space-collapse="false" 
  font-family-generic="swiss" font-family="Calibri" 
  xml:space="preserve">This is an embedded field <xsl:value-of select="(.//YEARPARAM)[1]" xdofo:field-name="YEARPARAM"/> in a sentence.</fo:inline>


If we change the border on tis, it will apply to the complete sentence. not just the field.
So how could I isolate that field. Well we could actually do anything to the field. embolden, italicize, etc ... I settled on changing the background color (its easy to change it back with a single attribute call.) Using the highlighter tool on the Home tab in Word I change the field to have a yellow background. I now have:

 This gives me the following code.

<fo:block linefeed-treatment="preserve" text-align="start" widows="2" end-indent="5.4pt" orphans="2"
 start-indent="5.4pt" height="0.0pt" padding-top="0.0pt" padding-bottom="10.0pt" xdofo:xliff-note="YEARPARAM" xdofo:line-spacing="multiple:13.8pt"> 
 <fo:inline height="0.0pt" style-name="Normal" font-size="11.0pt" style-id="s0" white-space-collapse="false" 
  font-family-generic="swiss" font-family="Calibri" xml:space="preserve">This is an embedded field </fo:inline>
  <fo:inline height="0.0pt" style-name="Normal" font-size="11.0pt" style-id="s0" white-space-collapse="false" 
   font-family-generic="swiss" font-family="Calibri" background-color="#ffff00">
    <xsl:attribute name="background-color">white</xsl:attribute> <xsl:value-of select="(.//YEARPARAM)[1]" xdofo:field-name="YEARPARAM"/> 
  </fo:inline> 
 <fo:inline height="0.0pt" style-name="Normal" font-size="11.0pt" style-id="s0" white-space-collapse="false" 
  font-family-generic="swiss" font-family="Calibri" xml:space="preserve"> in a sentence.</fo:inline> 
</fo:block> 

Now we have the field isolated we can easily set other attributes that will only be applied to the field and nothing else. I added the following to my YEARPARAM field:

<?attribute@inline:background-color;'white'?> >>> turn the background back to white
<?attribute@inline:border-color;'black'?> >>> turn on all borders and make black
<?attribute@inline:border-width;'0.5pt'?> >>> make the border 0.5 point wide
<?YEARPARAM?> >>> my original field

The @inline tells the BIP XSL engine to only apply the attribute values to the immediate 'inline' code block i.e. the field. Collapse all of this code into a single line in the field.
When I run the template now, I see the following:

 


Its a little convoluted but if you ignore the geeky code explanation and just highlight/copy'n'paste, its pretty straightforward.

Categories: BI & Warehousing

Are you ready for Windows 10? Dodeca Is Ready!

Tim Tow - Wed, 2014-10-15 23:38
Microsoft recently released its first preview version of their next desktop operating system, Windows 10. In our business, we have seen many large companies just migrate to Windows 7 and the general consensus seems to be that Windows 8 is a 'consumer' operating system. In other words, it will be a while before you have to think about Windows 10, but it is our business to be thinking about this now. We have the Windows 10 preview downloaded and installed in our labs. We made a very smart decision years ago when we decided to write the Dodeca client layer in .NET technology because it works in Windows 10 unchanged!

We are ready. Will you be ready?
Categories: BI & Warehousing

Kscope15 Abstract deadline is Wednesday, October 15!

Tim Tow - Sat, 2014-10-11 14:08
Kscope15 abstract submission will close in just a few days and the ODTUG content teams want you!

I am sure there are many of you who have never done a presentation at a conference and may not feel comfortable speaking in public. Well, I have a secret for you, despite the fact that I won the Best Speaker Award at Kscope14, I share your discomfort with speaking. What works for me, however, is that I only speak about topics that I know very well. Do you have some area of expertise that you know really well? If you do, please share it! It will be fun and all of your peers who learn something new, from you, will appreciate it!

With that, take the next step and click here to submit session!
Categories: BI & Warehousing

Multi Sheet Excel Output

Tim Dexter - Thu, 2014-10-02 18:28

Im on a roll with posts. This blog can be rebuilt ...

I received a question today from Camilo in Colombia asking how to achieve the following.

‘What are my options to deliver excel files with multiple sheets? I know we can split 1 report in multiple sheets in with the BIP Advanced Options, but what if I want to have 1 report / sheet? Where each report in each sheet has a independent data model ….’

Well, its not going to be easy if you have to have completely separate data models for each sheet. That would require generating multiple Excel outputs and then merging them, somehow.

However, if you can live with a single data model with multiple data sets i.e. queries that connect to separate data sources. Something like this:


Then we can help. Each query is returning its own data set but they will all be presented together in a single data set that BIP can then render. Our data structure in the XML output would be:

<DS>
 <G1>
  ...
 </G1>
 <G2>
  ...
 </G2>
 <G3>
  ...
 </G3>
</DS>

Three distinct data sets within the same data output.

To get each to sit on a separate sheet within the Excel output is pretty simple. It depends on how much native Excel functionality you want.

Using an RTF template you just create the layouts for each data set on a page(s) separated by a page break (Ctrl-Enter.) At runtime, BIP will place each output onto a separate sheet in the workbook. If you want to name each sheet you can use the <?spreadsheet-sheet-name: xpath-expression?> command. More info here. That’s as sophisticated as it gets with the RTF templates. No calcs, no formulas, etc. Just put the output on a sheet, bam!

Using an Excel template you can get more sophisticated with the layout.



This time thou, you create the layout for each data model on separate sheets. In my example, sheet 1 holds the department data, sheet 2, the employee data and so on. Some conditional formatting has snuck in there.

I have zipped up the sample files here.

FIN!

Normal 0 false false false EN-US X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:Calibri; mso-bidi-theme-font:minor-latin;}
Categories: BI & Warehousing

Oracle Exalytics X4-4 - Bigger, Better, Stronger

Look Smarter Than You Are - Sun, 2014-09-28 11:49
X4-4 - Same price as the X3-4 but with more powerThe big announcement about it is today at OpenWorld (it would be awesome if they mentioned it during the Intel keynote tonight), but the Exalytics X4-4 is actually available now.  It's the same price as the X3-4 ($175,000 at list not including software, maintenance, tax, title, license, yada yada).  This does mean the X3 is - effective immediately - no longer available, but then again, since the new one is the same price, I'm not sure why anyone would want the older one.  No word yet on if you can upgrade an X3 to an X4, but since they did offer an upgrade kit from X2 to X3 (though I never heard of anyone buying it), I'm guessing there will be one for those wanting to make an X3 into an X4.
X4-4 SpecsThe main improvement over the X3 is the number of cores: it's still 4 Intel chips, but those chips all now have 15 cores on them, meaning the X4 has 60 cores compared to the X3's 40 cores.  Here are the important details:

  • 4 Intel Xeon E7-8895v2 processors running at 2.8 - 3.6 GHz
  • 8 - 60 cores (capacity on demand, more on this in a second)
  • 2 TB of RAM
  • 2.4 TB of PCI flash
  • 7.2 TB of hard disk running at 10K RPMs (not that fast these days)
  • 2 Infiniband ports running at 40 Gb/s
  • 4 Ethernet ports running at up to 10 Gb/s
Cool Thing 1: Variable Speed & Cores

You probably heard about this last July.  Oracle worked with Intel to design a line of their Xeon E7-889x chips specifically for Oracle.  What we didn't realize until we saw it show up on the X4 spec sheet was that the chips were going in the Exalytics X4.  Simply put, on the fly, Exalytics can vary how many cores it uses and when it's fewer cores, the speed goes up.  If it's running 15 cores per chip, Intel sets the speed to 2.8 GHz.  If it's only using 2 cores per chip the speed goes all the way to 3.6 GHz (a GHz is one billion clock ticks per second).


But wait, you math geniuses say.  Isn't 3.6 * 2 less than 2.8 * 15 (so why wouldn't Oracle just always leave all 60 cores on at the slower speed)?  Well, yes, if you're actually using all those cores, and this is where you know the chip was apparently designed for Essbase (though it did premiere in Exadata first).  As much as I love my Essbase, there are still transactions that end up single threading (or using far less than the available cores on the box).

Say I'm running a massive allocation and despite my best efforts (and FIXPARALLEL), it's still single threading or running at 8 CPUs or fewer.  In this case, Exalytics is now smart enough to talk to those impressive new E7-8895v2 chips, scale down to as few cores as are actually needed, and in the process, up the clock speed for the remaining cores.  Take that, commodity hardware.  This really is the killer feature that makes Exalytics do something no other server running Essbase can do.

On a side note, Intel seems to be dropping the power on the non-used cores to nearly zero when not in use meaning the power consumption of your Exalytics box actually lowers on-demand.  So if your boss won't sign off on your new Exalytics X4, tell her she hates the planet.

Cool Thing 2: You Don't Need BIFS
Per the current Engineered Systems Price List (buried down in note 13), you longer have to purchase BIFS (BI Foundation Suite) to buy Exalytics (either the X4 or T5).  You can now own BIFS, OBIEE, Essbase+, or Hyperion Planning+ without having to get a VP to sign off for a special exemption.  That's right, Planning people preferring to purchase pure premium power, you can now buy Exalytics.  With this change, I presume that any new Planning customer looking for the best user experience will be buying Exalytics X4 along with Planning.

Also buried in the footnotes, you apparently can now buy Exalytics for as few as 20 named users.  Last time I checked (and I don't read every edition of the footnotes, haters who think I have no life), the minimum was 100 named users.

What's Next: HFM on Exalytics
We heard about it on the opening developer's day at Kscope: HFM should finally run on Exalytics in version 11.1.2.4 (which we're hoping to see by the end of 2014).  I'm not sure if it will run on both the T5 (Solaris) and the X4 (Linux) by year-end, but Linux is almost a given.  That said, I don't work for Oracle, so don't base any buying decisions on the belief that HFM will definitely run on the X4.  Just when it happens, be pleasantly surprised that you can now consolidate all your major Oracle Business Analytics apps together.

So any T5 news?  Not at the moment. It's still available running it's 128 cores with 4 TB of RAM (and other cool things) so if you're looking for major horsepower and server consolidation, look to the T5.

I'll be updating this post after the OpenWorld keynote to include any new Exalytics news but if you hear any other Exalytics updates in the meantime, post it in the comments.

Categories: BI & Warehousing

Database Links

Tim Dexter - Thu, 2014-09-25 14:39

Yeah, its been a while, moving on ...

I got a question a week back asking about how BI Publisher could handle dblinks. The customer currently has db links from DB1 to DB2 and uses them in their queries. Could BIP handle the syntax and pass it on to the database in its SQL or could it handle the link another way?

select e1.emp_name
, e1.emp_id
,e2.manager
from emps e1
, emps@db2 e2
where e1.manager_id = e2.id

Well, there is the obvious way to create the join in BIP. Just get rid of the db link alttogether and create two separate database connections (db1 and db2). Write query A against db1 and query B against db2. Then just create a join between the two queries, simple.

 But, what if you wanted to use the dblink? Well, BIP would choke on the @db2 you would have in the sql. Some silly security rules that, no, you can not turn off if you want to. But there are ways around it, the choking, not the security. Create an alias at the database level for the emp@db2, that way BIP can parse the resulting query. Lets assume I create an alias in the db for my db linked table as 'managers'. Now my query becomes:

select e1.emp_name
, e1.emp_id
,e2.manager
from emps e1
, managers e2
where e1.manager_id = e2.id

 BIP will not choke, it will just pass the query through and the db can handle the linking for it.

Thats it, thats all I got on db links. See you in 6 months :)




Categories: BI & Warehousing

Next Generation Outline Extractor Webcast - Oct 8th!

Tim Tow - Fri, 2014-09-12 17:07
I am doing a repeat of my Kscope15 'Best Speaker' award-winning presentation as part of the ODTUG webcast series. Here is the official announcement from ODTUG:

Wednesday, October 8, 2014 12:00 PM - 1:00 PM EDT

Next Generation Essbase Outline Extractor Tips and Tricks
Tim Tow, Applied OLAP

The Next Generation Outline Extractor is the follow-up to the classic OlapUnderground Essbase Outline Extractor used by thousands of Essbase customers. This session, which was the highest-rated session at Kscope15 in Seattle, explores some of the new capabilities of the Next Generation Outline Extractor, including command line operations exported directly to a relational database. Attend this session to learn how to leverage this free utility in your company.
Make you you sign up to join my on October 8th; you can register here!.
Categories: BI & Warehousing

Oracle Tours Africa and the Middle East

Look Smarter Than You Are - Sun, 2014-06-15 11:33
Happy Father's Day, everyone!  I got up early this morning to write about my recent experience traveling the world on Oracle's behalf.  I got to attend the first annual Oracle Technology Network tour of Africa and the Middle East.  It made 2 stops in North Africa (both in Tunisia), 2 stops in Saudi Arabia, and the final stop was in Dubai, UAE.

Tariq Farooq first mentioned the idea of doing a MENA (Middle East & North Africa) tour to me in Beijing last fall.  He asked if I'd be willing to travel half-way around the world to speak to people in English that primarily spoke French and Arabic, and I - of course - said "yes."  Here's Tariq being interviewed by Lillian Buziak at Collaborate 2014 (audio is a bit difficult to hear):


I had two reasons for wanting to go: I do love educating/evangelizing for Oracle EPM, BI, and Business Analytics.  The possibility of reaching new audiences for the first time was exciting. My other reason for going was that I wanted to experience totally different cultures than I ever have before.  I've spoken on 5 continents (now 6 after this tour and I'm anxiously awaiting the OTN Tour to Antarctica) before and have seen presented everywhere from a women's college in Mumbai that was 95F with no air conditioning in the presentation room to a ballroom in the Philippines that had 3 simultaneous English sessions going on (in one room!) all happily observed by smiling Filipinos.  From China to India to Australia to Germany, I have seen some amazing slices of life, but nothing prepared me for the differences I saw on this tour.

In each of the sections below, I have linked the header to a blog from my new best German friend, Bjoern Rost.  He blogged after every stop and unlike me, he actually understood all the Oracle RDBMS sessions on the tour.  Visit http://portrix-systems.de/blog/author/brost/ to see his entertaining blog posts.  (Warning: though I think Bjoern is hilarious, being German, you may find his posts to be 'not funny.'  German humor is an acquired taste.)

I left for the first stop, Tunisia, on Memorial Day (in the USA), May 26, 2014...




Tunisia, May 27To get to Tunisia in time for my session, I left Dallas. Texas at 10AM on Monday, flew to JFK (New York City), flew to Rome, flew to Tunis, and had a nice car waiting for me at the airport compliments of our hosts in Tunisia.  I landed at 10:30AM on Tuesday and considering I was flying to Africa, I felt that the trip went by quickly.  I made it through customs in Tunis in about 15 minutes, walked out the front door of the hotel, and was in an entirely different world.

Speaking in North America, South America, Europe, Asia, and Australia had done absolutely nothing to prepare me for Africa.  The closest thing I could compare it to in my life (but the comparison does not do it justice) was the cities of India: chaotic, dirty, cramped, foreign, and chaotic (worth mentioning again).  Now take all that, remove the cows, and make it Muslim.

My host picked me up in a nice car and we began the hour drive to Beja where the conference was being held.  We passed mounds of trash piled up in the center of the roads though thankfully the heat (high was ~75F when I arrived) didn't make the place smell horribly.  Traffic laws seemed to be non-existent, but it moved fairly quickly being in the middle of the day.  I loved looking out the window at the shops along the road and carts selling watermelon approximately every 100 feet (I was told by my host that it was watermelon season).

We left the city about 20 minutes after I got in the car and I was suddenly in Tuscany.  At least, it looked like Tuscany: fields of amber waves of grass, wide open spaces, wildflowers, lakes, olive groves, country life, gorgeous hills... truly one of the most beautiful countrysides I've seen in my entire life.  Since I had been traveling for over a day, the pleasant scenery soon lulled me to sleep.


My driver woke me up when we got to the technical university in Beja. I walked in to find Tariq trying to explain Oracle Enterprise Manager to a bunch of college students who didn't seem to understand databases let alone Oracle.  I tried to hide in one of the back seats, but Tariq immediately called me up on stage to answer a question about how to develop optimal databases.

Not long after I got there, we broke for lunch.  Our hosts took us to a traditional Tunisian restaurant in downtown Beja.  They were kind enough to make me vegetarian food.  Lunch is apparently a sacred event not-to-be-hurried in Tunisia, so we made it back to the university over 2 hours after we left, fully satiated.

I volunteered to give a session introducing Big Data and Analytics to the college kids, because I felt that it required little technical background.  It seemed to go over well.  My favorite part was when I told a joke about the differences in social media sites and only 10 people laughed.  The people on either side of those 10 then asked them to repeat in Arabic and French what I had said, which caused those people to laugh.  They then shared it around the room and it was like a disease vector of laughter that took 2 minutes to make it around to the 150+ students in the room.  In case you haven't seen it, here's essentially what I said out-loud:
After the sessions were over, I asked one of the professors why everyone listened so intently if they had no background in Oracle.  I mentally wondered if it was because I was an awesome presenter bringing the gospel of Oracle to the future of Africa.  I was told "they didn't understand a lot of what you were saying, but they love listening to people speak English."  So much for my future African disciples.
Our hosts offered to drive us to Dougga, the so-called "best preserved Roman ruins outside of Italy."  It sounded like an exaggeration, but it was actually an understatement. Dougga was once a "small" Roman town on the fringes of the empire... and the miles of town are for the most part still there.  We arrived shortly after they closed the gates for the day.  Our hosts got out of the front car in our 4-car caravan to talk to the guards.  I was in the last car and saw an interesting polite dialog when our hosts started pointing to my car.  I waved back.  The guard smiled and raised the gates for our caravan to enter.  I wondered if bribing had occurred, so I asked how we got in after hours.  Our gracious hosts explained that I was renowned historian, Edward Roske, a visiting professor from the United States of America whose sole purpose for being in Tunisia was to see the Dougga ruins.  They said I should take lots of pictures and walk around looking officially important.  I discovered later that they weren't kidding: they really did tell this to the guard, so I took at least 50 pictures and took some very official selfies to help sell my renowned historian status.




The ruins were truly majestic.  Every time I came around a bend, there was another temple, theatre, circus, road, market, tunnel, column, arch, statue, or something else 2,000 years old to be seen.  It is all in a semi-wild state with no borders separating the ruins from the countryside.  There were even wildflowers growing in the central square:
My favorite moment of the entire trip occurred when I broke away from the rest of our group to go explore some arches on the edge of the ruins.  I went to take a picture of one of the doorways, and I got photobombed:
I went through the doorway to discover a local sheepherder grazing his sheep right in the ruins:

They started on the edge of the ruins but eventually the herder marched his sheep right down the center of the 2,000 year old road leading through Dougga.  Tariq decided to join their herd:
I can't stress enough how amazing this site is.  I would encourage people to visit Tunisia if for no other reason than to see Dougga and Carthage.  You have never felt Roman society like you can wandering around the ancient town with only sheep to keep you company.

Eventually, the guards at the entrance (the only guards in the place, so far as we could tell) came to find our renowned historian group because they wanted to go home.  We stayed the night in the Golden Tulip Hotel in Carthage which was a 4-star hotel for under $200 USD per night.  I recommend it to anyone.  The next morning, the OTN MENA 5 (Tariq, Mike Ault, Bjoern, me, and Jim Czuprynski) headed for the flight to Cairo then on the Riyadh.


Saudi Arabia, May 29-31While the Islam is a part of the culture in Tunisia, Islam is the culture in Saudi Arabia.  I have never seen a country more dominated by a single religion than Saudi Arabia.  It is one of the most difficult places in the world to get a visa (they suspended tourist visas 5 years ago) and it's even harder for a non-Muslim like me.  I spent 4 hours clearing customs which gave me a lot of time to study up on what I was in for.

My guidebook (and several websites) told me about all the things I wasn't allowed to do, say, or maybe even think when I got to Saudi.  Here's what I was told versus what actually happened:
- Muslims everywhere.  Yes, 100% true.  They have calls to prayers everywhere and we had to stop presenting when it was time for prayer.  The whole city stops, for that matter, when it's prayer time.  I was lucky enough to be in a public park for evening prayer one day.  The sounds of the call to prayers across the city were beautiful.

- Traffic fatalities.  90% true.  Saudi apparently has the highest incidence of traffic fatalities in the world because traffic laws are more like traffic vague suggestions only to be followed if everyone has plenty of time and sort of feels like it.  I was prepared to almost die every time I got in a car, and while I saw no deaths, I saw multiple car accidents of the fender bender type each day I was in Saudi.  At one point, we were stopped at a red-light to make a left turn.  With traffic coming from both directions in front of us, a car behind us who wanted to make a left-hand turn felt that our stopping was delaying his day.  He didn't honk or behave rudely in any way: he just drove around us and made the left-turn on the red into oncoming traffic.  No one seemed annoyed at the man for doing it.
- Pornography.  100% nonexistent in Saudi.  They even go through the magazines in the shops and black out with a sharpie anything that's considered too revealing (shoulders, waists, knees, etc.).  They also sharply monitor the web and block out any site deemed inappropriate (including Bing.com with safe search set to anything but strict).
- Pictures of other people.  0% true.  I was told before I went that Muslims do not believe in having images taken of people.  What I actually found was the most selfie-absorbed culture I've ever seen, and I live in America with a teenage child.  I couldn't walk 10 feet at the Riyadh or Jeddah events without someone taking a picture.  I was also told that you couldn't take pictures of public buildings or in public buildings (like the national museum).  Totally untrue: people were taking pictures of just about anything except women.
- Women being covered.  100% true.  All the women wore black abbeyahs at all times.  You're not allowed to film them or talk to them.  That said, I didn't see that many.  Both the events in Riyadh and Jeddah were male-only.  We did see women in the public places particularly near retail outlets and in the city parks & museums.
- Men wearing suits.  25% true.  For the most part, the men wore traditional dress shirts and head coverings.  I wore a suit (no tie) which considering it was 110F+, was quite a sacrifice.  My fellow presenters wore ties in addition to their suits which proves they're more willing to sacrifice for the cause of Oracle.
- No alcohol. 100% true but weird.  The first night I got to Riyadh, I opened my minibar to find... a Budweiser.  Closer examination revealed it was a "Budweiser NA" signifying no alcohol.  Every restaurant we went to offered us "Saudi champagne" or "Saudi wine" which apparently means alcohol-taste without any actual alcohol.  Since I hate the taste of alcohol, I didn't think non-alcoholic alcohol would taste any better, so I avoided it.
- No narcotics.  100% true so far as I was willing to test it.  I actually panicked during customs at the thought that maybe I had some prescription drug in my laptop bag that had a narcotic in it.  Narcotics are a capital crime in Saudi, and I spent most of my 4 hours wondering if Ambien counts as a narcotic.  Luckily for me, I wasn't executed for sleep aid smuggling.

After the worst customs experience of my entire life at the airport in Riyadh, I got to the Marriott hotel for about 4 hours sleep before the day was to begin.  I got to go up to the concierge floor for an elaborate breakfast and a great view of the Riyadh skyline.

The Riyadh venue was spectacular.  The hosts (eSolutions) put on one of the best events I've ever attended on an OTN tour.  From the venue to the signage to the elaborate Lebanese food lunch to the photographer to the videographer to the speaker gifts, it was all top-notch.  Like in Beja, I presented on "Taming Big Data with Analytics" to an enthusiastic audience.  There were about 75 (all male) people there including several male children of the attendees.  They were some of the best behaved children I have ever seen even though some were elementary school age.
I got introduced at one point as "Edward Roske from the United States, a rich man who did not inherit his business from his father."  That was unique and though it left me speechless, the audience seemed very impressed at my ability to become head of a business without my father having to die first.

After the conference, our hosts took us to the National Museum.  Like much of Riyadh, it looks like it was built in the last 5 years, and in the case of a museum, this worked very well.  Since they built it all at once (versus many amazing museums around the world that were created over hundreds of years), it was able to tell a complete story from the beginning of the universe up through today (as opposed to just having rooms of collections).  It was a very Muslim-centric view of history, but I found the educational aspects fascinating.  The sign on the way in said no photography, but since the massive museum apparently only had 4 people working in it (all at the front desk), everyone ignored it.  I found a cube in the first hall that seemed Essbase-like.

My favorite exhibit was a scale model of Mecca and Medina as viewed at night.  Being non-Muslim, I will never see Mecca, so this is as close as I will ever get:
After the museum, our hosts took us to Kingdom Centre, the tallest building in Saudi Arabia (for now).  It has 30+ open stories at the top with a skybridge connecting them.  Here's a view from the ground of the building (notice the necklace like architecture with the bridge on top) and a view from the building of the ground:

Our hosts took us to dinner (which was unfortunately a meat-on-a-stick restaurant, unfortunately for me since I'm a vegetarian).  Since I wasn't able to eat much, I got to talk at length to the CEO of eSolutions.  He was a fascinating man who told me all about how Saudi Arabia and the Gulf Coast Community is ready for analytics and reporting.  He was a former Pakistani military man, but his love of country and family shined through in everything he said.

The following morning, we journeyed to Jeddah over on the Red Sea coast of Saudi near Mecca.  The weather was like Riyadh except with crazy humidity.  It reminded me of Houston on the hottest day of summer.  We were in Jeddah for less than 24 hours so I don't admittedly have a ton to say about it.  I saw nothing more than the airport, the drive to/from the hotel, the venue where we presented, and a fast casual restaurant (more tasty Lebanese food!) where we had dinner.

The venue was small since it was a half-day event and lightly attended.  I did ask if I could present on a different topic since I had gotten a bit bored talking Big Data all the time.  This time, I spoke on "In-Memory Databases" which is a hot topic these days thanks to the SAP guys saying "Hana" at least once every sentence.  After the morning event, the OTN MENA 5-1 (Bjoern went straight to Dubai since there weren't enough speaker slots in Jeddah) headed for the airport for the flight to the United Arab Emirates.  Getting out of Saudi took 2 minutes at Customs which goes to show, I guess, that they're a lot happier to get rid of you than let you in.

Dubai UAE, June 1I honestly don't know where to begin with Dubai.  It is truly one of the most amazing cities on Earth and almost indescribable to anyone who hasn't actually witnessed it.  My flight landed on Saturday evening in the largest airport I think I've ever seen.  Clearing customs took minutes then I headed for the cleanest (and probably newest since it just opened in 2009) train system in the world.  The view from the Dubai Metro was stunning.  If Riyadh looks like it was built in the last 10 years, Dubai looks like it was built in the last 10 minutes.  It reminds me of New York City if they took out all the advertising, 90% of the people, and any building under 1,000 feet tall.


I was lucky enough to stay at a hotel in the Burj Khalifa, the tallest building in the world (by far: the Burj is over 200 stories tall at over 2,500 feet).  The Burj sets all kinds of "tallest" records including the tallest outdoor observation deck.  Here's a view of several buildings below that are all over 1,000 feet tall (with an Edward in it for perspective).

My room had a view of the Dubai Fountains.  No offense, Vegas, but these fountains put the Bellagio fountains to shame. They're more agile, faster, better lit, larger, and frankly, classier. And if you're staying in the hotel in the Burj Khalifa, you can listen to the fountain music through the TV:


At the base of the Burj Khalifa is the Dubai Mall, the largest mall in the world (you hear a lot in Dubai of "that's the largest _____ in the world").  It's 3-4 times the size of Mall of America if that puts it in perspective.  It has over 1,200 shops, over 150 restaurants, and some of the strangest (yet up-scale) stores you've ever seen.  There was one selling full-size metal camels, for instance (these were the only camels I saw on my trip).  I ate at the Dubai Mall every day I was in Dubai and felt I could eat there every day for a year without repeating an entree.  I am normally not a fan of malls, but I loved the Dubai Mall.  It wasn't crowded and the people that were there kept to themselves.  This could actually be said about everywhere in the Middle East: the people are nice, but they keep to themselves.  For an introvert like me, it's heaven.  You can be alone in a crowd.


The first full day I was there, I presented in the morning, burned my feet on the sands of a beautiful beach at lunch, and was skiing in the afternoon.  Yes, skiing.  The Mall of the Emirates (another mall that dwarfs anything we have in the USA) is not only massive, it has its own indoor ski resort inside.  It's not a tiny ski hill either: it's 1,200 feet long with three separate runs (a blue/intermediate run, a green/beginner run, and a small terrain park).  They also have lots of other fun activities to do including a penguin exhibit, ziplining over the ski hill, sledding, and large transparent hamster balls to roll down the hill in.

For around $50 USD, you get a lift ticket, ski pants, ski jacket, boots, skis, and poles for 2 hours.  A full-day pass is only around $15 USD more but I opted for 2 hours.  (You can only ski indoors in Dubai so much, obviously.)  There were at most 15 people skiing, but there were hundreds of people from the Middle East paying their $50 to ride up and down the ski lift basking in the glory of being cold.  The whole place is chilled to around 23F and the snow is glorious: it's soft and velvety because they actually make it snow indoors every night (unlike the ice blowers we use for man-made "snow" in the USA).  I skied for my full 2 hours without a break and I loved that I was so cold by the end that I had to go back into the mall and get a soy hot chocolate at Starbucks.

I had too many amazing experience to recount (and this is already seeming a bit like an advertisement for Dubai) but the most fundamentally changing experience of my trip was a visit to a mosque in Dubai.  Hosted by an eloquent British Muslim woman in a black abbeyah, she spent an hour educating a group of Westerners all about Islam.  I was taken aback by how... peaceful their religion is.  She covered the 5 pillars of Islam in a way that made me understand 1,000,000,000+ Muslims far better than I ever have.  I think that if everyone in the Western world could attend that one hour I did, we would have a level of cultural understanding that would ease a ton of our current fears.  They let us take lots of pictures and even let us video them doing 5 minutes of prayers.  It was moving and if you are ever in Dubai, make sure you visit the Jumeirah Mosque during one of their visitation hours (it's free unlike most everything else in Dubai).

My final presentation was in Dubai and it was a bittersweet end to a whirlwind week.  Every single day was spent traveling, speaking, or both, so it was nice to finally have a break.  That said, I will consider the other members of the OTN MENA 5 to be friends for life and I miss them already.  As I finished my presentation on Big Data (for the third time on the trip), I looked out at the anxious faces in the audience and realized that I would miss the Muslim world far more than I ever expected to.  

If there's ever a 2nd OTN tour of the Middle East, Africa, or both, sign me up.  Until then, thank you for letting me be a part of the most inspirational, educational tour I've ever experienced.

Categories: BI & Warehousing

Fun with SQL - Silver Pockets Full

Chet Justice - Wed, 2014-06-04 17:13
Silver Pockets Full, send this message to your friends and in four days the money will surprise you. If you don't, well, a pox on your house. Or something like that. I didn't know what it was, I just saw this in my FB feed:



Back in November, I checked to see the frequency of having incremental numbers in the date, like 11/12/13 (my birthday) and 12/13/14 (kate's birthday). I don't want to hear how the rest of the world does their dates either, I know (I now write my dates like YYYY/MM/DD on everything, just so you know, that way I can sort it...or something).

Anyway, SQL to test out the claim of once every 823 years. Yay SQL.

OK, I'm not going to go into the steps necessary because I'm lazy (and I'm just lucky to be writing here), so here it is:
select *
from
(
select
to_char( d, 'yyyymm' ) year_month,
count( case
when to_char( d, 'fmDay' ) = 'Saturday' then 1
else null
end ) sats,
count( case
when to_char( d, 'fmDay' ) = 'Sunday' then 1
else null
end ) suns,
count( case
when to_char( d, 'fmDay' ) = 'Friday' then 1
else null
end ) fris
from
(
select to_date( 20131231, 'yyyymmdd' ) + rownum d
from dual
connect by level <= 50000
)
group by
to_char( d, 'yyyymm' )
)
where fris = 5
and sats = 5
and suns = 5
So over the next 50,000 days, this happens 138 times. I'm fairly certain that doesn't rise to the once every 823 years claim. But it's cool, maybe.
YEAR_MONTH       SATS       SUNS       FRIS
---------- ---------- ---------- ----------
201408 5 5 5
201505 5 5 5
201601 5 5 5
201607 5 5 5
201712 5 5 5
128 more occurrences...
214607 5 5 5
214712 5 5 5
214803 5 5 5
214908 5 5 5
215005 5 5 5

138 rows selected
I'm not the only dork that does this either, here's one in perl. I'm sure there are others, but again, I'm lazy.
Categories: BI & Warehousing

The Riley Family, Part III

Chet Justice - Thu, 2014-04-10 21:44


That's Mike and Lisa, hanging out at the hospital. Mike's in his awesome cookie monster pajamas and robe...must be nice, right? Oh wait, it's not. You probably remember why he's there, Stage 3 cancer. The joys.

In October, we helped to send the entire family to Game 5 of the World Series (Cards lost, thanks Red Sox for ruining their night).

In November I started a GoFundMe campaign, to date, with your help, we've raised $10,999. We've paid over 9 thousand dollars to the Riley family (another check to be cut shortly).

In December, Mike had surgery. Details can be found here. Shorter: things went fairly well, then they didn't. Mike spent 22 days in the hospital and lost 40 lbs. He missed Christmas and New Years at home with his family. But, as I've learned over the last 6 months, the Riley family really knows how to take things in stride.

About 6 weeks ago Mike started round 2 of chemo, he's halfway through that one now. He complains (daily, ugh) about numbness, dizziness, feeling cold (he lives in St. Louis, are you sure it's not the weather?), and priapism (that's a lie...I hope).

Mike being Mike though, barely a complaint (I'll let you figure out where I'm telling a lie).

Four weeks ago, a chilly (65) Saturday night, Mike and Lisa call. "Hey, I've got some news for you."

"Sweet," I think to myself. Gotta be good news.

"Lisa was just diagnosed with breast cancer."

WTF?

ARE YOU KIDDING ME? (Given Mike's gallows humor, it's possible).

"Nope. Stage 1. Surgery on April 2nd."

FFS

(Surgery was last week. It went well. No news on that front yet.)

Talking to them two of them that evening you would have no idea they BOTH have cancer. Actually, one of my favorite stories of the year...the hashtag for Riley Family campaign was #fmcuta. Fuck Mike's Cancer (up the ass). I thought that was hilarious, but I didn't think the Riley's would appreciate it. They did. They loved it. I still remember Lisa's laugh when I first suggested it. They've dropped the latest bad news and Lisa is like, "Oh, wait until you hear this. I have a hashtag for you."

"What is it?" (I'm thinking something very...conservative. Not sure why, I should know better by now).

#tna

I think about that for about .06 seconds. Holy shit! Did you just say tna? Like "tits and ass?"

(sounds of Lisa howling in the background).

Awesome. See what I mean? Handling it in stride.

"We're going to need a bigger boat." All I can think about now is, "what can we do now?"

First, I raised the campaign goal to 50k. This might be ambitious, that's OK, cancer treatments are expensive enough for one person, and 10K (the original amount) was on the low side. So...50K.

Second, Scott Spendolini created a very cool APEX app, ostensibly called the Riley Support Group (website? gah). It's a calendar/scheduling app that allows friends and family coordinate things like meals, young human (children) care and other things that most of us probably take for granted. Pretty cool stuff. For instance, Tim Gorman provides pizza on Monday nights (Dinner from pizza hut...1 - large hand-tossed cheese lovers, 1 - large thin-crispy pepperoni, 1 - 4xpepperoni rolls, 1 - cheesesticks).

Third. There is no third.

So many of you have donated your hard earned cash to the Riley family, they are incredibly humbled by, and grateful for, everyone's generosity. They aren't out of the woods yet. Donate more. Please. If you can't donate, see if there's something you can help out with (hit me up for details, Tim lives in CO, he's not really close). If you can't do either of those things, send them your prayers or your good thoughts. Any and all help will be greatly appreciated.
Categories: BI & Warehousing

What's New in Dodeca 6.7.1?

Tim Tow - Mon, 2014-03-24 11:28
Last week, we released Dodeca version 6.7.1.4340 which focuses on some new relational functionality. The major new features in 6.7.1 are:
  • Concurrent SQL Query Execution
  • Detailed SQL Timed Logging
  • Query and Display PDF format
  • Ability to Launch Local External Processes from Within Dodeca
Concurrent SQL Query ExecutionDodeca has a built-in SQLPassthroughDataSet object that supports queries to a relational database.  The SQLPassthroughDataSet functionality was engineered such that a SQLPassthroughDataSet object can include multiple queries that get executed and returned on a single trip to the server and several of our customers have taken great advantage of that functionality.  We have at least one customer, in fact, that has some SQLPassthroughDataSets that execute up to 20 queries in a single trip to the server.  The functionality was originally designed to run the queries sequentially, but in some cases it would be better to run the queries concurrently.  Because this is Dodeca, of course concurrent query execution is configurable at the SQLPassthroughDataSet level.

Detailed SQL Timed LoggingIn Dodeca version 6.0, we added detailed timed logging for Essbase transactions.  In this version, we have added similar functionality for SQL transactions and have formatted the logs in pipe-delimited format so they can easily be loaded into Excel or into a database for further analysis.  The columns of the log include the log message level, timestamp, sequential transaction number, number of active threads, transaction GUID, username, action, description, and time to execute in milliseconds.

Below is an example of the log message.

Click to enlarge

Query and Display PDF formatThe PDF View type now supports the ability to load the PDF directly from a relational table via a tokenized SQL statement.  This functionality will be very useful for those customers who have contextual information, such as invoice images, stored relationally and need a way to display that information.  We frequently see this requirement as the end result of a drill-through operation from either Essbase or relational reports.

Ability to Launch Local External Processes from Within DodecaCertain Dodeca customers store data files for non-financial systems in relational data stores and use Dodeca as a central access point.  The new ability to launch a local process from within Dodeca is implemented as a Dodeca Workbook Script method which provides a great deal of flexibility in how the process is launched.

The new 6.7.1 functionality follows closely on the 6.7.0 release that introduces proxy server support and new MSAD and LDAP authentication services.  If you are interested in seeing all of the changes in Dodeca, highly detailed Dodeca release notes are available on our website at http://www.appliedolap.com/resources/downloads/dodeca-technical-docs.

Categories: BI & Warehousing

Internal Links

Tim Dexter - Wed, 2014-03-05 20:06

Another great question today, this time, from friend and colleague, Jerry the master house re-fitter. I think we are competing on who can completely rip and replace their entire house in the shortest time on their own. Every conversation we have starts with 'so what are you working on?' He's in the midst of a kitchen re-fit, Im finishing off odds and ends before I re-build our stair well and start work on my hidden man cave under said stairs. Anyhoo, his question!

Can you create a PDF document that shows a summary on the first page and provides links to more detailed sections further down in the document?

Why yes you can Jerry. Something like this? Click on the department names in the first table and the return to top links in the detail sections. Pretty neat huh? Dynamic internal links based on the data, in this case the department names.

Its not that hard to do either. Here's the template, RTF only right now.


The important fields in this case are the ones in red, heres their contents.

TopLink

<fo:block id="doctop" />

Just think of it as an anchor to the top of the page called doctop

Back to Top

<fo:basic-link internal-destination="doctop" text-decoration="underline">Back to Top</fo:basic-link>

Just a live link 'Back to Top' if you will, that takes the user to the doc top location i.e. to the top of the page.

DeptLink

<fo:block id="{DEPARTMENT_NAME}"/>

Just like the TopLink above, this just creates an anchor in the document. The neat thing here is that we dynamically name it the actual value of the DEPARTMENT_NAME. Note that this link is inside the for-each:G_DEPT loop so the {DEPARTMENT_NAME} is evaluated each time the loop iterates. The curly braces force the engine to fetch the DEPARTMENT_NAME value before creating the anchor.

DEPARTMENT_NAME

<fo:basic-link  internal-destination="{DEPARTMENT_NAME}" ><?DEPARTMENT_NAME?></fo:basic-link>

This is the link for the user to be able to navigate to the detail for that department. It does not use a regular MSWord URL, we have to create a field in the template to hold the department name value and apply the link. Note, no text decoration this time i.e. no underline.

You can add a dynamic link on to anything in the summary section. You just need to remember to keep link 'names' as unique as needed for source and destination. You can combine multiple data values into the link name using the concat function.

Template and data available here. Tested with 10 and 11g, will work with all BIP flavors.

Categories: BI & Warehousing

The OLAP Extension is now available in SQL Developer 4.0

Keith Laker - Tue, 2014-03-04 14:57


The OLAP Extension is now in SQL Developer 4.0.

See
http://www.oracle.com/technetwork/developer-tools/sql-developer/downloads/sqldev-releasenotes-v4-1925251.html for the details.

The OLAP functionality is mentioned toward the bottom of the web page.
You will still need AWM 12.1.0.1.0 to
  • Manage and enable cube and dimension MV's.
  • Manage data security.
  • Create and edit nested measure folders (i.e. measure folders that are children of other measure folders)
  • Create and edit Maintenance Scripts
  • Manage multilingual support for OLAP Metadata objects
  • Use the OBIEE plugin or the Data Validation plugin
What is new or improved:
  • New Calculation Expression editor for calculated measures.  This allows the user to nest different types to calculated measures easily.  For instance a user can now create a Moving Total of a Prior Period as one calculated measure.  In AWM, it would have required a user to create a Prior Period first and then create a Moving Total calculated measure which referred to the Prior Period measure.  Also the new Calculation Expression editor displays hypertext helper templates when the user selects the OLAP API syntax in the editor.
  • Support for OLAP DML command execution in the SQL Worksheet.  Simply prefix OLAP DML commands by a '~' and then select the execute button to execute them on the SQL Worksheet.  The output of the command will appear in the DBMS Output Window if it is opened, or the Script Output Window if the user has executed 'set serveroutput on' before executing the DML command.
  • Improved OLAP DML Program Editor integrated within the SQL Developer framework.
  • New diagnostic reports in the SQL Developer Report navigator.
  • Ability to create a fact view with a measure dimension (i.e. "pivot cube").  This functionality is accessible from the SQL Developer Tools-OLAP menu option.
  • Cube scripts have been renamed to Build Specifications and are now accessible within the Create/Edit Cube dialog.  The Build Specifications editor there, is similar to the calculation expression editor as far as functionality.
Categories: BI & Warehousing

Waterfall Charts

Tim Dexter - Fri, 2014-02-28 19:35

Great question came through the ether from Holger on waterfall charts last night.

"I know that Answers supports waterfall charts and BI Publisher does not.
Do you have a different solution approach for waterfall charts with BI Publisher (perhaps stacked bars with white areas)?
Maybe you have already implemented something similar in the past and you can send me an example."

I didnt have one to hand, but I do now. Little known fact, the Publisher chart engine is based on the Oracle Reports chart engine. Therefore, this document came straight to mind. Its awesome for chart tips and tricks. Will you have to get your hands dirty in the chart code? Yep. Will you get the chart you want with a little effort? Yep. Now, I know, I know, in this day and age, you should get waterfalls with no effort but then you'd be bored right?

First things first, for the uninitiated, what is a waterfall chart? From some kind person at Wikipedia, "The waterfall chart is normally used for understanding how an initial value is affected by a series of intermediate positive or negative values. Usually the initial and the final values are represented by whole columns, while the intermediate values are denoted by floating columns. The columns are color-coded for distinguishing between positive and negative values."

We'll get back to that last sentence later, for now lets get the basic chart working.

Checking out the Oracle Report charting doc, search for 'floating' their term for 'waterfall' and it will get you to the section on building a 'floating column chart' or in more modern parlance, a waterfall chart. If you have already got your feet wet in the dark arts world of Publisher chart XML, get on with it and get your waterfall working.

If not, read on.

When I first starting looking at this chart, I decided to ignore the 'negative values' in the definition above. Being a glass half full kind of guy I dont see negatives right :)

Without them its a pretty simple job of rendering a stacked bar chart with 4 series for the colors. One for the starting value, one for the ending value, one for the diffs (steps) and one for the base values. The base values color could be set to white but that obscures any tick lines in the chart. Better to use the transparency option from the Oracle Reports doc.

<Series id="0" borderTransparent="true" transparent="true"/> 

Pretty simple, even the data structure is reasonably easy to get working. But, the negative values was nagging at me and Holger, who I pointed at the Oracle Reports doc had come back and could not get negative values to show correctly. So I took another look. What a pain in the butt!

In the chart above (thats my first BIP waterfall maybe the first ever BIP waterfall.) I have lime green, start and finish bars; red for negative and green for positive values. Look a little closer at the hidden bar values where we transition from red to green, ah man, royal pain in the butt! Not because of anything tough in the chart definition, thats pretty straightforward. I just need the following columns START, BASE, DOWN, UP and FINISH. 

START 200
BASE 0
UP 0
DOWN 0
FINISH 0
START 0
BASE 180
UP 0
DOWN 20
FINISH 0
START 0
BASE 150
UP 0
DOWN 30
FINISH 0
 Bar 1 - Start Value
 Bar 2 - PROD1
 Bar 3 - PROD2

and so on. The start, up, down and finish values are reasonably easy to get. The real trick is calculating that hidden BASE value correctly for that transition from -ve >> + ve and vice versa. Hitting Google, I found the key to that calculation in a great page on building a waterfall chart in Excel from the folks at Contextures.  Excel is great at referencing previous cell values to create complex calculations and I guess I could have fudged this article and used an Excel sheet as my data source. I could even have used an Excel template against my database table to create the data for the chart and fed the resulting Excel output back into the report as the data source for the chart. But, I digress, that would be tres cool thou, gotta look at that.
On that page is the formula to get the hidden base bar values and I adapted that into some sql to get the same result.

Lets assume I have the following data in a table:

PRODUCT_NAME SALES PROD1 -20 PROD2 -30 PROD3 50 PROD4 60

The sales values are versus the same period last year i.e. a delta value.  I have a starting value of 200 total sales, lets assume this is pulled from another table.
I have spent the majority of my time on generating the data, the actual chart definition is pretty straight forward. Getting that BASE value has been most tricksy!

I need to generate the following for each column:

PRODUCT_NAME

STRT

BASE_VAL

DOWN

UP

END_TOTAL

START
200
0
0
0
0
PROD1
0
180
20
0
0
PROD2
0
150 30 0
0
PROD3
0 150 0 50 0 PROD4
0 200
0 60 0 END
0 0 0 0 260

Ignoring the START and END values for a second. Here's the query for the PRODx columns:

 SELECT 2 SORT_KEY 
, PRODUCT_NAME
, STRT
, SALES
, UP
, DOWN
, 0 END_TOTAL
, 200 + (SUM(LAG_UP - DOWN) OVER (ORDER BY PRODUCT_NAME)) AS BASE_VAL
FROM
(SELECT P.PRODUCT_NAME
,  0 AS STRT
, P.SALES
, CASE WHEN P.SALES > 0 THEN P.SALES ELSE 0 END AS UP  
, CASE WHEN P.SALES < 0 THEN ABS(P.SALES) ELSE 0 END AS DOWN
, LAG(CASE WHEN P.SALES > 0 THEN P.SALES ELSE 0 END,1,0) 
      OVER (ORDER BY P.PRODUCT_NAME) AS LAG_UP
FROM PRODUCTS P
)

The inner query is breaking the UP and DOWN values into their own columns based on the SALES value. The LAG function is the cool bit to fetch the UP value in the previous row. That column is the key to getting the BASE values correctly.

The outer query just has a calculation for the BASE_VAL.

200 + (SUM(LAG_UP - DOWN) OVER (ORDER BY PRODUCT_NAME))

The SUM..OVER allows me to iterate over the rows to get the calculation I need ie starting value (200) + the running sum of LAG_UP - DOWN. Remember the LAG_UP value is fetching the value from the previous row.
Is there a neater way to do this? Im most sure there is, I could probably eliminate the inner query with a little effort but for the purposes of this post, its quite handy to be able to break things down.

For the start and end values I used more queries and then just UNIONed the three together. Once note on that union; the sorting. For the chart to work, I need START, PRODx, FINISH, in that order. The easiest way to get that was to add a SORT_KEY value to each query and then sort by it. So my total query for the chart was:

SELECT 1 SORT_KEY
, 'START' PRODUCT_NAME
, 200 STRT
, 0 SALES
, 0 UP
, 0 DOWN
, 0 END_TOTAL
, 0 BASE_VAL
FROM PRODUCTS
UNION
SELECT 2 SORT_KEY 
, PRODUCT_NAME
, STRT
, SALES
, UP
, DOWN
, 0 END_TOTAL
, 200 + (SUM(LAG_UP - DOWN) 
      OVER (ORDER BY PRODUCT_NAME)) AS BASE_VAL
FROM
(SELECT P.PRODUCT_NAME
,  0 AS STRT
, P.SALES
, CASE WHEN P.SALES > 0 THEN P.SALES ELSE 0 END AS UP  
, CASE WHEN P.SALES < 0 THEN ABS(P.SALES) ELSE 0 END AS DOWN
, LAG(CASE WHEN P.SALES > 0 THEN P.SALES ELSE 0 END,1,0) 
       OVER (ORDER BY P.PRODUCT_NAME) AS LAG_UP
FROM PRODUCTS P
)
UNION
SELECT 3 SORT_KEY 
, 'END' PRODUCT_NAME
, 0 STRT
, 0 SALES
, 0 UP
, 0 DOWN
, SUM(SALES) + 200 END_TOTAL
, 0 BASE_VAL
FROM PRODUCTS
GROUP BY 1,2,3,4,6
ORDER BY 1 

A lot of effort for a dinky chart but now its done once, doing it again will be easier. Of course no one will want just a single chart in their report, there will be other data, tables, charts, etc. I think if I was doing this in anger I would just break out this query as a separate item in the data model ie a query just for the chart. It will make life much simpler.
Another option that I considered was to build a sub template in XSL to generate the XML tree to support the chart and assign that to a variable. Im sure it can be done with a little effort, I'll save it for another time.

On the last leg, we have the data; now to build the chart. This is actually the easy bit. Sadly I have found an issue in the online template builder that precludes using the chart builder in those templates. However, RTF templates to the rescue!

Insert a chart and in the dialog set up the data like this (click the image to see it full scale.)

Its just a vertical stacked bar with the BASE_VAL color set to white.You can still see the 'hidden' bars and they are over writing the tick lines but if you are happy with it, leave it as is. You can double click the chart and the dialog box can read it no problem. If however, you want those 'hidden' bars truly hidden then click on the Advanced tab of the chart dialog and replace:

<Series id="1" color="#FFFFFF" />

with

<Series id="1" borderTransparent="true" transparent="true" />

and the bars will become completely transparent. You can do the #D and gradient thang if you want and play with colors and themes. You'll then be done with your waterfall masterpiece!

Alot of work? Not really, more than out of the box for sure but hopefully, I have given you enough to decipher the data needs and how to do it at least with an Oracle db. If you need all my files, including table definition, sample XML, BIP DM, Report and templates, you can get them here.

Categories: BI & Warehousing

Wildcard Filtering continued

Tim Dexter - Mon, 2014-02-24 13:27

I wrote up a method for using wildcard filtering in your layouts a while back here. I spotted a followup question on that blog post last week and thought I would try and address it using another wildcard method. 

I want to use the bi publisher to look for several conditions using a wild card. For example if I was sql it would look like this:

if name in ('%Wst','%Grt')

How can I utilize bi publisher to look for the same conditions.

This, in XPATH speak is an OR condition and we can treat it as such. In the last article I used the 'starts-with' function, its a little limiting, there is a better one, 'contains'. This is a much more powerful function that allows you to look for any string within another string. Its case insensitive so you do not need to do upper or lowering of the string you are searching to get the desired results.
Here it is in action:

For the clerks filter I use :

<?for-each-group:G_1[contains(JOB_TITLE,'Clerk')];./JOB_TITLE?>

and to find all clerks and managers, I use:

<?for-each-group:G_1[contains(JOB_TITLE,'Clerk') or contains(JOB_TITLE,'Manager')];./JOB_TITLE?>

Note that Im using re-grouping here, you can use the same XPATH with a regular for-each. Also note the lower case 'or' in the second expression. You can also use an 'and' too.

This works in 10 and 11g flavors of BIP. Sample files available here.

Categories: BI & Warehousing

Filtered Charts

Tim Dexter - Tue, 2014-02-11 17:19

A customer question this week regarding filtering a chart. They have a report with a bunch of criteria with monetary values but, rather than show all of the criteria in a pie chart, they just want to show a few. For example:

 This ...
 rather than this

 There are a couple of ways to tackle this:

1. Filter the chart data in the chart definition. Using an XPATH expression you can filter out all of the criteria you do not want to see. Open the chart definition and update the definition. You will need to update the RowCount, RowLabels and DataValues attributes in the chart definition. Adding in the following XPATH expression:

    [DEPARTMENT_NAME='Accounting' or DEPARTMENT_NAME='Marketing' or DEPARTMENT_NAME='Executive']

so the DataValues value becomes:

    <DataValues><xsl:for-each-group select=".//G_1[DEPARTMENT_NAME='Accounting' or 
                                                     DEPARTMENT_NAME='Marketing' or 
                                                        DEPARTMENT_NAME='Executive']" ...

2. Create a variable in the template to hold just the values you want to chart.

    <?variable: filterDepts; /DATA_DS/LIST_G_1/G_1[DEPARTMENT_NAME='Accounting' or 
                                                     DEPARTMENT_NAME='Marketing' or 
                                                       DEPARTMENT_NAME='Executive']?>

Then update the chart definition with the variable for the same three attributes above, the RowCount, RowLabels and DataValues. For example:

    <DataValues><xsl:for-each-group select="$filterDepts" ...

These both work admirably, but they both require some manual updating of the chart definition which can get fiddly and a pain to maintain. I'm also just filtering for three departments, when you get up to 5 or 6 then the XPATH starts to become a pain to maintain. Option 2 alleviates this somewhat because you only need to define the filter once to create the filtered variable.
A better option may be ...

3. Force the effort down into the data layer. Create another query in the report that just pulls the data for the chart.

LIST_G2/G_2 holds the data for the chart. Then all you need do is create a vanilla chart on that particular section of the data.

Yes, there is some overhead to re-fetch the data but this is going to be about the same if not less than the extra processing required in the template with options 1 and 2. This has another advantage, you can parametrize the criteria for the user. You can create a parameter to allow the user to select, in my case, the department(s) they want to chart.


Its simple enough to create the multi-select parameter and modify the query to filter based on the values chosen by the user.

Sample report (including data model and layout template here) just un-archive into your catalog.
RTF Template plus sample data available here.




Categories: BI & Warehousing

Pages

Subscribe to Oracle FAQ aggregator - BI &amp; Warehousing