Jeff Moss

Subscribe to Jeff Moss feed Jeff Moss
Information Warriors
Updated: 7 hours 14 min ago

Fixing blank charts on ambari home page (Hortonworks Data Platform)

Thu, 2017-02-09 15:54

I created a 4 node container based (Proxmox LXC) Hortonworks Data Platform 2.5 Hadoop cluster recently and all went well apart from all the charts on the Ambari homepage were blank or showing “N/A”, like this:

An outline of the environment:

  • 4 node cluster of LXC containers on Proxmox host
  • Centos 7 Linux OS
  • Nodes are called bishdp0[1-4], all created from same template and identical configuration
  • All containers are on network
  • DNS Server also available on same network and all hosts can resolve each other via DNS
  • Hortonworks Data Platform version 2.5
  • Proxmox host sits on a corporate network and the host has iptables set to allow the containers on to reach the internet via the corporate proxy server, e.g. for yum access
  • Other than the blank charts everything appears to be working fine

After much reading around it turns out that I hadn’t quite set up the proxy serving correctly, specifically that I hadn’t told Ambari to ignore some hosts, namely the bishdp0[1-4] hosts on the network, when proxying. I can’t find a 2.5 HDP version of the document for setting up the proxy serving for Ambari but the 2.2 instructions worked.

Steps I took to fix the problem:

First stop the services on the cluster. Log on to the node with the Ambari Server where I have a script called which I created based on part of this article. Thanks slm.

Run the script:


Now stop the Ambari agent on all the servers:

pdsh -w bishdp0[1-4] service ambari-agent stop

Now stop the Ambari Server:

service ambari-server stop

Now edit the Ambari environment script:

vi /var/lib/ambari-server/

Look for the line that begins “export AMBARI_JVM_ARGS” and ensure it has entries for the following parameters:

  • http.proxyHost
  • http.proxyPort
  • http.proxyUser
  • http.proxyPassword
  • http.nonProxyHosts

It’s the last one that was missing in my case, which meant that Ambari was trying to go to the proxy server even for these containers on the network.

After editing, the line looked like this (I’ve redacted the specifics – just replace the entries with values suited to your environment):

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS’ -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Dhttp.proxyHost=<proxy IP> -Dhttp.proxyPort=<proxy port> -Dhttp.proxyUser=<user> -Dhttp.proxyPassword=<password> -Dhttp.nonProxyHosts=<*.domain>$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf’

Now restart everything, Ambari server first:

service ambari-server start

…then the agents on all nodes (pdsh is great – thanks Robin Moffatt for your notes!)

pdsh -w bishdp0[1-4] service ambari-agent start

And finally start the services on the cluster using the script.


After I did this, the charts started showing details:

Transferring default domains for SQL Developer Data Modeller

Wed, 2016-11-16 08:46
*{-webkit-box-sizing: border-box;-moz-box-sizing: border-box;box-sizing: border-box}div{display: block}.kc-container{width:100%;max-width:1170px;margin:0 auto;padding-left:15px;padding-right:15px;box-sizing:border-box}.kc-row-container:not(.kc-container){padding-left:0;padding-right:0;width:100%;max-width:100%}.kc-elm{float: left;width: 100%;}.kc_wrap-video-bg{height:100%;left:0;overflow:hidden;pointer-events:none;position:absolute;top:0;width:100%;z-index:0}.kc_single_image img{max-width:100%}.kc-video-bg .kc_column{position:relative}.kc-infinite-loop{text-align:center;padding:50px;font-size:18px;color:red;width:100%;display:inline-block}.kc_row:not(.kc_row_inner){clear:both;display:block;width:100%}.kc-wrap-columns,.kc_row_inner{margin-left:-15px;margin-right:-15px;clear:both}.kc_row.kc_row_inner{width: calc(100% + 30px);}.kc_tab_content>.kc_row_inner{width:100%;margin:0}.kc_column,.kc_column_inner{min-height:1px;position:relative;padding-right:15px;padding-left:15px;width:100%;float:left}div.kc_column,div.kc_column_inner{clear:none}div[data-kc-fullheight]{min-height:100vh}html body div[data-kc-parallax=true]{background-position:50% 0;background-size:100%!important;background-repeat:no-repeat!important;background-attachment:fixed!important}div[data-kc-fullwidth]{margin-left:0!important;margin-right:0!important;position:relative;box-sizing:content-box}.kc_text_block{display:inline-block;clear:both;width:100%}@media screen and (min-width:999px){body div[data-kc-equalheight=true],body div[data-kc-equalheight=true]>.kc-container{display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important}body div[data-kc-equalheight-align=middle]>.kc-container>.kc-wrap-columns>.kc_column>.kc-col-container{display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;align-items:center;flex-wrap:wrap;justify-content:center;height:100%}body div[data-kc-equalheight-align=bottom]>.kc-container>.kc-wrap-columns>.kc_column>.kc-col-container{display:-webkit-flex!important;display:-ms-flexbox!important;display:flex!important;align-items:flex-end;flex-wrap:wrap;justify-content:center;height:100%}body div[data-kc-fullheight=middle-content],body div[data-kc-fullheight=middle-content]>.kc-container{display:-webkit-flex;display:-ms-flexbox;display:flex;align-items:center}.kc-wrap-columns,.kc_row_inner{display:-webkit-flex;display:-ms-flexbox;display:flex}.kc_row_inner, .kc-row-container.kc-container .kc-wrap-columns{width: calc(100% + 30px)}}@media screen and (max-width: 767px){html body .kc_column,html body .kc_column_inner{width: 100%}div.kc_row{display: block}}@media screen and (max-width: 999px){.kc_col-sm-3, div.kc_col-of-5{width: 50%}}.kc_col-sm-1{width: 8.33333%}.kc_col-sm-2{width: 16.6667%}div.kc_col-of-5{width: 20%;float: left}.kc_col-sm-3{width: 25%}.kc_col-sm-4{width: 33.3333%}.kc_col-sm-5{width: 41.6667%}.kc_col-sm-6{width: 50%}.kc_col-sm-7{width: 58.3333%}.kc_col-sm-8{width: 66.6667%}.kc_col-sm-9{width: 75%}.kc_col-sm-10{width: 83.3333%}.kc_col-sm-11{width: 91.6667%}.kc_col-sm-12{width: 100%}.kc-off-notice{display:none;}

Notice: You are using wrong way to display KC Content, Correct It Now

I got a new laptop the other day and installed all the software, including SQL Developer Data Modeller all fine. I then opened a model which had a bunch of tables with columns based off Domains…the columns did not have Domains against them but had been replaced with Logical data types instead.

After some research, the fix, in this case, involved copying the file “defaultdomains.xml” from the following directory on my old laptop, to the same place on the new laptop:

%SQL Developer Home%\sqldeveloper\extensions\oracle.datamodeler\types

After restarting and reopening the model all was back to normal.

What I probably could have done in the first place was to have created my own Domains file for the Design, saved in the Design folder and then when I transferred the Design by copying across the Design folder the domains would have come with it. I could have then just opened the Domain file on the new laptop. I guess it depends on whether I would want these domains to be Design specific or part of the defaults for all designs.

SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

Creating Oracle Big Data Lite VM on Proxmox

Wed, 2016-11-16 02:42

The Oracle Big Data Lite VM available on Oracle technet, provides a pre built environment for learning about a number of key Oracle products, including Oracle 12c database, Big Data Discovery and Data integrator as well as Cloudera Distribution – Apache Hadoop (CDH 5.8.0).

The download ultimately delivers an OVA “appliance” file for use with Oracle VirtualBox, but there isn’t anything to stop you running this as a VM on proxmox 4, with a bit of effort, as follows.

NOTE – Things to read which can help with this process:

  1. Oracle Big Data Lite Deployment Guide.
  2. How to upload an OVA to proxmox guide by James Coyle:
  3. Converting to RAW and pushing to a raw lvm partition:
  • Firstly download the files that make up the OVA from here.
  • Follow the instructions on the download page to convert the multiple files into one single OVA file.
  • For Oracle Virtualbox, simple follow the rest of the instructions in the Deployment Guide.
  • For Proxmox, where I was running LVM storage for the virtual machines, first rename the single OVA file to .ISO, then upload that file (BigDataLite460.iso) to a storage area on your proxmox host, in this case, mine was called “data”. You can upload the file through the Proxmox GUI, or manually via the command line. My files were uploaded through the GUI and end up in “/mnt/pve-data/template/iso”.
  • Now, bring up a shell and navigate to the ISO directory and then unpack the ISO file by running “tar xvf BigDataLite460.iso”. This should create five files which include one OVF file (Open Virtualisation Format) and four VMDK files (Virtual Machine Disk).
root@HP20052433:/mnt/pve-data/template/iso# ls -l
total 204127600
-rw------- 1 root root   8680527872 Oct 25 02:43 BigDataLite460-disk1.vmdk
-rw------- 1 root root   1696855040 Oct 25 02:45 BigDataLite460-disk2.vmdk
-rw------- 1 root root  23999689216 Oct 25 03:11 BigDataLite460-disk3.vmdk
-rw------- 1 root root       220160 Oct 25 03:11 BigDataLite460-disk4.vmdk
-rw-r--r-- 1 root root  34377315328 Nov 14 10:59 BigDataLite460.iso
-rw------- 1 root root        20056 Oct 25 02:31 BigDataLite460.ovf
  • Now, create a new VM in proxmox via the GUI or manually. The VM I created had the required memory and CPUs as per the deployment guide, together with four Hard Disks – mine were all on the SCSI interface and were set to be 10G in size initially – this will change later.
  • The hard disks were using a storage area on Proxmox that was defined as type LVM.
  • Now convert the VMDK files to RAW files which we’ll then push to the LVM Hard Disks as follows:
qemu-img convert -f vmdk BigDataLite460-disk1.vmdk -O raw BigDataLite460-disk1.raw
qemu-img convert -f vmdk BigDataLite460-disk2.vmdk -O raw BigDataLite460-disk2.raw
qemu-img convert -f vmdk BigDataLite460-disk3.vmdk -O raw BigDataLite460-disk3.raw
qemu-img convert -f vmdk BigDataLite460-disk4.vmdk -O raw BigDataLite460-disk4.raw
  • Now list those raw files, so we can see their sizes:
root@HP20052433:/mnt/pve-data/template/iso# ls -l *.raw
-rw-r--r-- 1 root root 104857600000 Nov 16 07:58 BigDataLite460-disk1.raw
-rw-r--r-- 1 root root 214748364800 Nov 16 08:01 BigDataLite460-disk2.raw
-rw-r--r-- 1 root root 128849018880 Nov 16 08:27 BigDataLite460-disk3.raw
-rw-r--r-- 1 root root  32212254720 Nov 16 08:27 BigDataLite460-disk4.raw
  • Now resize the lvm hard disks to the corresponding sizes (the ID of my proxmox VM was 106 and my hard disks were scsi):
qm resize 106 scsi0 104857600000
qm resize 106 scsi1 214748364800
qm resize 106 scsi2 128849018880
qm resize 106 scsi3 32212254720
  • Now copy over the content of the raw files to the corresponding lvm hard disks:
dd if=BigDataLite460-disk1.raw of=/dev/vm_storage_group/vm-106-disk-1
dd if=BigDataLite460-disk2.raw of=/dev/vm_storage_group/vm-106-disk-2
dd if=BigDataLite460-disk3.raw of=/dev/vm_storage_group/vm-106-disk-3
dd if=BigDataLite460-disk4.raw of=/dev/vm_storage_group/vm-106-disk-4
  • Now start the VM and hey presto there it is.
  • You could stop there as it’s a self contained environment, but obviously you can also do a whole bunch of networking stuff to make it visible on your network as well.
SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

Microsoft Accounts Fail To Log In To Windows 10 with “User Profile Service failed the login” Error.

Wed, 2016-10-19 01:05

My kids are getting to the age where they can’t keep away from the laptop, various pads or the Smart TV to go online…time for some protection I thought.

I figured, for the Windows 10 laptops, that I’d use the Microsoft Accounts approach and use the “big brother” features there to stop the kids watching things they shouldn’t and restrict their access time.

First step was to convert my local account into a Microsoft one – simple enough and worked fine.

Next step was to create additional Microsoft accounts and then have them linked up as part of the “Family” – again, fine.

Then tell the PC to add those users – again all fine and simple to do.

All going well up until now, but then when I try to logout of my working Microsoft account on the laptop and login to one of the Family Microsoft accounts, it fails with the “User profile Service failed the login:

Image result for user profile service service logon fail windows 10


After much googling and trying various things, the one which worked for me was to copy the directory C:\Users\Default from a working Windows 7 Ultimate machine onto the laptop with the problem (where the directory did not exist at all). The advice I found actually referred to copying from another Windows 10 machine, but I didn’t have one of those – only a Windows 7 one.

I then added the family Microsoft accounts back in and after logging out and trying to login as one of these added accounts then worked fine!

I can’t be certain what the issue was, but various reading suggested an issue where the machine was upgraded from Windows 7/8 to 10 and where the local profile (C:\Users\Default) was either missing or corrupted. Copying in a working one from another machine fixed the issue in my case.

SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

OTN Appreciation Day: Automatic Storage Management (ASM)

Tue, 2016-10-11 04:48

Big shout out to Tim for kicking this off!

Automatic Storage Management (ASM) provides optimised volume management and filesystem capabilities for Oracle databases, whether they be single or multi instance (RAC) implementations.

Although introduced with Oracle 10g Release 1 in 2004, I first used it in a production scenario around 2008, when upgrading a hardware platform for a Data Warehouse. It seemed like a logical choice for myself and the DBAs at the time, although the storage team were less pleased at losing some control. Ultimately it proved a big success on that project and is still in stable, reliable use today.

Things I like about ASM include:

  • Simplifies storage management
  • Automatic rebalancing when capacity is added
  • Visibility within Enterprise Manager for monitoring
  • Availability of detailed metrics within the database
  • Reliable, balanced and consistent performance
  • Works with RAC
  • Rolling upgrades and patching
  • Provides a reliable cluster filesystem (ACFS)
  • Even more cool features coming in 12c such as Flex ASM


Some useful links:

ASM Administrators Guide 12cR1 (Oracle Docs)

The Mother Of All ASM Scripts (John Hallas)

Technical overview of new features for ASM in 12c (Whitepaper)

SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

Transferring default domains for SQL Developer Data Modeler

Mon, 2016-10-03 10:27

I got a new laptop the other day and installed all the software, including SQL Developer Data Modeler all fine. I then opened a model which had a bunch of tables with columns based off Domains…the columns did not have Domains against them but had been replaced with Logical data types instead.

After some research, the fix, in this case, involved copying the file “defaultdomains.xml” from the following directory on my old laptop, to the same place on the new laptop:

%SQL Developer Home%\sqldeveloper\extensions\oracle.datamodeler\types

After restarting and reopening the model all was back to normal.

What I probably could have done in the first place was to have created my own Domains file for the Design, saved in the Design folder and then when I transferred the Design by copying across the Design folder the domains would have come with it. I could have then just opened the Domain file on the new laptop. I guess it depends on whether I would want these domains to Design specific or part of the defaults.

SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

Oracle Virtual Columns – Can’t use plain column or duplicate expressions

Thu, 2016-09-08 16:38

I had a scenario today where I was loading a table and a particular column is known by multiple names in different source systems and thus to different people. In order to make everyone happy on this occasion, I wondered if I could create a normal column for one of the multiple names and then use virtual columns pointing at the normal column, for the other names.

I’m aware there are several ways of skinning this cat and that virtual columns was probably not the best choice in the first place, but I was just playing with an idea and it didn’t quite end up where I thought…so it was interesting in that respect.

The test code:

     column1    INTEGER NOT NULL 
    ,virtual_column1 INTEGER AS ( column1 ) VIRTUAL NOT NULL 
     column1    INTEGER NOT NULL 
    ,virtual_column1 INTEGER AS ( column1 + 0 ) VIRTUAL NOT NULL 
     column1    INTEGER NOT NULL 
    ,virtual_column1 INTEGER AS ( column1 + 0 ) VIRTUAL NOT NULL 
    ,virtual_column2 INTEGER AS ( column1 + 0 ) VIRTUAL NOT NULL 
     column1    INTEGER NOT NULL 
    ,virtual_column1 INTEGER AS ( column1 + 0 ) VIRTUAL NOT NULL 
    ,virtual_column2 INTEGER AS ( column1 + 1 - 1 ) VIRTUAL NOT NULL 

And the results:

    ,virtual_column1 INTEGER AS ( column1 ) VIRTUAL NOT NULL
ERROR at line 4:
ORA-54016: Invalid column expression was specified

Table created.

Table dropped.

    ,virtual_column2 INTEGER AS ( column1 + 0 ) VIRTUAL NOT NULL
ERROR at line 5:
ORA-54015: Duplicate column expression was specified

Table created.

Table dropped.

I expected it to just work, but I clearly ran in to two problems which scuppered my idea. Firstly, the virtual column can’t simply refer to a normal column without any changes to it, otherwise it fails with ORA-54016. The error isn’t particularly helpful, but eventually I worked out that it was because the column is simply a mapping to a non virtual column. Working around that by adding zero to the numeric column gets it to work, but it’s an ugly hack.

In my scenario there are three different names for this column, depending on the users involved, which then leads on to the next issue, which is that I’d then need two virtual columns pointing at the same source column. Unfortunately if I use the same hack twice, it fails with ORA-54015, because you can’t have two virtual columns with exactly the same expression! A slight variant to the hack and it works, but it’s getting uglier and uglier!

Time to seek out plan B!

SGMB_URL = ""; jQuery(".dropdownWrapper").hide();

Assertions in a future Oracle release

Fri, 2016-06-10 14:34

I just found this link on OTN to vote for including assertions in a future release of the Oracle database.

A great idea – please vote for it.

One of the most important votes this month…well, I do live in England! Smile

“Unstructured Data” – No such thing!

Fri, 2016-06-10 06:58

I keep hearing this term lately and I dislike it.

There is no such thing as Unstructured Data. All data has structure. If it didn’t have structure we wouldn’t be able to use it.

What about free text? Well, that’s just a single column value (stored in a CLOB in Oracle, for example) and the free text is, more often than not, on a row with other columns, such as identifiers and timestamps, i.e. yet more structure.

I think what people mean when they use this “marketing foam”TM term is “data that we have not yet defined the structure for”, but in order to use it at some later stage, the structure will need to be defined – that definition process doesn’t actually give the data structure in and of itself, it simply defines what that structure is, in order to be able to use it.

Interestingly, the Wikipedia article for Unstructured Data calls out the imprecise nature of the term:

The term is imprecise for several reasons:

  1. Structure, while not formally defined, can still be implied.
  2. Data with some form of structure may still be characterized as unstructured if its structure is not helpful for the processing task at hand.
  3. Unstructured information might have some structure (semi-structured) or even be highly structured but in ways that are unanticipated or unannounced.

In other words, it does have structure, but maybe we’ve not written it down, or the structure isn’t helpful to processing or is structured in ways we were not expecting – so what?…it’s still structured!

All of the above seem to me to support the view that all data does indeed have structure.

Temporary Post Used For Theme Detection (4a0f3a36-664d-47d9-93c7-24f2096e004d – 3bfe001a-32de-4114-a6b4-4005b770f6d7)

Fri, 2016-06-10 02:18

This is a temporary post that was not deleted. Please delete this manually. (31074fcb-66af-42a8-a620-ab6d94cd18a0 – 3bfe001a-32de-4114-a6b4-4005b770f6d7)

UNPIVOT Examples

Fri, 2016-05-20 08:47

Remove Linux package with apt

Wed, 2016-05-18 16:40

Resize filesystem

Wed, 2016-05-18 11:16

Oracle 12cR1 setup on Centos 7 LXC

Wed, 2016-05-18 10:57

Why You Should Never Use MongoDB « Sarah Mei

Thu, 2016-05-05 06:44

An interesting article from Sarah – much good advice there!

Source: Why You Should Never Use MongoDB « Sarah Mei


Wed, 2016-05-04 08:17

A colleague asked if there was a way to do column level dependency tracking recently. He wanted to know for a given view, which tables and the columns on those tables, it was dependent upon, without, of course, reading through the code.

I was vaguely aware that since 11gR1 Oracle has been tracking fine grained (column) dependencies, but couldn’t find a way of seeing the details stored, until I found this interesting article from Rob Van Wijk:


I passed the details on to our DBA who implemented it and it seemed to work, for us. Your mileage may vary, of course.

Some comments on Rob’s blog post, bearing in mind, of course, that it was written in 2008 and refers to 11gR1:

  1. D_ATTRS contains values other than “000100000A”. I’ve observed this in a basic 12c install and a production 11gR2 install
  2. D_ATTRS is commented in $ORACLE_HOME/rdbms/admin/dcore.bsq as “/* Finer grain attr. numbers if finer grained */”
  3. D_REASON is commented in $ORACLE_HOME/rdbms/admin/dcore.bsq as “/* Reason mask of attrs causing invalidation */”.  On my basic 12c installation, all rows contain NULL for this value although on a production 11gR2 database I observed a handful of rows with values in this column.
  4. Noted from the comments against Rob’s article is the opportunity to vote for this feature on OTN here


Enable Copy n Paste in SQL*Plus command line on Windows

Wed, 2016-05-04 01:04

My colleague asked me yesterday how to enable copy and paste in the command line SQL*Plus window on Windows 7 – a simple enough task…

On the shortcut that starts the command line version of SQL*Plus, right click and bring up the Properties dialog. Nagivate to the Options tab and make sure the QuickEdit mode is checked on, as below:


Now start SQL*Plus and you’ll find that you can hold the left mouse button down whilst dragging a selection area and then pressing return copies the selected text, whilst pressing the right mouse button pastes the copied text.

If you’d prefer to read this from a Microsoft source, try here, where other methods of setting this up are detailed as well as enabling the Autocomplete facility.

Thanks Oracle, for making my life easier!

Tue, 2016-05-03 06:11

So, I’m really pleased to see that Oracle has acquired Opower. Why? Well, for the last few months I’ve occasionally had to do an extract for Opower….and another for Oracle on the same stuff…..hopefully I’ll only need to do the one extract from now on…yay!