Darwin IT

Subscribe to Darwin IT feed
Darwin-IT professionals do ICT-projects based on a broad range of Oracle products and technologies. We write about our experiences and share our thoughts and tips.Martien van den Akkerhttps://plus.google.com/110503432901891966671noreply@blogger.comBlogger389125
Updated: 15 hours 8 min ago

BPM 12.2.1.3: Exception when deploying BPM project with Human tasks

Mon, 2017-12-18 04:25
This week I deliver a BPM 12c Workshop, that I based on the 12.2.1.3 BPM QuickStart. When the students worked on the lab on Human Workflow, they hit an error deploying the Composite, where in the log you can find something like:

Caused By: oracle.fabric.common.FabricException: Error occurred during deployment of component: RequestHolidayTask to service engine: implementation.workflow, for composite: HolidayRequestProcess: ORABPEL-30257

exception.code:30257
exception.type: ERROR
exception.severity: 2
exception.name: Error while Querying workflow task metadata.
exception.description: Error while Querying workflow task metadata.
exception.fix: Check the underlying exception and the database connection information. If the error persists, contact Oracle Support Services.
: exception.code:30257
exception.type: ERROR
exception.severity: 2
exception.name: Error while Querying workflow task metadata.
exception.description: Error while Querying workflow task metadata.
exception.fix: Check the underlying exception and the database connection information. If the error persists, contact Oracle Support Services.

Caused By: oracle.fabric.common.FabricDeploymentException: ORABPEL-30257


Caused By: java.sql.SQLSyntaxErrorException: Column 'WFTM.PACKAGENAME' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE statement then 'WFTM.PACKAGENAME' is not a column in the target table.

Apparently in the repository of 12.2.1.3 a column is missing in the Workflow Metadata table.

Luckily, I stumbled upon a question in the community.oracle.com forum that hit this 'bug' as well; and provided a solution. You need to do an alter table to resolve this:
ALTER TABLE SOAINFRA.WFTASKMETADATA ADD PACKAGENAME varchar (200);

The smart guy that provided the answer, used a separate Database UI tool. But fortunately, JDeveloper is perfectly capable to provide you de means as well.

First open the Resource Pallette in JDeveloper. Make sure that you have started your Integrated WebLogic already (since that will run the DerbyDB.

Then in the Resource Pallette, create a new Database Connection:


Provide the following details:
Give it a name, like soainfraDB, as a Connection Type select 'Java DB / Apache Derby'. You can leave Username and Password empty. Then as a Driver Class, choose the 'org.apache.derby.jdbc.ClientDriver' (not the default). Then as a Host Name provide localhost, as a JDBC Port enter 1527 en as  a Database Name enter soainfra.

You can  Test Connection  and then, if Successfull, hit OK.

Then from the IDE Connections pallette  right click your newly created database connection, and choose 'Open in Databases Window':



And from that right click on the database connection and choose 'Open SQL Worksheet':

There you can enter and execute the alter statement:
After this, deployment should succeed. Since it is persisted in the DerbyDB it will survive restarts.

This might apply to the SOA QuickStart as well (did not try).

OSB: Disable Chunked Streaming Mode recommendation

Fri, 2017-12-01 05:37
Intro These weeks I got involved in a document generation performance issue. This ran for several months, maybe years even. But it stayed quite unclear what the actual issue was.

Often we got complaints that document generation from the front-end application (based on Siebel) was taking very long. End users often hit the button several times, but with no luck. Asking further, it did not mean that there appeared a document in the content management system (Oracle UCM/WCC). So, we concluded that it wasn't so much a performance issue, but an exception along the process of document generation. Since we upgraded BI Publisher to 12c, it was figured that it might got something to do with that. But we did not find any problems with BI Publisher, itself. Also, there was an issue with Siebel it's self, but that's also out of the scope of this article.
The investigationFirst, on OSB the retry interval of the particular Business Service was decreased from 60 seconds to 10. And the performance increased. Since the retry interval was shorter, OSB does a retry on shorter notice. But of course this did not solve the problem.

As Service developers we often are quite laconical about retries. We make up some settings. Quite default is an interval of 30 seconds and a retry count of 3. But, we should actually think about this and figure out what the possible failures could be and what a sensible retry setting would be. For instance: is it likely that the remote system is out of order? What are the SLA's for hoisting it back up again? If the system startup is 10 minutes, then a retry count of 3 and interval of 30 seconds is not making sense. The retries are done long before the system's up again. But of course, in our case sensible settings for system outage would cause delays being too long. We apparently needed to cater for network issues.

Last week our sysadmins encountered network failures, so they changed the LoadBalancer of BIP Publisher, to get chunks/packets of one requests routed to the same BI Publisher node. I found SocketReadTimeOuts in the logfiles. And from the Siebel database a query was done and plotted out in Excel showing lots of request in the  1-15 seconds range, but also some plots in ranges around 40 seconds and 80 seconds. We wondered why these were.

The Connection and Read TimeOut settings on the Business Service were set to 30s. So I figured the 40 and 80 seconds range could have something to do with a retry interval of 10s added to a time out of 30 seconds.

I soon found out that in OSB on the Business Service, the Chunked Streaming Mode  was enabled. This is a setting we struggled with a lot. Several issues we encountered were blamed on this one. As a Helpdesk employee would ask you if you have restarted your system, on OSB questions I would ask you about this setting first... Actually, I did for this case, long before I got actively involved.
Chunked Streaming Mode explainedLet's start with a diagram:

In this diagram you'll see that the OSB is fronted by a Load Balancer. But since 12c the Oracle HTTP Server is added to the Weblogic Infrastructure. And following the Enterprise Deployment Guide we added an OHS to the Weblogic Infrastructure Domain, as a co-located OHS Instance. And since the OSB as well as the Service Provider (in our case BI Publisher) are clustered, the OHS will load balance the requests.

Now, the Chunked transfer encoding is an HTTP 1.1 specification. It is an improvement that allows clients to process the data in chunks right after the chunk is read. But in most (of our) cases a chunk on itself is meaning-less, since a SOAP Request/XML Document need to be parsed as a whole.
The Load Balancer also process the chunks as separate entities. So,by default, it will route the first one to the first endpoint, and the other one to the next. And thus each SP Managed Server gets an incomplete message and there for a so-called Bad Request. This happens with big requests, where for instance a report is requested together with the complete content. Then chances are that the request is split up in chunks.

But although the SysAdmins adapted the SP Load Balancer, and although I was involved in the BIPublisher 12c setup, even I forgot about the BIP12c OHS! And even when the LoadBalancer tries to keep the chunks together, then again the OHS will mess with them. Actually, if the LoadBalancer did not keep them together, the OHS instances could reroute them again to the correct end-node.
The SolutionSo for all those Service Bus developers amongst you, I'd like you to memorize two concepts: "Chunked Streaming Mode" and "disable", and the latter in combination with the first, of course.
In short: remember to set Chunked Streaming Mode to disable in every SOAP/http based Business Service. Especially with services that send potentially large requests, for instance document check-in services on Content/Document Management Systems.
The proof of the puddingAfter some discussion and not being able to test it on the Acceptance Test environment, due to rebuilds, we decided to change this in production (I would/should not recommend that, at least not right away).

And this was the result:


This picture shows that the first half of the day, plenty requests were retried at least once, and several even twice. Notice the request durations around the 40 seconds (30 seconds read timeout + 10 seconds retry interval) and 80 seconds. But since 12:45, when we disabled the Chunked Streaming Mode we don't see any timeout exceptions any more. I hope the end users are happy now.

Or how a simple setting can throw a spanner in the works. And how difficult it is to get such a simple change into production. Personally I think it's a pity that the Chunked Streaming Mode is enabled by default, since in most cases it causes problems, while in rare cases it might provide some performance improvements. I think you should rationalize the enablement of it, in stead of actively needing to disable it.

BPM BAC Subversion Server refusing connections Revised

Fri, 2017-11-24 05:34
A little backgroundIn April I wrote about our BPM Server installation, that is actually a single host cluster on dev and test. But installed like it was a dual-node  server, so we had a cloned domain.

A BPM installation has a component that is called the Process Asset Manager. Under the hood it uses a replicated SubVersion server. Each node has one, so they had to synchronize. But since we're on the same host, we needed to differentiate in port numbers. Although we used virtual host names for each server node. In an article in april 'BPM BAC Subversion Server refusing connections' I wrote about that.

It turns out: it did not work appropriately. After some investigation, apparently the bac_node1/subversion server calls the subversion server on bac_node2, with it's own address, but with the port of the other:
...
<Jul 24, 2017, 2:37:29,543 PM CEST> <Debug> <oracle.bpm.bac.svnserver.replication> <darlin01> <bpm_server1> <Active Sync Thread [/54680efc-9328-478e-953c-834bbb250725/]> <<anonymous>> <> <20a54a15-d8d8-4e58-a4f6-11a90e992967-00000008> <1500899849543> <[severity-value: 128] [rid: 0:490:13:19] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <[oracle.bpm.log.Logger:debug] About to synchronize against node Member(Id=2, Timestamp=2017-07-12 08:01:13.309, Address=10.0.0.38:37055, MachineId=9002, Location=site:tst.darwin-it.local,machine:bpm_machine1,process:24566,member:bpm_server2, Role=bpm_cluster), url svn://syncuser@t-bpm-1-bpm-1-vhn.tst.darwin-it.local:8424/54680efc-9328-478e-953c-834bbb250725>
...

Of course there's no listen address so unsurprisingly we get errors like:
...
[2017-05-30T12:55:51.704+02:00] [bpm_server1] [ERROR] [] [oracle.bpm.bac.svnserver.replication] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: ] [ecid: ed0bc6ce-fefb-4608-8dbe-46f7206a1573-0000000a,0:527] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server[[
oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException.wrap(RepositoryException.java:56)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:98)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:74)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:59)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.runImpl(ActiveAARepository.java:341)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.run(ActiveAARepository.java:304)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:280)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:95)
... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:112)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:68)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 10 more

]]

(to get this page searchable).
SolutionBut now, since this week, Oracle Support presents us.... (drum rolls)  Patch 26775572 for version 12.2.1.2.0, the official fix for this issue, is finally ready and available for download.
I hope to be able to install it next monday. But feel free to try with me: 'Patch 26775572: BAC node using the wrong port when attempts synchronization with Virtual IP '

By the way, for about 2 months we have diagnostic patch running that solves this. But named patch is the official delivery of our diagnostic variant.

By the way, since 12.2.1.3.0 is out in the field for quite some time already, I expect that this did not landed in that version yet. If you're running in the same issue for 12.2.1.3.0 then create a SR to request for a port of this patch.


SOASuite 12c: keep running instances using ANT

Wed, 2017-11-15 08:41
At my current customer I implemented a poor man's devops solution for Release and Deploy. It was based on a framework created as bunch of Ant projects, that I created years ago. It was based on scripts from Edwin Biemond. See for instance here, here and here. I never wrote about my solution, because although I refactored them quite intensively, the basics were already described thoroughly by him.

What I did was that I modularized the lot, split the environment property files, added logging, added OSB 12c  support, based on the config jar tool, etc.

One thing I ran into this week was that at first deployment from our team to the test environment using my framework, the running instances for the BPM projects were aborted.

Now, if you take a look at the deploy.sarLocation target in Edwin's article about deploying soa suite composites  you'll find that he also supported the overwrite and forceDefault properties.

When re-deploying a composite from JDeveloper you're probably familiar with the 'keep running instances' checkbox. I was looking for the ANT alternative in the ${oracle.home}/bin/ant-sca-deploy.xml scripting. First I looked in the 12c docs (see 47.9.4 How to Use ant to Deploy a SOA Composite Application), but it is not documented there.

But when I opened the particular ant-sca-deploy.xml script I read:
 <condition property="overwrite" value="false">
<not>
<isset property="overwrite"/>
</not>
</condition>
<condition property="forceDefault" value="true">
<not>
<isset property="forceDefault"/>
</not>
</condition>
<condition property="keepInstancesOnRedeploy" value="false">
<not>
<isset property="keepInstancesOnRedeploy"/>
</not>
</condition>
...
<target name="deploy">
<input message="Please enter server URL:" addproperty="serverURL"/>
<input message="Please enter sar location:" addproperty="sarLocation"/>
<input message="Please enter username:" addproperty="user"/>
<input message="Please enter password:" addproperty="password">
<handler classname="org.apache.tools.ant.input.SecureInputHandler" />
</input>
<deployComposite serverUrl="${serverURL}" sarLocation="${sarLocation}" realm="${realm}" user="${user}" password="${password}"
overwrite="${overwrite}" forceDefault="${forceDefault}" keepInstancesOnRedeploy="${keepInstancesOnRedeploy}"
regenerateRuleBase="${regenerateRuleBase}" configPlan="${configplan}" scope="${scope}"
sysPropFile="${sysPropFile}" failOnError="${failOnError}" timeout="${timeout}" folder="${folder}"/>
</target>

So, the script kindly supports the keepInstancesOnRedeploy property. And thus I implemented a deploy.keepInstancesOnRedeploy property the same way as the deploy.forceDefault/deploy.overwrite properties.

This probably is usefull for Maven based (re-)deployments.

Enable WebService test client on SOA/BPM production mode environments

Thu, 2017-11-02 06:13
At my current assignment I need to trouble shoot the identity service because of a BPM->OID coupling. I use the support document 1327140.1 for it, that suggest to test http://<soa-server>:<port>/integration/services/IdentityService/identity

Doing so in a production mode soa or bpm environment, you'll soon find out that it uses the WebService test client via uri /ws_utc, and that this does not work. Resulting in http-404 Not found errors.

First I found a blog of Maarten of Amis mentioning this as well. But unfortunately, he could not get around it either. But luckily I found note 1915317.1, that tells me that the WebServices test Client is not enabled by default.

You can enable it on your domain via the EM:

And then expand the Advanced node:


Check the 'Enable Web Service Test Page' check box.

Since it is about a production mode environment, you need to 'lock & edit'. The note suggests to do that in the /console and then do the change in /em. And back in /console do the activate. I found that peculiar, since you can do it in the change center in /em as well.

You need to restart the servers (apparently including the AdminServer) to get this in effect.

So now lunch and check if my restart worked.

Implementing the KeyStore Service with Fusion MiddleWare 12c

Fri, 2017-09-08 09:31
For the passphrases, use the passphrases used earlier.Thinking about TLS (Transport Layer Security, the succesor of Secure Socket Layer, SSL) and WebLogic and Oracle HTTP Server, allways gave me Cold Water Fear. You have to create keystores with keys, wallets, certificate signing requests, import signed and trusted certificate chains. Not to mention the configuration of WebLogic and OHS.

Now, creating keystores with the Java Keytool turns out not that hard. And generating the Certificate signing requests and importing the certificates are also a walk in the park, nowadays. The internet world is full of example so I'm not going to do that here.

But lately, our Service Bus developers found that they needed to replace the configured demo identity and trust key stores with Custom Stores. But this broke the connection between the AdminServer and the Nodemanagers, resulting in TLS/SSL Handshake errors. By default, the nodemanagers work with the demo-identities when running in TLS.

This drove us to work out an infrastructural configuration of TLSin our FMW environments, in a  way that the SB developers can extend that with their certificates.

In this article I want to describe how to configure TLSin Weblogic using the KeyStore Service, and also how to reconfigure the nodemanagers to have them running TLS using the custom stores.

Keystores and the KeyStore Service (KSS)When implementing in TLS in Fusion MiddleWare 12c you have the choice to use the new KeyStore Service for creating keys and certificates directly in the KSS, or the Java Key Tool.

Until now WebLogic preferred a Java Key Store (JKS) for storing certificates. This is a file that functions as a vault to store your keys and certificates savely. You can use the commandline tool Keytool that is delivered with the JDK. But there are several graphical tools, like Portecle, that can make your encrypting life even more simple.

Since a keystore is a atomic file, you need to copy it or put it on a share to have a complete multi-node clustered domain use the same store.

Per Fusion MiddleWare 12c Oracle introduced the KeyStoreService. The KSS is part of the Oracle Platform Security Services, and stores the keys in so-called Stripes in the MDS. For some components it is necessary to sync them to a keystores.xml file in the domain. But we'll get to that.
The KSS enables you to create, delete and manage keystores from the Enterprise Manager/FustionMiddleWare Control.

As said, you can start with creating your keys using the KSS. We however did choose not to. We started with creating a JKS using the KeyTool. The reaons for that were:
  • Using the keytool, you can already create a store and a Certificate Signing Request (CSR) before configuring your domain.
  • We wanted to make sure we had the configuration of Weblogic and Nodemanager correct, before transition to KSS. 
  • With a JKS you have a backup of your identity in case you mess up your configuration. Keep in mind that you need to import the CSR into the keystore from where you generated the key. Even a new JKS created the exact sameway won't work.
  • Using OraPKI, you can migrate your certificates from a JKS to a wallet. We are still in the 'figure-out-phase' of implementing OHS with KSS.
  • Oh, and we wanted to have scripts and easy to document commands for creating and importing the stores.
Identity and Trust StoreSo we started with creating  an Identity Keystore with the Java Keytool, created a CSR and had it signed.
We had several virtual hostnames refering to the hostname (darlin001.org.darwin-it.local) loadbalancer (o-dwn-1.ont.org.darwin-it.local), managed servers (o-dwn-1-dwn-1-vhn.ont.org.darwin-it.local), Admin Virtual host (o-dwn-1-admin-vhn.ont.org.darwin-it.loca), etc. To have them accepted with the same certificate, you need to create the Certificate Signing Request with an extention called 'Subject Alternative Names'. This can be done with the -ext parameter including the SAN keyword as follows:
... -ext SAN=dns:darlin001.org.darwin-it.local,dns:o-dwn-1-admin-vhn.ont.org.darwin-it.local,dns:o-dwn-1-dwn-1-vhn.ont.org.darwin-it.local,dns:o-dwn-1-dwn-2-vhn.ont.org.darwin-it.local,dns:o-dwn-1-ohs-1-vhn.ont.org.darwin-it.local,dns:o-dwn-1-internal.ont.org.darwin-it.local,dns:o-dwn-1-admin.ont.org.darwin-it.local,dns:o-dwn-1.ont.org.darwin-it.local
This is a comma seperated list, with each addres need to be prefixed with dns:. See also the keytool documentation.

Don't forget to add the Common Name (CN) to the SAN, since clients are supposed to ignore the CN, when using a SAN. Also you need to add the -ext SAN parameter to the CSR. You can use it when creating the Key, but that's not enough for the CSR.

For the Trust Store, I copied the cacerts from the JDK. I changed the password and imported the CA and root certificates. But you need to consider if you want to accept all the JDK's trusted certificates. Or only those you really need to trust.

 Scope of the KeystoreWe choose to have the domain the scope of the keystore, not the host. So as a CommonName I choose the loadbalancer address. The thing is that we want to have the managed servers use the same keystore. And with a whole server migration, a managed server could migrate to a new host, that is not yet in the certificate. Although we add the known hosts, we could be forced to use a new host. And with whole server migrations, you can't way for your CSR to be signed.
As name of the keystores and the identity alias I choose a reference to our domain name. As an environment shortage I choose 'dwn', that could also refer to the type of environment. Like BPM, OSB, SOA, etc. The first letter ('o') denotes the development environment (Dutch: 'ontwikkeling'). And the digit denotes the environment number, where you could have multiple development or test environment.
Import KeyStores into KSSHaving created your KeyStores and made sure Weblogic works with them, you can import them into the KSS. The UI (EM) does allow you to create a keystore, and import certificates. It does not allow you to import a complete JKS. But you can do so using WLST. This caters also for my quest to script as much as possible.

The commands are executed in WLST Online mode, connected to the AdminServer. So start wlst and connect to the AdminServer.

In the following command commands, I use the following environment variables:
  • $JAVA_HOME: refering to, yes, you're right....
  • $JKS_LOC=/u01/oracle/config/keystore
  • $JKS_NAME=o-dwn-1.jks
  • $JKS_TRUST_NAME=o-dwn-1-trust.jks
Connected to your AdminServer you need to get a handle on the 'KeyStoreService object', that has methods to do the imports,  etc.:
wls:/o-dwn-1_domain/serverConfig/> svc = getOpssService(name='KeyStoreService')

With the handle you can import the Identity keystore:
wls:/o-dwn-1_domain/domainRuntime/> svc.importKeyStore(appStripe='system', name='o-dwn-1-id', password='<password>', aliases='o-dwn-1', keypasswords='<password>', type='JKS', permission=false, filepath='/u01/oracle/config/keystore/o-dwn-1.jks')
Already in Domain Runtime Tree

Keystore imported. Check the logs if any entry was skipped.

The parameters aliases and keypasswords contain a comma-separated list of key-aliases and corresponding key-passphrases.
My advise would be to use the same password for your key as for your keystore. Many products (like Oracle B2B) allow you to provide only one password, that is used for both.

Importing the Trust Store, works similar:
svc.importKeyStore(appStripe='system', name='o-dwn-1-trust', password='<password>', aliases='ca_dwn_org,our-root-ca', keypasswords='none,none', type='JKS', permission=false, filepath='/u01/oracle/config/keystore/o-dwn-1-trust.jks')
Already in Domain Runtime Tree

Keystore imported. Check the logs if any entry was skipped.

One thing with Trusts: they don't contain Keys. So no keyprases are applicable. But the keypasswords property is mandatory and also need to contain the same number of passphrases as there are aliases named to be imported in the KSS. Apparently it is sufficient to use 'none' as a passphrase.

A wallet can also be imported the same way. For a type then choose 'OracleWallet', and in filepath a reference to the wallet folder.

To check the content of the keystore, we can list it:
wls:/o-dwn-1_domain/domainRuntime/> svc.listKeyStoreAliases(appStripe="system",name="o-dwn-1-id",password='<password>',type="*")
Already in Domain Runtime Tree

o-dwn-1

And for the trust:
wls:/o-dwn-1_domain/domainRuntime/> svc.listKeyStoreAliases(appStripe="system",name="o-dwn-1-trust",password='<password>',type="*")
Already in Domain Runtime Tree

ca_dwn_org
our-root-ca

Now the keystores are in the KSS. You can see them in EM. But some FMW components like nodemanagers, (but it appeared to me even Weblogic Servers), don't work right away. They can't look into the KSS, but use a keystores.xml file in the ${DOMAIN_HOME}/config/fmwconfig. This turns out to be an export of the keystores. To synchronize the KSS with that file you need to do:
wls:/o-dwn-1_domain/domainRuntime/> syncKeyStores(appStripe='system',keystoreFormat='KSS')
Already in Domain Runtime Tree

Keystore sync successful.
List the content of KSS with FMW ControlTo list the KSS within FMW Control, logon and navigate to Weblogic Domain -> Security -> Keystore:
This brings you to the content of the KSS with the particular Stripes. For Weblogic we used the system stripe to import our custom keystores:

Select a store, and click the manage button in the bar. It will ask for the keystore password (the one you used creating the JKS with the keytool). Then it shows:


Here you can import, export certificates, create keypairs and generate a CSR for it.

Reconfigure WeblogicNow we can proceed in configuring Weblogic and the nodemanagers.

For the weblogic servers, logon to the Admin console, navigate to the particular Server, tab Keystores and select 'Custom Identity Keystore and Custom Trust keystore' in the pop-list.

Then enter the following values:
Attribute
Value
Custom Identity Keystore kss://system/o-dwn-1-id Custom Identity Keystore Type KSS Custom Trust Keystore kss://system/o-dwn-1-trust Custom Trust Keystore Type KSS
For the passphrases, use the passphrases used earlier.
Then on the SSL tab you can provide the Identity alias and the key-passphrase.

Restart the Server (or at least Restart the SSL, under the control tab).

Reconfigure Nodemanagers for KSSFor the nodemanager, navigate to the nodemanager.properties file in ${DOMAIN_HOME}/nodemanager.
At the top of the file add the following properties:
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=kss://system/o-dwn-1-id
CustomIdentityKeyStoreType=KSS
CustomIdentityKeyStorePassPhrase=
CustomIdentityAlias=o-dwn-1
CustomIdentityPrivateKeyPassPhrase=

Restart the nodemanager and check if it succesfully loads the kss://system/o-dwn-1-id as a keystore and starts the listener in SSL mode.

Then in the AdminServer console, under Domain Structure -> Machines -> <Machine_Name> check if the NodeManager is 'Reachable'.

ConclusionThat's about it. Don't forget to Sync the KSS in wlst, when the nodemanager does not restart properly and the managed/admin server could not start SSL properly.

 I hope I'll soon be able to write down the directions to reconfigure SSL in OHS to use the KSS. But as said we're still in the 'figure-out-phase'.

PCS and Correlations: the next big thing cavemen already used...

Wed, 2017-07-26 07:21
You can use BPM, BPEL or Workflow to orchestrate or direct regular processes to get a job done from the beginning through a certain flow with a few decision-points, alternate and parallel flows to the end. A common use that is fine and usefull for most projects. And it can be seen as the driver for software companies to develop process/workflow engines.

However, there are cases that one process spawns off multiple other process instances that are some how related to one particular business case, involved person, or a uniquely distinguishable entity. And often those processes have to interact, with each other. For instance, this year in Split I came up with an idea for a role playing advanture game using chatbots and PCS. Each player would initiate a PCS instance, that when interacting with each other can detect if players meet each other at a certain location.
Correlation Sets are key here...Right after the acquisition of Collaxa in 2004, in my Oracle Consulting era, I got the chance to be a BPEL trainer for Oracle University, doing several trainings for a few bigger consulting companies in the Netherlands. One of the subjects was about Asynchronous Processes and how the Process Manager used WS-Addressing to correlate the response to the running instance that sent out the request. But together with that Correlation Sets were mentioned. And I did not understand it: why would you need Correlation Sets when the Process Manager handles it all transparently using WS-Addressing? Otherwise put: it was time for me to do some projects.

PCS, BPM Suite and SOA Suite share the same process engine that originated from the early BPEL Process Engine. And as you can detect from my anecdote: Correlation Sets are key in this story. And this functionality is around from the medieval ages. In fact, recently they discovered char-coal drawings in a cave in France, indicating that people form pre-historic times already used Correlation Sets in their BPEL's...

Prototype this...Let's say you have a customer that is a large company with several responsible participants that are allowed to sign the contract. Some of them are full-authorised signers, while others only have partial authorisation. So either one of the fully authorised participants signs, which would complete the contract, or some of the partial signers have signed that pass the contract over the signing threshold.

For this case we keep it simple: either one of the full authorised participant should have signed or all of the partial signers should have signed. But we'll handle this in a (set of) business rule(s), so it can be made more complex to resemble real-world cases.

The singing process spawns off single signing processes for each participant asynchronously:
This is a quite simple looping, as explained in my previous PCS-Article. This can be made simpler using a multi-instance embedded sub-process:

However, for us, this was a later addition.

This single sign process results in a task for the particular signer. When signed it will respond back to the main signing process, that receives the responses of the actual signers:
For every respondent it checks if it results in the contract being fully signed. If it isn't it will wait for a subsequent sign-response. For the technical interested: since this Receive is coupled to the end activity of the signle-sign process:
it is using the earlier mentioned WS-Addressing to correlate each responding asynchronous sub-process to this particular instance. When one or more signers respond just at the moment the process is busy with determining if it has to wait for another receive, these responses are kept by the process manager and delivered at the moment the process instance activates the receive activity again.

Now the thing is, when the contract is fully signed, there still can be several single sign process active. For each of those a task resides in the task-list of the particular participant, but that does not add to the signing-validity of the contract anymore. Those tasks need to be withdrawn:
In this loop, all the signers are checked and for those that did not sign yet, a Withdraw Signing Event is thrown. Correlated to the single signing instance will cause that process to end, removing the particular task from the participants list.
Let's get into the details...The list of signers is defined as:



We need to know the signers for a contract.  For each Signer we have some attributes. Two toggles are important here:
  • signed: did the signer sign the contract?
  • fullSignAuthority: is the signer fully authorised to sign the contract? 
I also declared a separate object to contain a list of signers. I don't like list based attributes among single attributes. I want to make it explicit that an attribute contains a list of elements.

The main sign process is initiated with a BusinessCustomerSignEvent:
It contains a reference to the Chamber of Commerce number and the account, opportunity and quote id's in SalesCloud or Configure Price Quote Cloud Services. But also the list of Signers component.

When initiating the Single Sign Process, the BusinessCustomerSingleSignEvent is used:
As shown, its the same as the BusinessCustomerSignEvent exept for the signer: it's a single element here. But to be able to identify the particular signer, we introduced a signerIndex.

Then, lastly, there is the WithdrawSigner event:
Here the signer is not particularly important. The correlating attributes are:
  • accountPartyId
  • opportunityId
  • signerIndex
These correlating attributes are defined by the accountPartyId identifying the customer. For that customer, multiple opportunities can arrise. So actually the contract relates to a particular opportunity. And each signer is identified by a sequence/index.
Implementing the details...First the Single Sign Process is modelled like:


Referring to the Single Sign Process, defining the correlation starts with defining the correlation key:

This pane is called using the 'shuffle'-icon in the button bar, next to the play button.

The Correlation Key is used to identify a Correlation Set, and can consist of one or more properties. Those properties are then mapped to the correlating activities. You define a key using the plus icon. Then you define one or more properties. Then you assign those to the key, using the shuttle icons in the middle.

The  process starts with the Single Sign event, with the interface defined based on the BusinessCustomerSignEvent :

 But the interesting part are the correlations:

In the properties of the Start activity, click on the Correlations tab.
Since it it's a Start event, the only possible choice we have is to initialize a correlation set.
You need to add a correlation key, the one you defined before.

Then map the properties of the key to attributes of the events arguments.

When the Single Sign proces is initiated, the flow splits in a parallel flow. Both conditional paths are based on the condition that the signed attribute is false. If the signed attribute is true, however unlikely, the process is ended.

As can be seen the flow separates into branches leading to a Sign Contract activity, and a Catch  Withdraw Signer Event  activity. So, both are waiting. The Catch event is based on the WithdrawSigner event and is also based on the correlation key:

Here we have the option to Initialize a correlation key. But, in this case this is not quite sensible. We do want to correlate to the earlier initialized Correlation Set. And here we map the properties to the attributes of the WithdrawSigner event activity argument.

Sharp eyes would notice that the Sign Contract Activity has a boundary event on it:
It also catches a WithdrawSigner event, with a comparable correlation set. Actually, this boundary event could have sufficed for our case. But then the process would be boringly simple... And I wanted to show also a construct that allows for intra-process interactions. Here events are thrown and catched between the branches within an instance.
Flowing through the Single Sign process
So, what happens is that either a signer signs the contract, or the main process throws a WithdrawSigner event.

If the Signer signs, the WithdrawSigner Catch event needs to be released. So, in the main flow after the Sign Contract activity, a throw WithdrawSigner event is done:
 It targets the Withdraw Signer event, from the same process. The flow will look like:


This releases the Catch event in the lower flow-branch.
When this signer was a full signer, the main process flows on to do a withdraw of the not signed signer-tasks:
It would be nice to drill down into the decision functions. They're partly used as Script Tasks. And partly to determine the looping. The Check Signing Status checks if one of the fully authorised signers signed, or if there aren't any partial signers that haven't signed yet.

Maybe the most interesting one is the Determine Signers to Withdraw Tasks for:

If the signer has signed, then the signer is removed from the list. This adds to the list functions mentioned in my previous article. The resulting list is traversed to throw a withdraw event for each of the tasks.

If the main process throws a Withdraw Signing Event for an instance, and the signer has not signed then (which is probably the case), it should throw an event to the cancel the Sign Contract boundary event:
What results in a flow like:


And this will remove the tasks from the lists of all the participants.

ConclusionThis was one of the nicest constructs I made in PCS so far. In the "BPEL pre-history" I found that the Correlation Set concept was a very differentiating feature in the Process Engine market. I could not solve some of the more complex problems in the past without Correlation Sets. Maybe this experience made me very fond of this feature. And it's nice to be able to demonstrate it in PCS.

As said, the Boundary event on the Sign Contract activity would already suffice. It turned out I did not need the complex parallel flow with the extra branch to wait for the Withdraw Signer event. But wouldn't you agree this process looks more interesting? It at least also demonstrates how to release blocking paralallel branches within a process.

Process Cloud Service and how to loop and select elements from a list

Fri, 2017-07-21 10:15
For more than half a year I've been 'dying' to write posts about some of the constructs I've developed in my last Process Cloud Service project. At last I have the opportunity. And I hope I'll be able to write some more. But for starters, one of the problems I encountered is that I needed to process a list of something(s) and select elements from it. Or even better, how to build up a new list based on an input-list and some rules. Oh and do looping, or actually determine how to finish a loop based on a list of elements. Without a count() function in the PCS Expressions...
QuestionsIf you have met these kinds of problems and the tool at hand is PCS, then you probably ran in (some of) the following questions:
  • Why don't we try to solve this in SOA CS or possibly ICS?
  • Where are the script tasks we have in BPM Suite? (Sorry, this is an obvious one, but still)
  • How to count the elements of a list? Or, where are the functions in PCS?
  • How to add elements to a list?
  • Etc.
To address a few of those...

Many of these things we ran into are actually orchestration issues. And as with the all-time discussions on when to use SOA and when BPM, we advise doing complex service orchestration  (with message processing) in SOA CS, or if possible in ICS. But when we started with this project, the tools we were given were PCS and ICS. And where ICS lacked the more advanced logic processing in the orchestration integrations, at the time (it's improved over time). And sometimes it really is fun to try to accomplish things that were mentioned not being possible. Go where no man has gone before, that sort of stuff.

Script tasks? I guess PCS Product Management gets tired of answering this question. But the real thing is: we do need to do determinations based on the outcome of services. But also doing logic before doing an activity. In the input data association of an activity, you can only assign into the input-arguments of the activity. You can't update process variables. You can do that in the output data-association. But not all activities have a output-data-associations. And there are cases where you don't have an applicable previous activity. For instance in loops.
The caseLet's get into a case. Let's say you have a list of product-subscriptions that your customer has acquired from you. And for some reason some of those products need a review. In our case the customer faces a takeover, and is part of a hierarchy. So you want to fetch all the applicable products in the company hierarchy and add all of those in a list of Review Product Tasks.

The list of products is acquired by calling an ICS service that fetches a list of products from SalesCloudService and calls Oracle Policy Automation to filter the applicable products from the list. That's the list we need to add to the list that should contain the Review Product Tasks for products acquired from all the companies in the hierarchy. The result list looks like:
The cocNumber is the Chamber of Commerce number of the company. For the rest we keep things simple.

The part of interest of the process is:
In the basis, its not that hard. Except for:
  • I don't have an other activity with an output data-association in the loop to do calculations to base my test upon.
  • Again, we don't have script tasks with functions. So I miss-use a decision function as a script task. Which does the job quite well by the way.
  • Oh, and the nice thing of the Business Rules Engine (the same as used in SOASuite and BPM Suite) is that it does have functions. But we'll get into the magic of that in a minute.
The first activity is the service call to the ICS service.  In the output we copy the response message to a DO and it initializes the productIdx to 1. It's a nice example of the fact that we can do (simple) calculations and Data Object manipulations in the output data-association. Those aren't confined to the arguments of the activity.
Within the response message we have a list of products, but of course, those are in the structure of the service definition. We need to copy the elements over to the input of the decision function.

The input of the Add Review Product Task decision function looks like:

The output is:

So in goes the list of products and the product to add. The Decision Functions results into an updated list.
But, how to do that?
First, let's look in to the input data association:

The first rule is to copy the reviewProductTasksDO (the list we're building) to the corresponding input argument. Then the particular product is created by entering the different elements. As said, some where in the structure of the response message of the ICS service, we have a list of products. Using the productIdx variable we select one if the products and that indexed product is copied element by element.

Now, the rule is:

The first 2 conditions are actually declarations of identifiers we refer to in the rest of the rule: reviewProductTask, being the single object as input  and reviewProductTaskList containing the list of reviewProductTasks. Then we only want to add the reviewProductTask if it does not exist, to prevent the rule getting into a infinite loop. Since reviewProductTaskList is in fact an object that has an attribute being a list of reviewProductTask elements, that list contains a set of functions. One of those is the contains() function:
reviewProductTaskList.reviewProductTask.contains(reviewProductTask)

The goal of this function is quite obvious, it returns true() or false(). If the contains() function returns true, then a a call action is performed. And there the magic happens:
reviewProductTaskList.reviewProductTask.append

The reviewProductTaskList.reviewProductTask list also has an append() function. The input of that is the reviewProductTask object variable, the same that went into contains() as a parameter. The real nice thing? This also works in BPM Suite and SOA Suite! Remember? The Business Rules Engine in those products is the exact same thing! To me this was a bit of a revelation. I wasn't aware of the list functions in BRE.

So, now the rule is fired and the list is extended. We need to assign the result to the particular DataObject. So the output data association is:
Quite obvious, again, but the output argument, being the reviewProductTasksOut object, needs to be assigned to the reviewProductTasksDO Data Object. But also the productIdx needs to be increased.

The Decision Function activity loops back to the Exclusive Gateway. How to determine if we're done? Remember we don't have a count() function in the expression builder? The trick is done in the No-transition:





The condition is:
getCriticalProductsResponseMessage4TakeOverCpyDO.data.products.product[productIdx] == null 
When the productIdx is increased and it gets a value greater then the number of product elements in the list, then indexing that (non-existent) product will lead to null. And that can be tested.

And again, it's too obvious, and yet it took me a while to get to this.

The check if we need to get another company in the hierarchy is comparable, only it is indicated by the ICS service. If it is the top-level company, then it has no reference to a parent:
getCriticalProductsResponseMessage4TakeOverCpyDO.data.parentAccountPartyId != null

ConclusionIt might seem so obvious when you read this. And I suppose it is. But, some of those constructs and tests/conditions are not so apparent from the UI. So I hope I have been able to get some of you at the right track.

In one of the follow-up blog-posts I'd like to address Correlations.

Weblogic log level mapping

Tue, 2017-07-04 07:21
For quite some time now I wondered about the differences in log-levels. For instance, if you configure the log levels in the classes in SB12c or SOA Suite,  You see levels as INCIDENT , ERROR, and TRACE, even with several sub leverls (1, 16, 32). But on the Server Log configuration in Weblogic, you see levels a s ERROR, NOTICE and DEBUG, or TRACE.  And then in Java, JDK logging we even have other log levels.

Today I experienced the urge to look it up and fout this nice document. It describes the configuration of logging in Fusion Middleware. It contains this nice table:


ODL
WebLogic Server
Java
OFF OFF 2147483647 - OFF INCIDENT_ERROR:1 (EMERGENCY) 1100 INCIDENT_ERROR:4 EMERGENCY 1090 INCIDENT_ERROR:14 ALERT 1060 INCIDENT_ERROR:24 CRITICAL 1030 ERROR:1 (ERROR) 1000 - SEVERE ERROR:7 ERROR 980 WARNING:1 WARNING 900 - WARNING WARNING:7 NOTICE 880 NOTIFICATION:1 INFO 800 – INFO NOTIFICATION:16 (DEBUG) 700 - CONFIG TRACE:1 (DEBUG) 500 – FINE TRACE:1 DEBUG 495 TRACE:16 (TRACE) 400 - FINER TRACE:32 (TRACE) 300 - FINEST TRACE:32 TRACE 295
This helps quite neatly. Especially, because to have ODL logging get through to the Server log-files, you need to set the appropriate level on the Weblogic server.

OHS URL Rewrite

Fri, 2017-06-30 05:23
First half of this year I did two SAML2 implementations on Weblogic. One of those was to implement Single Sign On against ADFS for Apex applications.

In short, we installed an adapted version of ORDS on a Weblogic server and configured SAML2 for Service Provider initiated SSO, as can be read here.

We added an Oracle HTTP Server as reversed proxy to the story. For the other customer I found out how to create a separate saml2 routing, since we wanted a SP specific URI, but Weblogic only listen to the /saml2 URI. I solved that using the PathTrim and PathPrepend options in OHS, as can be read here.

But one thing that left was a 'Nice' URI. When we now want to get to the application we need to have an url like https://host.customer.nl/ords/f?p=ApexId  But we wanted to use something like https://apexappname.customer.nl. I thought to be smart and just do a PathPrepend of ords/f?p=ApexId. But this is considered to be a folder, so OHS/Apache adds a trailing slash '/', resulting in an url like: https://host.customer.nl/ords/f?p=ApexId/  .This routes to Apex nicely, but Apex considers the slash as part of the ApexId,  so you'll get a message indicating that Apex cannot find the 'ApexId/' application.

I was lost until I finaly found this blog. What we want is that if the user enters into the root of the server, the URL is rewritten to the Apex application. Actually not completely, because we rather have the ords extension left out the equation.

The trick is in the following three lines:
    RewriteEngine On
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^(.*)$ https://host.customer.nl/ords/f?p=ApexId [R,L]

When you use SSL,  and need to add this to the ssl.conf, you should add it to the virtual host configuration, just
    RewriteEngine On
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^(.*)$ https://host.customer.nl/ords/f?p=ApexId [R,L]

</IfModule>
</VirtualHost>


We could fine tune this to do something like the following:
    <Location /f>
SetHandler weblogic-handler
WLSRequest ON
WebLogicCluster weblogic.host1.customer.local:7003, weblogic.host2.customer.local:7003
PathPrepend /ords
KeepAliveEnabled on
KeepAliveSecs 10
</Location>
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^(.*)$ https://host.customer.nl/f?p=ApexId [R,L]
</IfModule>
</VirtualHost>

Although I did not test this yet, that would at least remove the append of the /ords in the URI. It would only append the /f?p=150 part.Currently I don't know how to prevent that, since Apex does need these parameters. For Weblogic and OHS Apex is an application that works with parameters in the URI.


Oracle Linux: Clean Yum dbcache

Fri, 2017-06-16 06:52
These weeks we provide a course on Weblogic 11g and 12c Basic and Advanced Administration and Tuning & Troubleshooting. We use an Oracle Linux 6 VM with a Database and Weblogic 11g and 12c.

We got some complaints (issued in a friendly way, by the way) from the attendees that a filesystem of the VM got filled up.

One of the attendees found out that the following folder containt some very large files:
/var/cache/yum/x86_64/6Server/public_ol6_latest

-rw-r--r--. 1 root root 3225280512 Jun 15 21:10 other.xml.gz.sqlite

-rw-r--r--. 1 root root  475535360 Jun 15 21:07 primary.xml.gz.sqlite

This is apparently a yum cache that can be emptied with the following command:
# yum clean dbcache./startStopDmn.sh stop Servers fmw $1

I should remember this to do before sharing a VM.
Thanks Frederique for this smart tip.

HTTP Server redirects for Weblogic 12c and SAML2

Tue, 2017-05-16 09:28
Last few months I got busy with SAML2 and Weblogic 12c as Service Provider. One with ADFS and another using SurfConext as an IdP.

In both cases a HTTP server is used as a reversed proxy, in one case it is Oracle HTTP Server 12c, in the other we use Apache. Although OHS is based on Apache, of course, it has the Weblogic proxy plugin enabled by default. With Apache this is not the case.

So there are a few things to consider.

WebLogic Proxy Plugin in ApacheThe Managed server needs to ‘know’ that the End User approaches the application over TLS (HTTPS), although the HTTP Server ‘offloads’ the security. During the SAML authentication, Weblogic and the Identity Provider redirect the browser back-and-forth to authenthicate and eventually process the saml token. In the end the browser should be redirected to the application. If Weblogic does not ‘know’ the application is approached via an HTTP server over HTTPS, it might redirect to the HTTP channel.

To solve that, the HTTP server should use the Weblogic Proxy Plugin. To configure that, see Oracle® Fusion Middleware Using Oracle WebLogic Server Proxy Plug-Ins 12.1.2 - Configuring WLS Web Server Proxy Plug-In for Apache HTTP Server.

Set Proxy Plugin
To make use of the Weblogic proxy plugin, so that the AdminServer considers it, you need to tell it that it is 'fronted' by it.

To set it on managed server level, go to the server, tab Configuration->General:



Under Advanced, find the 'Weblogic Plug-In Enabled' option and set it to 'yes':
 You can do this on domain, cluster and managed server level. See also this A-team blog. In recent upgrades of Weblogic, the checkbox is replaced by a pulldown list.
Setting the Frondend HostAnother thing is that the URL that is used in the browser to connect to your application is the HTTP Server's host, not the Weblogic host. Also, propably the HTTP Server listens on port 80 (HTTP) and 443 (HTTPS), while your managed server might listen on a port in the 7000, 8000 or 9000 ranges.

But in the redirects, Weblogic has to redirect the application to the HTTP Server, so it needs to know what that address is. This registered in the FrontEnd Host. This is also a setting that  can be set on both Server and Cluster level. To set it on Server level, go to that server, tab Protocols->HTTP.

Then set the Frontend Host and the ports to the particular values of your HTTP Server.




Setting the Frontend HTTP Port to 0, means that the port is not fetched from the Header. So it will use the Managed Server port. However, the configuration of Apache (using the Weblogic Plugin) should be such that HTTP is not routed, but only HTTPS should be passed through. Otherwise set the HTTP Port also to 80.
Application Routing
The application has it's root URI. Let's say it is /MyServiceApp. Let's say OHS is running on the URL https://www.darwin-it.nl. Then the application is reachable on https://www.darwin-it.nl/MyServiceApp. This has to be routed to the sp_server managed server, running on port 7003. Then the routing in the HTTP Server using the following location definition:
<Location /MyServiceApp>
NSSRequireSSL
WLSRequest On
WebLogicHost sp_server.darwin-it.local
WebLogicPort 7003
</Location>

SAML2 URLs RoutingDuring the saml interchange the browser is at one point directed to https://www.darwin-it.nl/MyServiceAppsaml2/sp/acs/post. The part until 'MyServiceAppsaml2' is the url registered as 'Published Site URL' in the saml2 configuration, under "Servers" –> Managed Server –> "Federation Services" –> "SAML 2.0 General". The managed server expects to be called on the root URI '/saml2'. You could simply append the HTTP Servers listen-address with /saml2  and provide that as a Published Site URL. Like https://www.darwin-it.nl/saml2. This is ok if you have only one application. But what if you expect multiple Service Provider Applications, deployed on different Managed Servers? You might want to differentiate on the different services. So what if I wanted to use the URL:  https://www.darwin-it.nl/MyServiceAppsaml2? That is ok, if you rewrite the url in the HTTP Server. With the Weblogic Proxy Plugin you can do that with the PathTrim
and PathPrepend options:
<Location /MyServiceAppsaml2>
NSSRequireSSL
WLSRequest On
WebLogicHost sp_server.darwin-it.local
WebLogicPort 7003
PathTrim /MyServiceAppsaml2
PathPrepend /saml2
</Location>

From the request URI it removes the /MyServiceAppsaml2 part, as defined with the PathTrim option. With PathPrepend the URL is prepended/prefixed with /saml2.

Time Skew and TimeZones
With SAML2 timing is everything.  With only a bit of time difference between the Service Provider and Identity Provider, even less then a minute, you can get a succesful SAML token, but with the checking of it by the Service Provider, that is on your server, you can get:
<[Security:090377]Identity Assertion Failed, weblogic.security.spi.IdentityAssertionException: [Security:090377]Identity Assertion Failed, weblogic.security.spi.IdentityAssertionException: [Security:096537]Assertion is not yet valid (NotBefore condition).> 
####<Apr 26, 2017 1:27:26 PM CEST> <Debug> <SecuritySAML2Service> <Oracle5> <ManagedServer_1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <884d7111-ffc4-46ad-b877-77395aa690a3-000031ce> <1493206046903> <BEA-000000> <exception info
javax.security.auth.login.LoginException: [Security:090377]Identity Assertion Failed, weblogic.security.spi.IdentityAssertionException: [Security:090377]Identity Assertion Failed, weblogic.security.spi.IdentityAssertionException: [Security:096537]Assertion is not yet valid (NotBefore condition).

To be able to compare the dates of the assertions requests and responses with the server time, it might be handy to have the service provider server run with the same time zone setting. You can do that in the setUserOverrides.cmd/.sh script in the $DOMAIN_HOME/bin folder.


Edit it and add the following lines (Windows format):
@rem set EXTRA_JAVA_PROPERTIES=%EXTRA_JAVA_PROPERTIES% -Duser.timezone='Europe/Amsterdam'
set EXTRA_JAVA_PROPERTIES=%EXTRA_JAVA_PROPERTIES% -Duser.timezone=GMT

Choose which line you want to have uncommented.

And make sure that server time is synced with internet or a central time server. Have it adapted regularly. And maybe force it to be updated.

Conclusion
The saml2 configuration on Weblogic is not so hard. But the difficulty is in the several layers and different parties involved. But I provided a few extra considerations and solutions here.

BPEL 12.2.1.2 and UMS: no wire target for reference...

Wed, 2017-05-10 04:13
A moment ago I stumbled on this question on the Oracle community. User Saurabh  tries to build a composite to send email from BPEL. But on testing the deployed composite in EM it fails with a remote fault. This despite of a correct configuration of the email driver and being able to test that   using soa-infra -> Service Engines -> Human Workflow -> Notification Management.

He found that the problem was a bug in JDeveloper/SOA QuickStart, causing the email activity not being wired to a Email UMS notification reference.

Fortunately there is a patch for it, as described in DocId 2235669.1 on support. Apply Patch 24898307 to solve this.

Found this interesting enough to have it (b)logged. At least for my self...



Introducing Darwin Oracle Type Accelerator

Mon, 2017-05-08 06:37

Years ago, I created a set of XSL templates and queries to create object types out of queries on the datadictionary of the Oracle Database. It only did basic types on tables, and selects on those. I wrote an article on it that still can be downloaded here. Later, I extended the framework to also do inserts and updates and follow foreign keys. The thing with foreignkeys is that you can handle them a as a detail-tables like Order Lines with an Order. Or lookup, like a place of birth, or country of origin, etc.

I must say I was quite pleased with what it could do.

Lately I stumbled on a question on community.oracle.com that ran about updating multiple tables using the database adapter. Normally you would need to do multiple invokes of a Database Adapter definitions, at least one per table, to do inserts and or updates.  I created thus because Oracle Types are a very powerful means to get data from a datamodel with multiple tables in just one invoke. Or insert data into it. You just create a pl/sql procedure with a parameter based on the root object. The database adapter wizard creates the accompanying XSD's and you only need to do a proper mapping in to the input variable and do the invoke. In the Pl/Sql procedure you call the particular method (sel, ins, upd) of the root object, that propagates into all the child and lookups. Most of the times that is enough. In more complex models, you might enhance the calling pl/sql.

I'm happy to announce that a moment ago I put my source on Github. I renamed it to Darwin Oracle Type accelerator. I did not have the change to create elaborate documentation. But in short:

  • In $GitHub/Dotacc/Source/Dotacc/ddl/owner\ you'll find setup scripts for the framework.
  • With setupTables.sql and setupPlsql.sql you create a bunch of tables and pl/sql with XXX_ as a prefix to support the generation of types.
  • insertXslNL2.0.sql or insertXslEN2.0.sql creates the xsl in the xxx_xmldocuments table.
  • $GitHub/Dotacc/Source/Dotacc/Config contains a set of insert scripts to define what tables, foreignkeys etc. to handle: 
    • XXX_TABLES: defines the tables for which you want to generate the types. The column generation_order defines in which order the types are created or dropped at recreation. 
    • XXX_FK_DEFINITIONS: defines the foreignkeys to consider. The FK_TYPE column defines if it should be considered as a child table ('DETAIL') or Lookup ('LOOKUP'). 
    • XXX_DERIVED_COLUMNS: defines virtual columns that can be looked up from another table. You can define a method that is added to do the actual lookup based on a lookup value from a key column. 
    • XXX_CUSTOM_METHODS: can be used to add custom methods to the object type.
    • I added two datamodels (Doe_owner and hbc_owner under ddl) as a sample model. It would be nice to come up with a sample filling for named tables.
To do a recreate of the types, perform:
execute  xxx_gen_objects.recreate_objects;

 If you have remarks, post a comment on the blog and/or do a request to contribute on github or send me a PM on community.oracle.com.

List Weblogic 12c System Components

Thu, 2017-05-04 07:30
Besides starting and stopping servers, it turns out handy to be able to list the particular system components of a Weblogic domain. For most domains, you might have an embedded/colocated Oracle HTTP server.
But we're also busy with installing BI publisher domains, and there several BI Components are created. To list which ones are created (and determine where things might went wrong) it might be handy to list all the system components. For that I created the following script. For the referenced fmw12c_env.sh and fmw.properties, I refer to Start and stop a WebLogic (SOA/OSB) Domain.

The wlst script is listSystemComponents.py:
#############################################################################
# List System Components of FMW Domain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2017-04-20
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
pad=' '
#
#
def usage():
print 'Call script as: '
print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
print 'Property file should contain the following properties: '
print "adminUrl=localhost:7001"
print "adminUser=weblogic"
print "adminPwd=welcome1"
#
# Connect To the AdminServer
def connectToAdminServer(adminUrl, adminServerName):
print(lineSeperator)
print('Try to connect to the AdminServer')
try:
connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the AdminServer adminUser and adminPwd properties')
connect(adminUser, adminPwd, adminUrl)
#
# Get the Servers of Domain
def getSystemComponents():
print(lineSeperator)
print('\nGet SystemComponents from domain')
serverConfig()
cd("/")
sysComponents = cmo.getSystemComponents()
return sysComponents
#
# Boolean to string
def bool2str(bool):
result='false'
if bool:
result='true'
return result
#
# Descr system component type
def descSysCompType(sysComponentType):
result=sysComponentType
if sysComponentType=='OHS':
result='Oracle HTTP server'
elif sysComponentType=='OBIPS':
result='BI Presentation Service'
elif sysComponentType=='OBIS':
result='BI Server'
elif sysComponentType=='OBISCH':
result='BI Scheduler'
elif sysComponentType=='OBICCS':
result='BI Cluster Controller'
elif sysComponentType=='OBIJH':
result='BI JavaHost'
else:
result=sysComponentType
return result
#
# Start clusters
def showSystemComponents():
print(lineSeperator)
print ('Show SystemComponents')
sysComponents=getSystemComponents()
#
if (len(sysComponents) > 0):
print('SystemComponent '[:30]+'\t'+'Type '[:20]+'\tAutoRestart\tMachine')
for sysComponent in sysComponents:
sysCompName = sysComponent.getName()
sysCompNamePad=sysCompName+pad
sysCompType=descSysCompType(sysComponent.getComponentType())
sysCompTypePad=sysCompType+pad
machine=sysComponent.getMachine()
if machine is None:
machineName = 'None'
else:
machineName = machine.getName()
print sysCompNamePad[:30]+'\t'+sysCompTypePad[:20]+'\t'+bool2str(sysComponent.getAutoRestart())+'\t\t'+machineName
else:
print('No SystemComponents found!')
#
print ('\nFinished showing SystemComponents.')
#
# Main
def main():
try:
#
print(lineSeperator)
print('\nConnect to the AdminServer: '+adminServerName)
connectToAdminServer(adminUrl, adminServerName)
#
showSystemComponents()
#
print('\nExiting...')
exit()
except NameError, e:
print('Apparently properties not set.')
print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
usage()
except:
apply(traceback.print_exception, sys.exc_info())
exit(exitcode=1)
#call main()
main()
exit()

To call this easily for different environments I have the following bash listSystemComponents.sh script:
#!/bin/bash
#############################################################################
# List Domain System Components using wlst
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2017-04-19
#
#############################################################################
#
. fmw12c_env.sh
export ENV=$1
echo
echo "List Domain System Components"
wlst.sh ./listSystemComponents.py -loadProperties ${ENV}.properties

I introduced an ENV variable here, so you can call it as:
[oracle@darlin-vce scripts]$ ./listSystemComponents.sh fmw

This exends the fmw argument with '.properties'. So if you want to use this for different environments, just copy the fmw.properties to a bip-acc.properties file or something like that and adapt the properties. Then using it against that environment is as calling:

[oracle@darlin-vce scripts]$ ./listSystemComponents.sh bip-acc

Single Sign On for Apex with ADFS? With Weblogic 12c and ORDS: Yes, you can!

Tue, 2017-05-02 11:41
Lately we implemented a Single Sign On solution for Apex, based on Weblogic 12cR2, ORDS 3.0.9, and ADFS as a federated Identity Provider. This combination turns out to be a marriage of 3 different worlds. So we ran in to a several issues that were not described in one simple how-to document. So in this document I try to assemble the information needed to do the end 2 end configuration (apart from the OHS configuration).

For most of the SAML2 configuration on Weblogic, we could have my earlier article  on SAML2.0 on Weblogic 11g, as a guide: Service Provider initiated SSO on WLS11g using SAML2.0 .

This helped a great deal with regards to ADFS and 12c. The rest of the issues I'd like to cover here, for future reference.
ORDSORDS can be installed in the regular way. I downloaded ORDS and unzipped it in the weblogic domain home. Then I did the setup using:
[oracle@darlin-vce-db ords309]$ java -jar ords.war setup

Then create the i.war containing the Apex and ORDS images. First copy the Apex images from the Apex home to the ORDS images folder. And then create the i.war:
[oracle@darlin-vce-db ords309]$ java -jar ords.war static /u01/data/oracle/config/domains/SP_domain/ords309/images

It is important to provide a complete/absolute path to the this command. This command creates an i.war that contains a reference to the images folder. You can see this as a virtual directory configuration as you would do in Apache/Oracle HTTP Server. It in fact only contains a web.xml, sun-web.xml and a weblogic.xml that contain a reference to that folder. For instance, the weblogic.xml contains:
<weblogic-web-app xmlns="http://www.bea.com/ns/weblogic/weblogic-web-app">
<!-- This element specifies the context path the static resources are served from -->
<context-root>/i</context-root>
<virtual-directory-mapping>
<!-- This element specifies the location on disk where the static resources are located -->
<local-path>/u01/data/oracle/config/domains/SP_domain/ords309/images</local-path>
<url-pattern>/*</url-pattern>
</virtual-directory-mapping>
</weblogic-web-app>

Before deploying the ords.war and i.war files to Weblogic (with Custom Roles), you'll need to make a manual adjustment to the ords.war.
The thing is that ORDS must be instructed to hand over the authentication to Weblogic. To do so, you'll need to add the following to the WEB-INF/web.xml file in the ords.war:
    <!-- Security Constraint -->
<security-constraint>
<web-resource-collection>
<web-resource-name>SecurePages</web-resource-name>
<description>These pages are only accessible by authorized users.</description>
<url-pattern>/f/*</url-pattern>
<!-- <http-method>GET</http-method> -->
</web-resource-collection>
<auth-constraint>
<description>These are the roles who have access.</description>
<role-name>Anonymous</role-name>
</auth-constraint>
<user-data-constraint>
<description>This is how the user data must be transmitted.</description>
<transport-guarantee>NONE</transport-guarantee>
</user-data-constraint>
</security-constraint>
<!-- Login Config -->
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>myrealm</realm-name>
</login-config>
<security-role>
<role-name>Anonymous</role-name>
</security-role>

In WEB-INF/weblogic.xml add the security-role-assignment for the role Anonymous:
<weblogic-web-app  xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app
http://xmlns.oracle.com/weblogic/weblogic-web-app/1.6/weblogic-web-app.xsd"> <!-- Weblogic 12c -->
<container-descriptor>
<prefer-web-inf-classes>true</prefer-web-inf-classes>
</container-descriptor>
<session-descriptor>
<persistent-store-type>replicated_if_clustered</persistent-store-type>
</session-descriptor>
<security-role-assignment>
<!--<role-name>valid-users</role-name> -->
<role-name>Anonymous</role-name>
<principal-name>users</principal-name>
</security-role-assignment>
<context-root>/ords</context-root>
</weblogic-web-app>
This is necessary, because after the Authentication using SAML2, the Rolemapper kicks in. And that one has to find a valid role.
APEX-Schema's
APEX uses REST-calls to fetch the images. It is important that all the database users relating to ORDS and APEX are unlocked, that passwords are known. Check for each of the following schema's if you can logon with the known password.

  • ORDS_PUBLIC_USER
  • APEX_PUBLIC_USER
  • APEX_LISTENER
  • APEX_REST_PUBLIC_USER
  • ORDS_METADATA
Then perform the following steps from doc 2075837.1:
  1. Go into the directory where the extracted the full installation of APEX resides
  2. Login to SQLPlus as the SYS user (as sysdba)
  3. Rerun the @apex_rest_config.sql script to recreate the "connect through" privileges for the installation:
    @apex_rest_config
  4. Hit enter for the APEX_LISTENER and APEX_REST_PUBLIC_USER password prompt. (passwords apparently not used for this script)
  5. Verify with the following statement that the connect through grant is present:
    select * from dba_proxies
    where proxy='APEX_REST_PUBLIC_USER'
    and client='APEX_PUBLIC_USER';
  6. If the grant is still missing the customer should verify that the file apex_rest_config_core.sql contains the following statement and run the script again.
    -- Allow REST user to proxy into APEX_PUBLIC_USER for built-in RESTful Services
    alter user APEX_PUBLIC_USER grant connect through ^RESTUN.;
ORDS Validation
From ORDS 3.0.5 onwards ORDS expects a column in the table ords_metadata.apex_pool_config called pool_name. So you could end up with an ORDS installation/configuration that seems to work, but you might not get the Apex application images shown. This is described in this community question. If the applications images don't show up:
  1. Validate that this column exists, through a describe of the table, and check if the table contains any rows.
  2. If necessary, perform the following to chreate the column and fill the table:
cd /u01/data/oracle/config/domains/SP_domain/ords309
java -jar ordsname.war validate

APEX Authentication Schema
To have the Apex application use the authenticated user from Weblogic/ADFS, the authentication scheme needs to be changed. 
This has to be done in a way that in the authentication scheme (shared components --> security --> authentication) APEX fetches the user from the  REMOTE_USER header variable: 


Weblogic 12 and ADFSADFS will use SHA-256 for signing by default. But WLS currently does not support that for SAML2. Although for many other services WLS does 'know' how to do SHA-256. I found articles how to update the policies for OWSM to use SHA-256. But I could not find how to do the same for the SAML2 configuration of Weblogic. So have ADFS do the signing with SHA-1. This might seem not secure, but when using TLS this is a minor thing. Although Weblogic 12c should solve this, in my opinion. See also this blog on Weblogic with ADFS, point 5 under the Takeway or Gotchas.

Another thing we found is that ADFS expects a valid 'https://' url to the ServiceProvider for the entity-id. A regular unique string does not suffice. ADFS apparently checks this URL to be valid. So I used the default TLS-url to the Oracle HTTP server that I used to reverse proxy to the SP Weblogic.

Lastly, in ADFS we needed to add an extra explicit claim had to fill the urn:mace:dir:attribute-def:uid saml2-attribute. This is needed for the identy mapper.

Identity Mapper Class
I updated the IdentityMapper class that I used in my earlier blog. I found a little bug in determining the actual identity/username, that apparently did not occur in 11g. But I also refactored the class a bit, to my latest Java knowledge, for as far as applicable.

package nl.darwinit.wls.saml;

import com.bea.security.saml2.providers.SAML2AttributeInfo;
import com.bea.security.saml2.providers.SAML2AttributeStatementInfo;
import com.bea.security.saml2.providers.SAML2IdentityAsserterAttributeMapper;
import com.bea.security.saml2.providers.SAML2IdentityAsserterNameMapper;
import com.bea.security.saml2.providers.SAML2NameMapperInfo;

import java.security.Principal;

import java.util.ArrayList;
import java.util.Collection;
import java.util.logging.Logger;

import weblogic.logging.LoggingHelper;

import weblogic.security.service.ContextHandler;


public class WLSSaml2IdentityMapper implements SAML2IdentityAsserterNameMapper, SAML2IdentityAsserterAttributeMapper {
public static final String ATTR_PRINCIPALS = "com.bea.contextelement.saml.AttributePrincipals";
public static final String ATTR_USERNAME = "urn:mace:dir:attribute-def:uid";
private Logger lgr = LoggingHelper.getServerLogger();
private final String className = "nl.darwinit.wls.saml.WLSSaml2IdentityMapper";
//

/**
* Map Name Info to String
* @param saml2NameMapperInfo
* @param contextHandler
* @return
*/
@Override
public String mapNameInfo(SAML2NameMapperInfo saml2NameMapperInfo, ContextHandler contextHandler) {
final String methodName = className + ".mapNameInfo";
debugStart(methodName);
String user = null;
debug(methodName, "saml2NameMapperInfo: " + saml2NameMapperInfo.toString());
debug(methodName, "contextHandler: " + contextHandler.toString());
debug(methodName, "contextHandler number of elements: " + contextHandler.size());
// getNames gets a list of ContextElement names that can be requested.
String[] names = contextHandler.getNames();
// For each possible element
for (String element : names) {
debug(methodName, "ContextHandler element: " + element);
// If one of those possible elements has the AttributePrinciples
if (element.equals(ATTR_PRINCIPALS)) {
// Put the AttributesPrincipals into an ArrayList of CustomPrincipals
ArrayList<CustomPrincipal> customPrincipals =
(ArrayList<CustomPrincipal>) contextHandler.getValue(ATTR_PRINCIPALS);
int i = 0;
String attr;
if (customPrincipals != null) {
// For each AttributePrincipal in the ArrayList
for (CustomPrincipal customPrincipal : customPrincipals) {
// Get the Attribute Name and the Attribute Value
attr = customPrincipal.toString();
debug(methodName, "Attribute " + i + " Name: " + attr);
debug(methodName, "Attribute " + i + " Value: " + customPrincipal.getCollectionAsString());
// If the Attribute is "loginAccount"
if (attr.equals(ATTR_USERNAME)) {
user = customPrincipal.getCollectionAsString();
// Remove the "@DNS.DOMAIN.COM" (case insensitive) and set the username to that string
if (!user.equals("null")) {
user = user.replaceAll("(?i)\\@CLIENT\\.COMPANY\\.COM", "");
debug(methodName, "Username (from loginAccount): " + user);
break;
}
}
i++;
}
}
// For some reason the ArrayList of CustomPrincipals was blank - just set the username to the Subject
if (user == null || "".equals(user)) {
user = saml2NameMapperInfo.getName(); // Subject = BRID
debug(methodName, "Username (from Subject): " + user);
}
return user;
}
}
// Just in case AttributePrincipals does not exist
user = saml2NameMapperInfo.getName(); // Subject = BRID
debug(methodName, "Username (from Subject): " + user);
debugEnd(methodName);
// Set the username to the Subject
return user;
// debug(methodName,"com.bea.contextelement.saml.AttributePrincipals: " + arg1.getValue(ATTR_PRINCIPALS));
// debug(methodName,"com.bea.contextelement.saml.AttributePrincipals CLASS: " + arg1.getValue(ATTR_PRINCIPALS).getClass().getName());
// debug(methodName,"ArrayList toString: " + arr2.toString());
// debug(methodName,"Initial size of arr2: " + arr2.size());
}
//

/**
* Map Attribute Info to Collection<Principal>
* @param attrStmtInfos
* @param contextHandler
* @return
*/
public Collection<Principal> mapAttributeInfo(Collection<SAML2AttributeStatementInfo> attrStmtInfos,
ContextHandler contextHandler) {
final String methodName = className + ".mapAttributeInfo";
Collection<Principal> principals = null;
if (attrStmtInfos == null || attrStmtInfos.size() == 0) {
debug(methodName, "AttrStmtInfos has no elements");
} else {
principals = new ArrayList<Principal>();
for (SAML2AttributeStatementInfo stmtInfo : attrStmtInfos) {
Collection<SAML2AttributeInfo> attrs = stmtInfo.getAttributeInfo();
if (attrs == null || attrs.size() == 0) {
debug(methodName, "No attribute in statement: " + stmtInfo.toString());
} else {
for (SAML2AttributeInfo attr : attrs) {
CustomPrincipal principal = null;
String principalName = "";
Collection<String> attrValues = attr.getAttributeValues();
if (!attrValues.isEmpty()) {
int attrValIdx = 0;
for (String attrValue : attrValues) {
debug(methodName,
"Value " + ++attrValIdx + " of " + attr.getAttributeName() + "= " + attrValue);
if (attrValIdx == 1) {
principalName = attrValue;
}
}
} else {
principalName = attr.getAttributeName();
}
principal = new CustomPrincipal(principalName, attr.getAttributeValues());
debug(methodName, "Add principal: " + principal.toString());
principals.add(principal);
}
}
}
}
return principals;
}
//
private void debug(String methodName, String msg) {
lgr.fine(methodName + ": " + msg);
}
//
private void debugStart(String methodName) {
debug(methodName, "Start");
}
//
private void debugEnd(String methodName) {
debug(methodName, "End");
}

}

To be complete, here is the principal class:
package nl.darwinit.wls.saml;

import java.util.Collection;

import weblogic.security.principal.WLSAbstractPrincipal;
import weblogic.security.spi.WLSUser;


public class CustomPrincipal extends WLSAbstractPrincipal implements WLSUser {
@SuppressWarnings("compatibility:1303497257301553869")
private static final long serialVersionUID = 1L;

private String commonName;
private Collection<String> collection;

public CustomPrincipal(String name, Collection<String> collection) {
super();
// Feed the WLSAbstractPrincipal.name. Mandatory
this.setName(name);
this.setCommonName(name);
this.setCollection(collection);
}

public CustomPrincipal() {
super();
}

public CustomPrincipal(String commonName) {
super();
this.setName(commonName);
this.setCommonName(commonName);
}

public void setCommonName(String commonName) {
// Feed the WLSAbstractPrincipal.name. Mandatory
this.setName(commonName);
this.commonName = commonName;
System.out.println("Attribute: " + this.getName());
// System.out.println("Custom Principle commonName is " + this.commonName);
}

public Collection<String> getCollection() {
return collection;
}

public String getCollectionAsString() {
String collasstr = "null";
if (collection != null && collection.size() > 0) {
for (String value : collection) {
collasstr = value;
break;
}
/*for (Iterator iterator = collection.iterator(); iterator.hasNext();) {
collasstr = (String) iterator.next();
return collasstr;
}*/
}
return collasstr;
}

public void setCollection(Collection<String> collection) {
this.collection = collection;
// System.out.println("set collection in CustomPrinciple!");
if (collection != null && collection.size() > 0) {
/*for (Iterator iterator = collection.iterator(); iterator.hasNext();) {
final String value = (String) iterator.next();
System.out.println("Attribute Value: " + value);
}*/
for (String value : collection) {
System.out.println("Attribute Value: " + value);
}
}
}

@Override
public int hashCode() {
final int prime = 31;
int result = super.hashCode();
result = prime * result + ((collection == null) ? 0 : collection.hashCode());
result = prime * result + ((commonName == null) ? 0 : commonName.hashCode());
return result;
}

@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (!super.equals(obj))
return false;
if (getClass() != obj.getClass())
return false;
CustomPrincipal other = (CustomPrincipal) obj;
if (collection == null) {
if (other.collection != null)
return false;
} else if (!collection.equals(other.collection))
return false;
if (commonName == null) {
if (other.commonName != null)
return false;
} else if (!commonName.equals(other.commonName))
return false;
return true;
}

}

In my earlier article I described how to add a reference to the jar file containing these classes to the java classpath field on the Server Start tab in the console.
In 12c this apparently does not work, the class is not picked up. Add it to the class path by creating/editing the setUserOverrides.sh/.cmd file.

Conclusion
This must be about it. We had quite a bit of Trial & Error. But most of the gotchas are listed here I think. But feel free to hire us if you need help. (Because I think you'll need the A-team for different issues...)

'No such file or directory' on starting your domain

Wed, 2017-04-26 05:30
Today I was triggered by this question. Earlier I had a similar problem, where I searched, and searched and searched and found the last section on this blogpost. Yes indeed, that is how it works when you blog: you might find your self finding your own blogposts again.

What is it about? Last year I wrote a nice set of scripts on installing Fusion Middleware and creating Fusion Middleware Weblogic domains.

I created the domain-creation-script based on scripts of Edwin Biemond, to get the credits straight.

However, in those scripts the  managed servers as wel as the admin server get Java Arguments set.

Those arguments refer to the logsHome property. I found that if you set JavaArgs you need to set redirects for weblogic.Stdout and weblogic.Stderr as well, like:
'-XX:PermSize=256m -XX:MaxPermSize=512m -Xms1024m -Xmx1532m -Dweblogic.Stdout='+logsHome+'AdminServer.out -Dweblogic.Stderr='+logsHome+'AdminServer_err.out'

Where logFolder should be an absolute path. This has to do with the context in which the nodemanager is starting the server. From that context the relative reference apparently does not evaluate to the proper location. You can however, leave the Java args empty.

So I changed my scripts to not use the getServerJavaArgs function, anymore, but get them from the property file. I replaced the xxxJavaArgsBase with xxxJavaArgs variables. And left them empty. Or you could simply hash-out the lines:

#cd('ServerStart/'+server)
#print ('. Set Arguments to: '+javaArgs)
#set('Arguments' , javaArgs)

for both the AdminServer and ManagedServers.
The configurator doesn't set the JavaArgs, it leaves them over to the setDomain.sh/.cmd and setStartupEnv.sh/.cmd or setUserOverrides.sh/.cmd . If you do so, you can use a relative path, and the servers will start properly.

New and improved (re-)start and stop scripts for Fusion MiddleWare.

Tue, 2017-04-18 08:06
Last year I created a set of start and stop scripts for Weblogic/Fusion MiddleWare. Although I was quite happy with them at the time, I found that there was quite a lot of duplicate code. And I think I could improve them by combining at least the wlst scripts into one, making it a lot better maintainable, but also opening up for restart functionality. And doing so make the scripts more generic. I took my (re-)start and stop script for OHS components as an example and came up with an improved set of start/stop scripts.

For the property files (fmw12_env.sh, fmw.properties, user key files, etc. ) I refer to  start and stop scripts for Weblogic/Fusion MiddleWare.


WLSTAs said I created just one wlst script, startStopDomain.py:
#############################################################################
# Start, Stop, Restart FMW Domain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2017-04-20
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
print 'Call script as: '
print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
print 'Property file should contain the following properties: '
print "adminUrl=localhost:7001"
print "adminUser=weblogic"
print "adminPwd=welcome1"
#
#
def connectToNM():
try:
wlsDomainHome = wlsDomainsHome+'/'+wlsDomainName
print(lineSeperator)
print('Try to connect to the Node Manager')
try:
nmConnect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, host=nmHost, port=nmPort, domainName=wlsDomainName, domainDir=wlsDomainHome, nmType=nmType)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the NodeManager adminUser and adminPwd properties')
nmConnect(username=adminUser, password=adminPwd, host=nmHost, port=nmPort, domainName=wlsDomainName, domainDir=wlsDomainHome, nmType=nmType)
print('Connected to the Node Mananger')
except WLSTException:
message='Apparently NodeManager not Started!'
print (message)
raise Exception(message)
#
# Start Admin Server
def startAdminServer(adminServerName):
# Set wlsDomainHome
print ('Connect to the Node Manager')
connectToNM()
print('Start AdminServer')
nmStart(adminServerName)
#
# Connect To the AdminServer
def connectToAdminServer(adminUrl, adminServerName):
try:
print(lineSeperator)
print('Try to connect to the AdminServer')
try:
connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the AdminServer adminUser and adminPwd properties')
connect(adminUser, adminPwd, adminUrl)
except WLSTException:
print('Apparently AdminServer not Started!')
startAdminServer(adminServerName)
print('Retry to connect to the AdminServer')
try:
connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the AdminServer adminUser and adminPwd properties')
connect(adminUser, adminPwd, adminUrl)
#
# Get the Name of the AdminServer of Domain
def getAdminServerName():
serverConfig()
cd('/')
adminServerName=cmo.getAdminServerName()
return adminServerName
#
# Get the Servers of Domain
def getDomainServers():
print(lineSeperator)
print('\nGet Servers from domain')
serverConfig()
cd("/")
servers = cmo.getServers()
return servers
#
# Get the Servers of Cluster
def getClusterServers(clustername):
#Cluster config to be fetched from ServerConfig
print(lineSeperator)
print('\nGet Servers from cluster '+clustername)
serverConfig()
cluster = getMBean("/Clusters/" + clustername)
if cluster is None:
errorMsg= "Cluster " + clustername + " does not appear to exist!"
print errorMsg
raise(Exception(errorMsg))
print "Found cluster "+ clustername+ "."
servers = cluster.getServers()
return servers
#
# Get the Domain Clusters
def getClusters():
#Cluster config to be fetched from ServerConfig
print(lineSeperator)
print('\nGet Clusters')
cd('/')
clusters = cmo.getClusters()
return clusters
#
# Get status of a server
def serverStatus(serverName):
serverRuntime=getMBean('/ServerRuntimes/'+serverName)
if serverRuntime is None:
print("Server Runtime for " + serverName + " not found, server apparently SHUTDOWN")
serverState="SHUTDOWN"
else:
print "Found Server Runtime for "+ serverName+ "."
serverState = serverRuntime.getState()
return serverState
#
# Start Server
# Expected to be in domainRuntime to get to the serverRuntimes.
def startMServer(serverName):
print(lineSeperator)
print('ServerName: '+serverName)
serverState = serverStatus(serverName)
print('Server '+serverName+': '+serverState)
if serverState=="SHUTDOWN":
print ('Server '+serverName+' is not running so start it.')
start(serverName,'Server')
elif serverState=="RUNNING":
print ('Server '+serverName+' is already running')
else:
print ('Server '+serverName+' in state '+serverState+', not startable!')
#
# Start servers in a cluster one by one.
def startMServers(serverList):
print(lineSeperator)
print ('Start servers from list: '+serverList)
servers=serverList.split(',')
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for serverName in servers:
startMServer(serverName)
#
print ('\nFinished starting servers.')
#
# Start servers in a cluster one by one.
def startClusterServers(clusterName):
print(lineSeperator)
print ('Start servers for cluster: '+clusterName)
servers=getClusterServers(clusterName)
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for server in servers:
serverName = server.getName()
startMServer(serverName)
#
print ('\nFinished starting servers.')
#
# Start servers in domain one by one.
def startDomainServers():
print(lineSeperator)
print ('Start servers for domain')
servers=getDomainServers()
adminServerName=getAdminServerName()
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for server in servers:
serverName = server.getName()
if (serverName!=adminServerName):
startMServer(serverName)
else:
print('Skip '+adminServerName)
#
print ('\nFinished starting servers.')
#
# Stop Server
# Expected to be in domainRuntime to get to the serverRuntimes.
def stopMServer(serverName):
print(lineSeperator)
print('ServerName: '+serverName)
serverState = serverStatus(serverName)
print('Server '+serverName+': '+serverState)
if serverState=="RUNNING":
print ('Server '+serverName+' is running so shut it down.')
shutdown(name=serverName, entityType='Server', force='true')
elif serverState=="SHUTDOWN":
print ('Server '+serverName+' is already down.')
else:
print ('Server '+serverName+' in state '+serverState+', but try to stop it!')
shutdown(name=serverName, entityType='Server', force='true')
#
# Stop servers in a list
def stopMServers(serverList):
print(lineSeperator)
print ('Stop servers from list: '+serverList)
servers=serverList.split(',')
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for serverName in servers:
stopMServer(serverName)
#
print ('\nFinished stopping servers.')
#
# Stop servers in a cluster one by one.
def stopClusterServers(clusterName):
print(lineSeperator)
print ('Stop servers for cluster: '+clusterName)
servers=getClusterServers(clusterName)
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for server in servers:
serverName = server.getName()
stopMServer(serverName)
#
print ('\nFinished stopping servers.')
#
# Stop servers in a domain one by one.
def stopDomainServers():
print(lineSeperator)
print ('Stop servers for domain')
servers=getDomainServers()
adminServerName=getAdminServerName()
# Need to go to domainRuntime to get to the serverRuntimes.
domainRuntime()
#
for server in servers:
serverName = server.getName()
if (serverName!=adminServerName):
stopMServer(serverName)
else:
print('Skip '+adminServerName)
#
print ('\nFinished stopping servers.')
#
# Start cluster
def startCluster(clusterName):
print(lineSeperator)
print ('Start cluster: '+clusterName)
try:
start(clusterName,'Cluster')
except WLSTException:
print "Apparently Cluster in incompatible state!", sys.exc_info()[0], sys.exc_info()[1]
startClusterServers(clusterName)
state(clusterName,'Cluster')
#
print ('\nFinished starting cluster '+clusterName)
#
# Start clusters in a list
def startClusters(clusterList):
print(lineSeperator)
print ('Start clusters from list: '+clusterList)
clusters=clusterList.split(',')
#
for clusterName in clusters:
startCluster(clusterName)
#
print ('\nFinished starting clusters.')
#
# Start clusters
def startDomainClusters():
print(lineSeperator)
print ('Start clusters')
clusters=getClusters()
#
for cluster in clusters:
clusterName = cluster.getName()
startCluster(clusterName)
#
print ('\nFinished starting clusters.')
#
# Stop cluster
def stopCluster(clusterName):
print(lineSeperator)
print ('Stop cluster: '+clusterName)
try:
#shutdown(clusterName,'Cluster')
shutdown(name=serverName, entityType='Cluster', force='true')
state(clusterName,'Cluster')
except WLSTException:
print "Apparently Cluster in incompatible state!", sys.exc_info()[0], sys.exc_info()[1]
state(clusterName,'Cluster')
print ('Try to stop servers for cluster: '+clusterName+', one by one')
stopClusterServers(clusterName)
#
print ('\nFinished stopping cluster '+clusterName)
#
# Stop clusters in a list
def stopClusters(clusterList):
print(lineSeperator)
print ('Stop clusters from list: '+clusterList)
clusters=clusterList.split(',')
#
for clusterName in clusters:
stopCluster(clusterName)
#
print ('\nFinished stopping clusters.')
#
# Stop all clusters of the domain
def stopDomainClusters():
print(lineSeperator)
print ('Stop clusters')
clusters=getClusters()
#
for cluster in clusters:
clusterName = cluster.getName()
stopCluster(clusterName)
#
print ('\nFinished stopping clusters.')
#
# StopAdmin
def stopAdmin():
print(lineSeperator)
print('Stop '+adminServerName)
shutdown(force='true')
serverState = serverStatus(serverName)
print('State '+adminServerName+': '+ serverState)
print('\nFinished stopping AdminServer: '+adminServerName)
#
# Start admin Server
def startAdmin():
print(lineSeperator)
print('Start and connect to '+adminServerName)
connectToAdminServer(adminUrl, adminServerName)
print('\nFinished starting AdminServer: '+adminServerName)
#
# Stop a domain
def stopDomain():
print (lineSeperator)
stopDomainClusters()
#
stopAdmin()
print ('\nFinished stopping domain.')
#
#
# Start a Domain
def startDomain():
print (lineSeperator)
#
startAdmin()
#
startDomainClusters()
#
print ('\nFinished starting domain.')
#
#
# (Re-)Start or Stop Domain
def startStopDomain(startStopOption):
if startStopOption=="stop" or startStopOption=="restart":
stopDomain()
if startStopOption=="start" or startStopOption=="restart":
startDomain()
#
# (Re-)Start or Stop Admin
def startStopAdmin(startStopOption):
if startStopOption=="stop" or startStopOption=="restart":
stopAdmin()
if startStopOption=="start" or startStopOption=="restart":
startAdmin()
#
# (Re-)Start or Stop Clusters
def startStopClusters(startStopOption, clusterList):
if startStopOption=="stop" or startStopOption=="restart":
if clusterList is None:
stopDomainClusters()
else:
stopClusters(clusterList)
if startStopOption=="start" or startStopOption=="restart":
if clusterList is None:
startDomainClusters()
else:
startClusters(clusterList)
#
# (Re-)Start or Stop Servers
def startStopServers(startStopOption, serverList):
if startStopOption=="stop" or startStopOption=="restart":
if serverList is None:
stopDomainServers()
else:
stopMServers(serverList)
if startStopOption=="start" or startStopOption=="restart":
if serverList is None:
startDomainServers()
else:
startMServers(serverList)
#
# Main
def main():
try:
print ('Args passed: '+ str(len(sys.argv) ))
componentList = None
idx = 0
for arg in sys.argv:
if idx == 0:
print 'ScriptName: '+arg
elif idx == 1:
# startStopOption: start, stop, restart
startStopOption=arg
elif idx == 2:
# componentType: AdminSvr, Domain, Clusters, Servers
componentType=arg
elif idx == 3:
componentList=arg
else:
componentList = componentList+','+arg
idx=idx+1
#
#componentName=sys.argv[3]
# Set wlsDomainHome
wlsDomainHome = wlsDomainsHome+'/'+wlsDomainName
#
print(lineSeperator)
if componentList is None:
componentStr = componentType
else:
componentStr = componentType+' '+componentList
if startStopOption=="start" or startStopOption=="restart":
print('(Re)Start '+componentStr+' in '+wlsDomainHome+', with option '+startStopOption)
elif startStopOption=="stop":
print('Stop '+componentStr+' in '+wlsDomainHome+', with option '+startStopOption)
else:
raise Exception("Unkown startStopOption: "+ startStopOption)
#
print(lineSeperator)
print('\nConnect to the AdminServer: '+adminServerName)
connectToAdminServer(adminUrl, adminServerName)
#
print('Connected, so proceed with: '+startStopOption+' of '+componentType)
#
if componentType=="AdminSvr":
startStopAdmin(startStopOption)
elif componentType=="Domain":
startStopDomain(startStopOption)
elif componentType=="Clusters":
startStopClusters(startStopOption, componentList)
elif componentType=="Servers":
startStopServers(startStopOption, componentList)
else:
raise Exception('Undefined componentType: '+componentType);
#
print('\nExiting...')
exit()
except NameError, e:
print('Apparently properties not set.')
print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
usage()
except:
apply(traceback.print_exception, sys.exc_info())
exit(exitcode=1)
#call main()
main()
exit()

The script can be called with two parameters and a property file:

  • startStopOption: start, stop, restart
  • component: AdminSvr, Domain, Clusters, Servers
Based on the value of component the start, stop or restart option is issued on either the AdminServer, the Domain clusters, servers or the complete domain. The main function starts with connecting to the AdminServer. If the AdminServer is not started yet, it will connect to the NodeManager to start the AdminServer.

In case of a restart or a stop the particular components are stopped. And in case of a start or restart the particular components are started. 

The advancement of this script, in contrast with other start/stop scripts I've seen is that it will give a start command to a cluster. Doing so, all the servers in a cluster will get a start command simultaneously. For stopping the server in a cluster, the WLSException that can be raised in the stop command is caught, to try to stop try to stop the servers one by one. 

Another advancement of this script is that is determined which clusters there are in a domain, and these are traversed one by one.

Update 2017-04-20: Today I added also the possiblilty to start a list of clusters or servers. If you don't provide a list of clusters or servers, all the clusters or servers in the domain are (re-)started or stopped. Except for the AdminServer, by the way.

Bash
To call the wlst script I created a set of bash scripts.

To start the AdminServer, startAdmin.sh:
#!/bin/bash
#############################################################################
# Start AdminServer using wlst and connect to it.
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 2.2, 2016-04-18
#
#############################################################################
#
. fmw12c_env.sh
echo
echo Start AdminServer
wlst.sh ./startStopDomain.py start AdminSvr -loadProperties fmw.properties

To stop or restart the AdminServer, replace start in 'wlst.sh ./startStopDomain.py start AdminSvr' by either stop or restart. And save the file to either stopAdmin.sh or restartAdmin.sh.

To create the corresponding files for Domain or Clusters replace AdminSvr in the same line with either Domains or Clusters.

I think you'll get the drift. And of course you should adapt the comments and echo's.

But maybe a more generic (re-)start or stop script is startStopDmn.sh:

#!/bin/bash
#############################################################################
# Start Domain components using wlst
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2017-04-19
#
#############################################################################
#
. fmw12c_env.sh
export START_STOP_OPTION=$1
export COMPONENT_TYPE=$2
export ENV=$3
export COMPONENT_NAME=$4
echo
echo "(Re-)Start or stop Domain components"
wlst.sh ./startStopDomain.py ${START_STOP_OPTION} ${COMPONENT_TYPE} "${COMPONENT_NAME}" -loadProperties ${ENV}.properties

Then you could do a startAdmin.sh like:
./startStopDmn.sh start AdminSvr fmw

And a stopAdmin.sh like:
./startStopDmn.sh stop AdminSvr fmw

Or a restartServers.sh like:
./startStopDmn.sh restart Servers fmw $1

Here you can provide a comma-separated list of servers, between double-quotes:
./restartServers "soa_server1,soa_server2"
etc. etc.
Advantage of this approach is that it allows you to use the same script set for multiple environemts, by duplicating and adapting the fmw.properties file to other environments (like domains).

BPM BAC Subversion Server refusing connections

Thu, 2017-04-06 05:20
These days I work on setting up several development lifecycle environments for BPM, SOA and OSB. What means that we setup servers for Development and test, culminating eventually in supporting the setup of the acceptance and production servers.

Since we want to have development resemble production as much as possible, we installed a dual-node clustered BPM environment, including BAM. However, since it is development, we have the two Managed Servers per cluster on the same phsyical (actually virtual) host.

In 12c BPM introduced a new component the Process Asset Manager, as described here. But installing BPM on a dual node cluster on a single host, will give problems with the underlying BAC/Subversion Component.

Errors you could encounter in the logs are for example:

Status of Start Up operation on target /Domain_o-bpm-1_domain/o-bpm-1_domain/BPMComposer is STATE_FAILED
[Deployer:149193]Operation "start" on application "BPMComposer" has failed on "bpm_server2".
java.lang.RuntimeException: weblogic.osgi.OSGiException: Could not find bundle with Bundle Symbolic Name of:org.tmatesoft.svn.core
at weblogic.osgi.internal.OSGiAppDeploymentExtension.prepare(OSGiAppDeploymentExtension.java:273)
at weblogic.application.internal.flow.AppDeploymentExtensionFlow.prepare(AppDeploymentExtensionFlow.java:25)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:727)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:239)
at weblogic.application.internal.EarDeployment.prepare(EarDeployment.java:66)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:65)
at weblogic.deploy.internal.targetserver.operations.ActivateOperation.createAndPrepareContainer(ActivateOperation.java:229)
at weblogic.deploy.internal.targetserver.operations.StartOperation.createAndPrepareContainer(StartOperation.java:95)
at weblogic.deploy.internal.targetserver.operations.StartOperation.doPrepare(StartOperation.java:108)
at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:241)
at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:794)
at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1340)
at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:267)
at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:177)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:186)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:14)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:47)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:666)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
Caused by: java.lang.RuntimeException: Could not find bundle with Bundle Symbolic Name of:org.tmatesoft.svn.core
at weblogic.osgi.internal.OSGiAppDeploymentExtension.attachToApplicationBundleClassloader(OSGiAppDeploymentExtension.java:161)
at weblogic.osgi.internal.OSGiAppDeploymentExtension.prepare(OSGiAppDeploymentExtension.java:249)
... 26 more

[Deployer:149034]An exception occurred for task [Deployer:149026]start application BPMComposer on bpm_cluster.: weblogic.osgi.OSGiException: Could not find bundle with Bundle Symbolic Name of:org.tmatesoft.svn.core.

Or during deployment of OracleBPMBACServerApp on one of the managed servers:
[2015-02-04T07:04:05.848+00:00] [BPM_Ms1_2] [ERROR] [] [oracle.bpm.bac.svnserver] [tid: [ACTIVE].ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 88957d95-82b2-4365-aaea-ac834a54599d-00000003,0] [APP: OracleBPMBACServerApp] Unexpected error starting BAC SVN Server.[[
oracle.bpm.bac.subversion.server.exception.ServerException: java.net.BindException: Address already in use

WLS version: 12.1.3.0.0
authentication-provider: default
Servers: AdminServer+ local_bpel_ms1 + local_bpel_ms2
Cluster: local_bpel_cluster, cluster-messaging-mode: unicast

Configuration manager:
Protocol: PLAIN
Topology: ACTIVE_ACTIVE_CLUSTER


As described in Doc ID 1987120.1 on Oracle Suppport, this is due to the fact that by default in oracle.bpm.bac.server.mbeans.metadata MBean the Bac Admin node and the BacNode use the same ports. (I think the cause is a bit mis-phrased in the note....)

To solve this:
  1. Login to Enterprise Manager console : http://host:port/em
  2. Expand WebLogic Domain
  3. Right click on the domain and click on System MBean Browser
  4. In System MBean Browser search for: oracle.bpm.bac.server.mbeans.metadata and expand it
  5. Expand the first BacNode and check that the AdminPort and also the BacNode port are different:
  6. Do the same thing for the second BacNode:
  7. Make sure that all 4 ports are different.
  8. Do a netstat on the machine and check also that the ports used by BAC servers are not already in use. Change them if necessary.
Port check can be done by:
[oracle@darlin-vce-bpm logs]$ netstat -tulpn |grep 8323
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 192.168.1.239:8323 0.0.0.0:* LISTEN 23940/java
tcp 0 0 192.168.1.240:8323 0.0.0.0:* LISTEN 23988/java
[oracle@darlin-vce-bpm logs]$ netstat -tulpn |grep 8424
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 192.168.1.240:8424 0.0.0.0:* LISTEN 23988/java
[oracle@darlin-vce-bpm logs]$ netstat -tulpn |grep 8423
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8423 0.0.0.0:* LISTEN 23940/java
[oracle@darlin-vce-bpm logs]$
You see that the latest port, the AdminPort belonging to the empty AdminAddress on the bpm_server1 is listening on 0.0.0.0, in affect on every address. Apparently, it is not possible to set an explicit address. Since it is on one actual host, the bpm servers are attached to different virtual IP addresses.

Now, this enabled us to start both the OracleBPMBACServerApp  as well as the BPMComposer.
But when we started using the BPM Composer and opening a project and process, there were issues. We found that when you got in bpm_server2, all went well. But when the load balancer directed you to bpm_server1, you could get a Internal server error

We encountered the following errors in the diagnostic log on bpm_server1:
[2017-04-05T03:25:27.270+02:00] [bpm_server1] [NOTIFICATION] [] [oracle.bpm.bac.svnserver.configuration] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: <anonymous>] [ecid: 41a5a471-1881-4527-8638-c344af778e7c-0000000a,0:497] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] Default repository path: /u02/oracle/products/fmw/user_projects/domains/o-bpm-1_domain/bpm/bac/bpm_server2/repositories
[2017-04-05T03:25:27.273+02:00] [bpm_server1] [NOTIFICATION] [] [oracle.bpm.bac.svnserver.configuration] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: <anonymous>] [ecid: 41a5a471-1881-4527-8638-c344af778e7c-0000000a,0:497] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] Default repository path: /u02/oracle/products/fmw/user_projects/domains/o-bpm-1_domain/bpm/bac/bpm_server1/repositories
[2017-04-05T03:25:27.277+02:00] [bpm_server1] [ERROR] [] [oracle.bpm.bac.svnserver.replication] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: <anonymous>] [ecid: 41a5a471-1881-4527-8638-c344af778e7c-0000000a,0:497] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server[[
oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException.wrap(RepositoryException.java:56)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:98)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:74)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:59)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.runImpl(ActiveAARepository.java:360)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.run(ActiveAARepository.java:304)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:280)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:95)
... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:112)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:68)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 10 more

]]

This occurs just on one of the two servers, in our case bpm_server1. Apparently, the Bac server can't run more than once on the same host. Or, in an active-active configuration, one of the bpm servers does not know how to connect. I don't know yet, how it is supposed to work internally.

Toggling the configuration to Single node won't work. Restarting the servers will cause the bpm composer unable to start at all. But switching to active passive in the BacConfigurationManager helped:

Then restarted the bpm servers and the BPM Composer and the underlying PAM/Bac/Subversion were working on both servers.


Back from PaaSForum

Tue, 2017-04-04 09:32
So, and now we're back in business after a splendid week in Split, Croatia, to meet with about 200 great, enthusiastic con-colleagues from the fields of SOA and Weblogic. It was the yearly OPN Partner Community Forum, ran by the great Jürgen Kress. He did a formidable job to collect many informatic sessions on the latest news and products from Oracle. We could meet with product managers, eat  and drink with them. To my surprise I could shake hands with a Product Manager of PCS, with whom I worked closely on a try-out project at the end of last year.

It was at a great venu, Le Meridien, a short drive out side of Split. A nice hotel with fantastic views on the Adriatic Sea.

Last year on the 'OFM Forum' in Valencia, PaaS (Platform as a Service) as a 'MiddleWare Cloud' was an important subject. Oracle was heavily promoting  Cloud. This is why you could expect that this years forum was named 'PaaSForum'. And thus in one of the keynotes this was stressed by stating:


Now, our Dutch comedian Arjan Lubach made a video to introduce our country to president Trump:

So that made me introducing the statement:
I understand cloud 1st, but can Netherlands be 2nd? #PaaSForum pic.twitter.com/SMh4JX2zgs
— Makker (@Makker_nl) March 29, 2017 Fun aside, I heard many nice things. To sum up a view:

  • On ICS side, there would be a REPL CLI script to move iar's (ICS Archives) between environments, created by the A-Team. I should look it up, but don't know if it already released it.
  • There was an introduction of the API platform, with also a workshop on it.
  • PCS and ICS are suggested to be combined into the Integration Cloud. Which would be a very logical idea, bringing much better, uniformed user experience. Because, at my latest cloud project, last year, we actually defined the practice to always integrate via ICS not directly from PCS.
  • There would also be a script, created by a Product Manager, to 'massage' BPM projects to transform them into a PCS compliant project. Hope to get my hands on that too, that could certainly be helpful. The otherway around would be not so bad either.
  • Then there was the mention of AI Apps, or Application Intelligent apps. Intelligence from verticals. This could drive PCS in a more intelligent way, based on knowledge from different verticals. 
  • New DMN Decision service in #PCS . Got to check that out soon. #PaaS4SaaS, #PaaSForum . pic.twitter.com/PadknKyH9R
    — Makker (@Makker_nl) March 31, 2017
But most interesting I found was the introduction of CMMN in PCS. Or otherwise put: Dynamic Processes in PCS. Where conventional BPMN processes in PCS run in a strict predefined way, with Dynamic Processes Case Management capabilities are introduced in PCS. Much like Adaptive Case Management in BPM Suite. However, build in a new way with a graphical modeller in the composer:

So here you see vertical lanes that represent phases in the process. In each phase you can define activities that can represent a Human Task, or for instance, a (sub-)process:
Activities can be repeatable, required or Manually Activated. Much like we can do in ACM.

Then using Decision rules you can decide under which condition the case is transferred to another phase, activities are activated or deactivated, etc. Or this can be done by the knowledge-worker self.


So I'm very much looking forward to get my fingers on this new functionality.

Another interesting new development is the introduction of Chat bots. They were so heavily mentioned and demoed, there hardly weren't any presentations without mentioning them. I almost felt I was one of the few that liked them, although the heavily relying on Facebook messenger. Even the, in my perception, breeding ground of chat bots, Twitter, wasn't free of criticism on the usecase of chatbots. I, on the other hand found a great non-functional use of chatbots: a text based adventure:

If we could combine PCS's dynamic processes with the Oracle Chat bot cloud service, we could develop a nice text adventure where you could chase beacons with in the process. Each phase in the process could represent a location, different processes can represent several areas on the map. But if we can have the process instances interact with eachother, for instance by registering their locations in a DBCS, or sending correlated signals (much nicer), then you could make it into a multiplayer text adventure. My colleague Rob introduced the idea to be able to upload pictures, of real-life locations. You could readout the Exif-data to check if the foto really represent a GPS-location.

So, that might be the next rolling-gag or app on the next forum in 2018, for which I already introduced the hash-tag: #ChatBotForum. In the aftermath of the week, at Jürgens afterparty, one of the others also mentioned text adventures. So, apparently, it wasn't such a far-fetched idea.

There's so much to think about, have it land, and much is already said. I'm looking forward to see how it all works out in the next year. The sun died on the #PaaSForum'17:


Working up to #ChatBotForum'18...

Pages