Andrejus Baranovski

Subscribe to Andrejus Baranovski feed
Blog about Oracle technology
Updated: 6 hours 57 min ago

Microservice Approach for Web Development - Micro Frontends

Thu, 2018-05-17 12:16
This post is based on my Oracle Code 2018 Warsaw talk. View presentation on slides share:


Wondering what micro frontends term means? Check micro frontends description here. Simply speaking, micro frontend must implement business logic from top to bottom (database, middleware and UI) in isolated environment, it should be reusable and pluggable into main application UI shell. There must be no shared variables between micro frontends. Advantage - distributed teams can work on separate micro frontends, this improves large and modular system development. There is runtime advantage too - if one of the frontends stops working, main application should continue to work.

I have implemented micro frontends architecture with Oracle JET. Source code is available on GitHub repository. There are three applications, two with micro frontends and one is the master UI shell. Both micro frontends are implemented as JET Composite Components. First is hosted on WebLogic, it calls ADF BC REST service in the backend. Second is hosted on Node.JS and returns static data. First micro frontend implements listener, it allows to handle actions from the outside.


When JET application is accessed in your browser, bunch of HTML, JS and CSS files are downloaded from the server. Core idea with micro frontends - instead of loading HTML, JS and CSS for micro frontend from the same host as master app - load it from different host. JET Composite Component rendered inside master application will be downloaded from different host. Not only downloaded, all backend calls should go to that host too, not to the master host. JET Composite Component integration into master application architecture:


This is how it works in practice. Each of these charts is separate JET Composite Component, loaded as micro frontend from different host into master application. We can see that in network monitor. Loader.js scripts for both micro frontends are downloaded from different hosts:


Runtime advantage - if one or multiple micro frontends are down, application continues to run:


JET Composite Component runs on the client, even it is hosted in its own micro frontend. This gives possibility to subscribe to the events happening in the component in the master app and route that event to another micro frontend. In this example, once item is selected in job chart - employees chart (another micro frontend) is filtered:


Technical implementation

Main application must be configured to support remote module loading for JET Composite Component. Read more about it in Duncan Mills blog post - JET Custom Components XII - Revisiting the Loader Script. In short, you should add Xhr config in JET application main.js:


Server where micro frontend is hosted, must set Access-Control-Allow-Origin header.

Main module where both micro frontends are integrated is using JET module component. Each micro frontend in master UI shell is wrapped into JET module. This allows main application to function, even when micro frontend in the module stops:


JET module is initialized from variable, which returns module name:


Jobs module contains Jobs micro frontend - JET Composite Component. It is hosted and WebLogic and calls ADF BC REST in the backend. Component is assigned with listener:


The most important part is in JS script. Here instead of referencing JET Composite Component locally, we load it from remote host. This allows to develop and host micro frontend JET Composite Component on its own:


Listener refers to c2 element and cals the method. Element c2 in the main app relates to second micro frontend:


This component is loaded from another host, from Node.JS:


Important hint - for JET Composite Component to load from remote host, make sure to add .js for JET Composite Component script, as highlighted (refer to source code):

Comparing Intent Classification in TensorFlow and Oracle Chatbot

Tue, 2018-04-03 10:46
I have created sample set of intents with phrases (five phrases per intent, and ten intents). Using this set of data to train and build classification model with TensorFlow and Oracle Chatbot machine learning. Once model is trained, classifying identical sample phrases with both TensorFlow and Oracle Chatbot to compare results. Using Oracle Chatbot with both Linguistic and Machine Learning models.

Summary:

1. Overall TensorFlow model performs better. The main reason for this - I was training TensorFlow model multiple times, until good learning output (minimized learning loss) was produced.

2. Oracle Chatbot doesn't return information about learning loss after training, this makes it hard to decide if training was efficient or no. As consequence - worse classification results, can be related to slightly less efficient training, simply because you don't get information about training efficiency

3. Classification results score: 93% TensorFlow, 87% Oracle Chatbot Linguistic model, 67% Oracle Chatbot Machine Learning. TensorFlow is better, but Oracle Chatbot Linguistic model is very close. Oracle Chatbot Machine Learning model can be improved, see point 2

Results table (click on it, to see maximized):


TensorFlow

List of intents for TensorFlow is provided in JSON file. Same intents are used to train model in Oracle Chatbot:


TensorFlow classification model is created by training 2-layer neural network. Once training is completed, it prints out total loss for the training. This allows to repeat training, until model is produced with optimal loss (as close as possible to 0): 0.00924 in this case:


TensorFlow classification result is good, it failed to classify only one sentence - "How you work?" This sentence is not directly related to any intent, although I should mention Oracle Chatbot Linguistic model is able to classify it. TensorFlow offers correct classification intent as second option, coming very close to correct answer:



Oracle Chatbot

Oracle Chatbot provides UI to enter intents and sample phrases - same set of intents with phrases is used as for TensorFlow:


Oracle Chatbot offers two training models - linguistic and machine learning based.


Once model is trained, there is no feedback about training loss. We can enter phrase and check intent classification result. Below is sample for Linguistic model classification failure - it fails to classify one of the intents, where sentence topic is not perfectly clear, however same intent is classified well by Oracle Chatbot Machine Learning model:


Oracle Chatbot Machine Learning model fails on another intent, where we want to check for hospital (hospital search) to monitor blood pressure. I'm sure if it would be possible to review training quality loss (may be in the next release?), we could decide to re-train model and get results close to TensorFlow. Classification with Oracle Chatbot Machine Learning model:

Socket.IO Integration with Oracle JET

Thu, 2018-03-29 08:42
Socket.IO is a JavaScript library for realtime web applications. It comes in two parts - a client-side library that runs in the browser and a server-side library for Node.js. In this post I will walk you through complete integration scenario with Oracle JET.

Here you can see it in action. Send Event button from JET - sends message through Socket.IO to Node.js server side. Message is handled on server side and response is sent back to client (displayed in browser console):


Server side part with Socket.io is implemented in Node.js application and it runs on Express. To create Node.js application (which is just one json file in the beginning), run command:

npm init

To add Express and Socket.io, run commands:

npm install express --save
npm install socket.io --save

To start Node.js application on Express, run command:

npm start

Double check package.json, it should contain references to Express and Socket.IO:


Here is server side code for Socket.IO (I created server.js file manually). When connection is established with the client, message is printed. Method socket.on listens for incoming messages. Method socket.emit transmits message to client. In both cases we can use JSON structure for payload variable. There is cheatsheet for socket.emit - Socket.IO - Emit cheatsheet. Socket.IO server side:


Socket.IO client side can be installed into JET application with NPM. There is separate section in Oracle JET documentation, where you can read step by step instructions about 3-rd party library installation into Oracle JET - Adding Third-Party Tools or Libraries to Your Oracle JET Application. I would recommend manually include Socket.IO dependency entry into package.json in JET:


Then run command to fetch Socket.IO library into JET application node modules. Next continue with instructions from Oracle JET guide and check my sample code:

npm update

To establish socket connection - import Socket.IO into JET module and use io.connect to establish socket connection. Connect to the end point where Express is running with server side Socket.IO listener. Client side is using same socket.on and socket.emit API methods as server side:


Download sample code from my GitHub repository.

ADF on Docker - Java Memory Limit Tuning for JVM

Wed, 2018-03-28 03:46
It might look like a challenge to run Java in Docker environment, by default Java is not aware of Docker memory limits. Check this article for example - Java inside docker: What you must know to not FAIL.  I was able to run WebLogic and ADF (Essential WebLogic Tuning to Run on Docker and Avoid OOM) on Docker previously without Java memory issues, using JAVA_OPTIONS=-XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC. However after Docker upgrade to latest version, these settings didn't help anymore. I did't want to hardcode memory setting with -Xmx.

Java started to consume all available memory in Docker and eventually was killed. You can see this from chart below - memory is growing, killed and after restart growing again:


To solve this behaviour, I have applied settings from Java Platform Group, Product Management Blog - Java SE support for Docker CPU and memory limits. I have replaced JAVA_OPTIONS=-XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC set previously with JAVA_OPTIONS=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+UseG1GC.

JAVA_OPTIONS=-XX:+UnlockExperimentalVMOptions - XX:+UseCGroupMemoryLimitForHeap -XX:+UseG1GC did the job - JVM stays in Docker memory limits sharp:


This chart shows Java memory behaviour before and after settings were applied. From March 27th - Java memory is a straight line with JAVA_OPTIONS=-XX:+UnlockExperimentalVMOptions - XX:+UseCGroupMemoryLimitForHeap -XX:+UseG1GC:

Oracle JET Offline Persistence Toolkit - Offline Update Handling

Tue, 2018-03-27 09:42
Oracle JET Offline Persistence Toolkit supports offline update, create and delete operations. In this post I will describe update use case. Read previous post related to offline toolkit, where I explain how to handle REST pagination, querying and shredding - REST Paging Support by Oracle Offline Persistence in JET.

This gif shows scenario, where we go to offline mode and then changing data in multiple rows. Data update happens offline and each PATCH request is tracked by offline persistence toolkit:


As soon as we go online (Offline checkbox value is changed in Chrome Developer Tools) - requests executed while offline are replayed automatically against backend server:


We should see, how update flow is implemented in JET in this particular case. Once data is changed, we call submitUpdate function. This function in turn calls JET Model API function save. This triggers PATCH call to back-end to update data. If we are offline, JET offline persistence toolkit, transparently records PATCH request to be able to replay it later while online. There are no specific code changes needed by developer to support offline logic during REST call:


Once we go online, listener is invoked and it calls our function synchOfflineChanges. This function triggers request replay to the backend. This means we can control, when requests are replayed. Besides this, we can control each request which failed to be replayed - this is important, when data conflict happens during update in backend:


Online handler is registered with window.addEventListener in the same module, where persistence manager is defined:


Offline Persistence Toolkit 1.1.1 supports extensive logging. You can update to 1.1.1 version by running: npm install @oracle/offline-persistence-toolkit command:


To enable persistence toolkit logger, add persist/impl/logger module to your target module and call logger.option('level', logger.LEVEL_LOG):


Logger prints useful information about offline update, this helps to debug offline functionality:


Download sample application from GitHub repository.

ADF Declarative Component Example

Thu, 2018-03-22 12:48
ADF Declarative Component support is popular ADF framework feature, but in this post I would like to explain it from slightly different angle. I will show how to pass ADF binding and Java bean objects into component through properties, in these cases when component must show data from ADF bindings, such approach could offer robustness and simplify component development.

This is component implemented in the sample app - choice list renders data from ADF LOV and button calls Java bean method to print selected LOV item value (retrieved from ADF bindings):


JDeveloper provides wizard to create initial structure for declarative component:


This is ADF declarative component, it is rendered from our own tag. There are two properties. List binding is assigned with LOV binding object instance and bean property with Java bean instance defined in backing bean scope. In this way, we pass objects directly into the component:


LOV binding is defined in target page definition file, where component is consumed:


Bean is defined in the same project, where page which consumes ADF declarative component is created. We need to define component property type to match bean type, for that reason, we must create class interface in the component library and in target project implement it:


Component and main projects can be in the same JDEV application, we can use JDEV working sets to navigate between projects (when running main project, we dont want to run component project, component project is deployed and reused through ADF JAR library):


Bean interface is defined inside component:


Property for list binding is defined with JUCtrlListBinding type, this allows to pass binding instance directly to the component. Same for bean instance, using interface to define bean instance type, which will be assigned from the page, where component is used:


Declarative component is based on combination of ADF Faces components:


Download sample application from GitHub repository.

Find In Cache By Key ADF BC API Method Usage

Sun, 2018-03-11 11:05
What if you need to verify - if row with given key exists in fetched rowset? This could be useful while implementing validation logic. ADF BC API method findByKey - will trigger SQL call and fetch row from DB, if row with given key doesn't exist in fetched rowset. Luckily there is ADF BC API method called findInCacheByKey, this method only checks for row in fetched rowset, without going to DB - very convenient in certain situations, when you actually don't want to bring record from DB, if it wasn't fetched.

Imagine table with pagination feature. First ten rows are fetched and exist in the cache:


Now if we call custom method, where findInCacheByKey is invoked twice - you will see different results. First call is using key from fetched rowset - this call will find a row. Second call is using key, which doesn't belong to the fetched rowset - row is not in cache and call will return zero rows:


Download sample app from my GitHub repository.

REST Paging Support by Oracle Offline Persistence in JET

Tue, 2018-03-06 12:32
Oracle Offline Persistence query handler - Oracle Rest Query Handler supports pagination for Oracle ADF BC REST service out of the box. Check my previous post to see how querying works through offline persistence toolkit for ADF BC REST service - Shredding and Querying with Oracle Offline Persistence in JET.

Pagination is a must for large REST resources, its great that Oracle offline persistence toolkit supports it. Let's see it in action.

I navigate through the data with left/right arrows, this triggers REST call with pagination parameters - limit and offset. These are standard parameters supported by ADF BC REST. Requests are executed online:


All pages of data are cached by offline toolkit, if while offline we try to access previously cached page by executing REST request with paging parameters - we will get data from offline toolkit. Now I switch offline and try to navigate to the one of cached pages - data is retrieved from cache automatically:


If I navigate to the page, which was not cached (meaning - not accessed while online) - no results returned. In such situation I can navigate back (paging parameters will be updated) and cached data will be displayed for the page which was cached:


Paging navigation control buttons are calling JS functions to update startIndex:


Sample application is using JET Collection API to execute fetch requests. Collection is extended with getURL function which sets limit and offset parameters to execute paging request:


Once again, make sure to use Oracle Rest Query Handler in offline persistence toolkit configuration:


Fetch function is called through JET Collection API. Start index value is calculated dynamically - this allows to execute paging requests. Same function works online and offline, no need to worry about connection status, all online/offline logic is handled by persistence toolkit:


Sample application for this post is available on GitHub.

Classification - Machine Learning Chatbot with TensorFlow

Sat, 2018-03-03 10:10
Visual conversation flow is a first thing to create, when you want to build chatbot. Such flow will help to define proper set of intents along with dialog path. Otherwise it is very easy to get lost in conversation transitions and this will lead to chatbot implementation failure. Our chatbot for medical system doesn't make any decisions, instead it helps user to work with enterprise system. It gets user input and during conversation leads to certain API call - which at the end triggers enterprise system to execute one or another action. If user is looking for patient blood pressure results, chatbot will open blood pressure module with patient ID. If user wants to edit or review blood pressure results in general, chatbot will load blood pressure results module without parameters. This kind of chatbot is very helpful in large and complex enterprise systems, this helps to onboard new users much quicker without extra training for system usage. Example of visual conversation flow for chatbot:


Conversation intents can be logged in JSON file. Where you should list conversation patterns mapped with tags, responses and contextual information. Chatbot is not only about machine learning and user input processing, very important is to handle conversation contextual flow and usually this is done outside of machine learning area in another module. We will look into it later. Machine learning with neural network is responsible to allow chatbot to calculate tag probability based on user input. In other words - machine learning helps to bring the best matching tag for current sentence, based on predefined intents patterns. As long as we get probability for the intent tag - we know what user wants, we can set conversation context and in the next user request - react based on current context:


TensorFlow runs neural network, which trains on supplied list of intents. Each training run may produce different learning results, you should check total loss value - lower value, better learning result. Probably you will run training multiple times to get optimal learning model:


TensorFlow can save learned model to be reusable by classification API. REST interface which calls classification API is developed as separate TensorFlow module. REST is handled by Flask library installed into TensorFlow runtime:


Classification function gets user input from REST call and runs it through TensorFlow model. Results with higher probability than defined by threshold are collected into ordered array and returned back. We have classification function without REST annotation for local tests within TensorFlow runtime:


Let's see how classification works, result of classification will drive next action for the chatbot. Each classification request returns matched tag and probability. User input is not identical to the patterns defined in intents, thats why matching probability may differ - this is core part of machine learning. Neural network constructed with TensorFlow, based on learned model, assumes the best tag for current user input.

User input "Checking blood pressure results for patient". This input can be related to both tags blood_pressure_search and blood_pressure, but classification decides higher probability for the first option, and this is correct. Similar for user input "Any recommendations for adverse drugs?":


Through REST endpoint we can call classification function outside of TensorFlow environment. This will allows us to maintain conversation context outside TensorFlow:


Useful resources:

- TensorFlow notebooks and intents JSON are available on GitHub repository.
- Excellent article about Contextual Chatbots with TensorFlow
- My previous post about Red Samurai chatbot

Microservices with Oracle ADF Remote Regions

Sat, 2018-02-24 13:50
ADF remote regions - functionality available in the latest ADF 12c versions. ADF remote region runs on different server and content is delivered to consuming module through ADF remote region servlet. This allows to decouple large enterprise system into separate modules, each running independently on separate servers. Gained advantage - system becomes more scalable and reliable, even if several modules will be down, system will continue to be functional.

Concept of ADF remote regions, reminds me closely microservices architecture. Microservices - also known as the microservice architecture - is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities. The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack (as describe here).

Let's see how ADF remote regions are configured and consumed. Sample application (available for download from GitHub repository) is based on Employees and Jobs modules. Each module is deployed on different servers, Employees module is consumed in Jobs. Microservice here - Employees table. This table comes from loosely coupled service and is consumed within Jobs module:


Employees module runs on ADF server in Docker container (read more: Essential WebLogic Tuning to Run on Docker and Avoid OOM):


I will stop this module:


Jobs module still works, even Employees module is not available anymore - it is stopped. Otherwise if both modules would be deployed as single application - if application is down, system will be completely unavailable. But now users can access part of the functionality:


Will dive into technical part. ADF remote region is not different from the way how regular ADF region is consumed. You still must use ADF region tag, to define region:


Key difference is in region bindings - there is one extra property called Remote Connection. This connection defines source, from where remote region content is transferred. All other properties are the same, we can pass parameters too:


Consuming module must define ADF remote region connection. If connection details are correct, you should see ADF task flows with remote access listed:


Remote region connection wizard. You should use module context root and point to ADF remote region servlets rtfquery and rr:


This wizard can be opened by right clicking on Connections folder and going to New Connections section:


Consumer module should be enabled with remote region consumer support:


Producer module should be enabled with remote region producer support:


Producer module is configured with required servlets automatically, as soon as you enable remote region producer support:


Read more about ADF remote regions in ADF developer guide - 26.13 Creating Remote Regions in a Fusion Web Application.

Shredding and Querying with Oracle Offline Persistence in JET

Sat, 2018-02-17 14:44
I think offline functionality topic should become a trend in the future. Its great that Oracle already provides solution for offline - Oracle Offline Persistence toolkit. This is my second post related to offline support, read previous post - Oracle Offline Persistence Toolkit - Simple GET Response Example with JET. I have tested and explained with sample app how it works to handle simple GET response offline. While today I would like to go one step further and check how to filter offline data - shredding and querying offline.

Sample app is fetching a list of employees - Get Employees button. It shows online/offline status - see icon in top right corner. We are online and GET response was cached by persistence toolkit:


We can test offline behaviour easily - this can be done through Chrome Developer Tools - turn on Offline mode. Btw, take a look into Initiator field for GET request - it comes from Oracle Offline Persistence toolkit. As I mention it in my previous post - once persistence toolkit is enabled, all REST calls are going through toolkit, this is how it is able to cache response data:


While offline, click on Get Employees button - you should see data returned from cache. Did you noticed - icon in the top right corner was changed to indicate we are offline:


Ok, now we will see how shredding mechanism works (more about it read on GitHub). While offline, we can search for subset of cached data. Search By Name does that, it gets from cache entry for Lex:


Switch online and call same action again, but with different name - REST call will be invoked against back-end server as expected. Again it is transparent to JET developer, no need to worry if app state is online/offline, same REST request is done in both cases:


Let's take a quick look into implementation part (complete example is available on my GitHub repository).

Online/offline status icon is controlled by observable variable:


It is very simple to determine online/offline state. We need to add event listener for online/offline and reset observable variable accordingly:


Persistence toolkit supports Simple and Oracle shredder/query handlers. I'm using ADF BC REST for backend and so my choice is oracleRestJsonShredding and oracleRestQueryHandler. Oracle shredder understands REST structure returned by ADF BC REST. Oracle query handler support filtering parameters for ADF BC REST for offline filtering - this allows to use same query format for both online and offline. I was happy to read that Oracle query handler explicitly supports ADF BC REST - queryHandlers:


Same REST call with filtering is executed online and offline:

Oracle JET Composite Components - Manual for JET Coder

Wed, 2018-02-07 07:20
JET Composite Components - are useful not only to build UI widgets, but also to group and simplify JET code. In this post, I will show how to wrap JET table into composite component and use all essential features, such as properties, methods, events and slots.

Sample app code is available on GitHub. JET table is wrapped into composite component, it comes with slot for toolbar buttons:


What is the benefit to wrap such components as JET table into your own composite? To name a few:

1. Code encapsulation. Complex functionality, which requires multiple lines of HTML and JS code resides in the composite component
2. Maintenance and migration. It is easier to fix JET specific changes in single place
3. Faster development. There is less steps to repeat and less code to write for developer, when using shorter definition of the wrapper composite component

Sample application implements table-redsam component, for the table UI you can see above. Here is component usage example, very short and clean:


All the properties specific to given table are initialised in the module. Developer should provide REST endpoint, key values, pagination size and column structure. The rest is happening in the composite component and is hidden from the developer, who wants to implement a table:


Properties

We should take a look into array type property. Such property allows to pass array into component. This can be useful either to pass array of data to be displayed or array of metadata to help with component rendering. In our case we pass array of metadata, which helps to render table columns. Array type property is based on two attributes - Header Text and Field. Properties are defined in composite component JSON file:


Properties are retrieved from variable inside component and are assigned to local variables:


This is table implementation inside component, columns are initialised from component property:


Slots

Slot defines a placeholder, where developer who is using composite component can add additional elements. Slot is defined in component JSON definition file:


To define slot, JET slot component should be defined inside composite. You can control layout and location where slot will be rendered:


In our case, we use slot for table toolbar buttons. These buttons are added later, when developer is using composite. To place button into slot, put button inside composite component tag and assign defined slot name for the button. This will allow to render button in the slot:


Methods

Method defined in composite component, can be called from outside. In example below, I call JS function from toolbar slot button:


Function gets composite by ID and calls exposed method:


Method should be defined in composite JSON definition:


Method is implemented inside composite JS module:


Events

Events allows to implement external listeners. Basically this allows to override composite logic in external functions. Event is declared in composite JSON definition:


Composite tag contains event property mapped with external JS function, which will be called when event happens inside composite:


Function code in the module, it prints current row selection key:


Table is defined with listener property inside composite:


Listener inside composite initiates event, which will be distributed outside and handled by method defined in composite tag on-handle-selection property:


Let's see how it works. Call Method button invokes method inside composite:


Table row selection first triggers listener inside composite, then it initiates event and external listener is invoked too:


I think this lists pretty much all of the essential functionality given by JET composite components. I hope you will find it useful in your development.

Say Hello to Red Samurai Contextual Chatbot with TensorFlow Deep Neural Network Learning

Sun, 2018-02-04 02:33
We are building our own enterprise chatbot. This chatbot helps enterprise users to run various tasks - invoice processing, inventory review, insurance cases review, order process - it will be compatible with various customer applications. Chatbot is based on TensorFlow Machine learning for user input processing. Machine learning helps to identify user intent, our custom algorithm helps to set conversation context and return response. Context gives control over  sequence of conversations under one topic, allowing chatbot to keep meaningful discussion based on user questions/answers. UI part is implemented in two different versions - JET and ADF, to support integration with ADF and JET applications.

Below is the trace of conversations with chatbot:


User statement Ok, I would like to submit payment now sets context transaction. If word payment is entered in the context of transaction, payment processing response is returned. Otherwise if there is no context, word payment doesn't return any response. Greeting statement - resets context.

Intents are defined in JSON structure. List of intents is defined with patterns and tags. When user types text, TensorFlow Machine learning helps to identify pattern and it returns probabilities for matching tags. Tag with highest probability is selected, or if context was set - tag from context. Response for intent is returned randomly, based on provided list. Intent could be associated with context, this helps to group multiple related intents:


Contextual chatbot is implemented based on excellent tutorial - Contextual Chatbots with Tensorflow. Probably this is one of the best tutorials for chatbot based on TensorFlow. Our chatbot code follows closely ideas and code described there. You could run the same on your TensowFlow environment - code available on GitHub. You should run model first and then response Python notebooks.

Model notebook trains neural network to recognize intent patterns. We load JSON file with intents into TensorFlow:


List of intent patterns is prepared to be suitable to feed neural network. Patterns are translated into stemmed words:


Learning part is done with TensorFlow deep learning library - TFLearn. This library makes it more simple to use TensorFlow for machine learning by providing higher-level API. In particular for our chatbot we are using Deep Neural Network model - DNN:


Once training is complete and model is created, we can save it for future reuse. This allows to keep model outside of chatbot response processing logic and makes it easier to re-train model on new set of intents when required:


In response module, we load saved model back:


Function response acts as entry point to our chatbot. It gets user input and calls classify function. Classification function, based on learned model, returns list of suggested tags for identified intents. Algorithm locates intent by its tag and returns random reply from associated list of replies. If context based reply is returned, only if context was set previously:


Stay tuned for more blog posts on this topic.

Avoid Blind SQL Call from ADF Task Flow Method

Mon, 2018-01-29 15:42
Keep an eye open on ADF Task Flow Method Call activities where methods from ADF Bindings are called. JDEV 12c sets deferred refresh for ADF binding iterators related to TF Method Call activities and this causing blind SQL to be executed. Blind SQL - query without bind variables.

Let me explain the use case, so that it will be more clear what I'm talking about.

Common example - TF initialization method call where data is prepared. Typically this involves VO execution with bind variables:


Such method call could invoke binding operation either directly (pay attention - bind variable value is set):


Or through Java bean method using API:


My example renders basic UI form in the fragment, after TF method call was invoked:


If you log SQL queries executed during form rendering, you will see two queries instead of expected one. First query is executed without bind variables, while second gets correct bind variable assigned:


What is the cause for first query without bind variables? It turns out - iterator (with setting Refresh = deferred) from page definition mapped with TF method call is causing this. Somehow iterator is initialized not at the right time, when bind variable is not assigned yet and this causing blind SQL call:


Workaround is to set Refresh = never:


With Refresh = never, only one query is executed as expected, with bind variable assigned:


This may look minor, but trust me - with complex queries such fix could be a great help for performance tuning. Avoid executing SQL queries without bind variables.

Download sample application - ADFTFCallBindingApp.zip.

Oracle Offline Persistence Toolkit - Simple GET Response Example with JET

Sun, 2018-01-28 11:56
We have new tool from Oracle which can help to simplify offline logic implementation for JS apps.  In this post I will describe how to use Oracle Offline Persistence Toolkit with Oracle JET. However Offline Persistence is not constrained by JET usage only, this toolkit is available on NPM and can be integrated with other JS solutions.

I should emphasise - offline toolkit primary role is to enable mobile hybrid apps to work offline. In my opinion, toolkit usage doesn't stop here. It can enable user to continue his work, when internet connection is available, but back-end server goes down. Technically user would remain online in this case, but in practice application will be broken - no response from back-end for REST calls. Offline persistence toolkit could help to solve such cases - user could continue working with local cache, until back-end is down.

If you want to learn how offline toolkit works and how to use its API, go to GitHub page - check readme, add it to your JET app and try to run/test. Hands-on is the best way to learn something new.

I will share few hints and sample app.

As per readme, first of all you should add Offline Persistence Toolkit and PouchDB modules from NPM. Run these commands within JET app directory:

1. npm install @oracle/offline-persistence-toolkit

2. npm install pouchdb pouchdb-find

Next you should follow four simple configuration steps and enable JET app to be able to access offline toolkit API.

Step 1 (standard, when adding any additional module)

Add paths to newly added modules in main.js require block:


Step 2 (standard, when adding any additional module)

Add paths to newly added modules in main-release-paths.js:


Step 3 (standard, when adding any additional module)

Added modules would not be copied to build directory automatically. We need to define copying in oraclejet-build.js. Modules should go to build directory. If you need to copy files from given folder and subfolders, use ** for src:


Build content is located in web directory. Offline toolkit and PouchDB modules should be copied to build directory:


Step 4

Initialize window.PouchDB variable in main.js:


Configuration is complete, now we can use Offline Persistence Toolkit API. Add persistence store manager and other modules:


Simplest option is to rely on default fetch listener from offline toolkit. We need to register store factory and map endpoint which we want to cache with persistence manager. When back-end is available - call will go to back-end and response will be cached. Next time, of back-end is not available - data will be fetched from cache. Toolkit intercepts HTTP(-S) request and stores response, if end-point was configured to be listened:


I'm testing offline toolkit with simple Employees REST end-point call from JET. Toolkit allows to execute this call successfully, even if there is no back-end or no connection (of course - if same call was executed at least once before):


UI part is simple - displaying list, when data is fetched:


Data is fetched, we are online:


Offline toolkit will work, if REST response doesn't include Cache-Control header. Make sure there is no Cache-Control header set in response:


ADF BC REST by default sets Cache-Control header, you can remove it Filter class (defined in ADF BC REST app):


Now I turned my connection to be offline, clicked on Get List button - JS calls REST and instead of getting network error, it executes successfully and returns data from cache through offline toolkit functionality:


You should open details for network call and check initiator. You will see that all calls mapped to offline endpoint are going through persistenceManager.js:


Let's double check - may be we are tricked somehow? Remove offline toolkit registration API in the code and re-run application:


As expected - network error is received and fetch fails. This proves - offline toolkit works :)


Sample JET application with offline toolkit configuration is available on GitHub (run ojet restore and ojet serve).

Linear Regression - Machine Learning with TensorFlow and Oracle JET UI Explained

Tue, 2018-01-16 11:33
Machine learning topic is definitely popular these days. Some get wrong assumptions about it - they think machine could learn by itself and its kind of magic. The truth is - there is no magic, but math behind it. Machine will learn the way math model is defined for learning process. In my opinion, the best solution is a combination of machine learning math and algorithms.  Here I could relate to chatbots keeping conversational context - language processing can be done by machine learning with neural network, while intent and context processing can be executed by programmable algorithms.

If you are starting to learn machine learning - there are two essential concepts to start with:

1. Regression
2. Classification

This post is focused around regression, in the next posts I will talk about classification.

Regression is a method which calculates the best fit for a curve to summarize data. Its up to you which type of curve to choose, you should assume which type will be most suitable (this can be achieved with trial and error too) based on given data set. Regression goal is to understand data points by discovering the curve that might have generated them.

In this example I will be using simplest regression possible - linear. Line is described by equation y = W*x + b. Where b is optional and can be 0 (line will cross (0, 0) point). For complex data sets, we might use polynomial equations and generate curves.

Here is Python code which implements linear regression with TensorFlow API (I have provided comments for all steps, reading code should be self explanatory):


Key element in any kind of machine learning - cost. The higher the cost, the worse is learning output. In linear regression, cost is typically defined by the sum of errors. The error in predicting x is calculated by the squared difference between the actual value f(x) and the predicted value M(w, x). The cost is the sum of squared differences between the actual and predicted values.

As you can see in the code above, we define cost function and ask TensorFlow to run optimizer to find the optimal values for model parameters. All the hard math calculation is happening in TensorFlow, our job is to prepare training data and choose right learning approach with correct equation.

Let's run JET UI, which talks to TensorFlow through REST. Training data is randomly generated (always 100 points) during each training session.

Training Epochs - number of learning iterations during training session
Learning rate - learning speed, smaller rate - more careful learning
W - learned model parameter to calculate equation y = W*x
Cost - value which shows how successful learning was, lower cost is better

1. We start from 1 training epoch and learning rate 0.001:


Learning result is not good - red line is result of linear regression, it doesn't represent best fit for training data. Cost is pretty high too, which indicates that learning wasn't successful.

2. 10 training epochs and learning rate 0.001:


As we repeat multiple learning iterations within the same training session - learning result is better. Cost becomes smaller and linear regression calculated line fits better, but still not ideal.

3. 100 training epochs and learning rate 0.001:


It helps to increase learning iterations, cost is significantly improving and line fits much better. This means outcome for W parameter learning is pretty good.

4. 1000 training epochs and learning rate 0.001



Let's make model to learn even harder and repeat more times - cost is becoming even better.

5. 2000 training epochs and learning rate 0.001


We could increase learning iterations further, but at some point it will not help. Learning process will start to suffer from overfitting. You can think about it - learning and repeating so many times, that at the end you start forgetting things. Cost is getting worse when repeating learning iterations further.

6. 2000 training epochs and learning rate 0.0001


It should help to make learning rate smaller, which result in more careful learning. This should allow to get better learning results with higher number of learning iterations. We get best learning cost result here and the most optimal line. You may ask - what is the use of that line? It can help to predict y values which were not available in training dataset.

7. 2000 training epochs and learning rate 0.01


On contrary if we increase learning rate, learning process will be faster - optimizer will run faster. This will result in decreased model output quality, cost will be higher and W parameter value will not produce such best fit line as in previous training run.

Few hints related to Oracle JET UI. You can achieve very good data visualization with JET chart components. For example I could control marker type rendered for training data points:


Line which represent learning result, can be displayed as reference line:


To display reference line, I'm using y-axis property which comes with JET chart:


References:

- Example for Linear Regression with Python and TensorFlow - Gist
- JET UI example - GitHub
- Accessing TensorFlow model through REST - Machine Learning with Oracle JET and TensorFlow
- Book - Machine Learning with TensorFlow 

JET UI Shell with Dynamic Tabs Implementation

Wed, 2017-12-27 08:47
I will describe and provide sample code - how to implement dynamic tabs in Oracle JET. This is useful for module loading, default JET template comes with static tabs - this works OK for the apps with smaller number of modules. When menu structure is more complex, it makes sense to load tabs for each module dynamically, when module is selected from the menu. This allows user to focus on modules he is working on, well just same as tabs in your Web browser.


Dynamic tabs are only visual placeholders, I'm using JET router to load module and keeping tab selection in synch. There are multiple benefits for such approach - module loading is following standard JET implementation with router, visually it looks like we have modules loaded using tabs. Overall implementation for dynamic UI Shell tabs with JET becomes quite simple, when relying on JET router.

In this post I will describe how to:

1. Add new tab and select module
2. Select existing tab from the menu/by clicking on the tab itself and select module
3. Remove tab and select next available module

Dashboard module is default one, it is loaded once application is accessed and we dont allow to remove it. Other modules are accessible from menu list:


Multiple menu items are opened with tabs displayed below. Let's see how tab removal works. If I remove Incidents tab:


Next tab on right will be selected as current:


If tab is removed, which is not selected -  Customers, current tab will remain selected:


If there is no tab on the right, while removing current tab About, first tab on the left will be selected:


Implementation part - menu list is implemented in this example with JET menu component (in real application you would use more complex menu structure, may be hierarchical):


Dynamic tabs solution is based on JET tab bar component, with removable option (you can try it in JET Cookbook - Add/Remove tab example):


Application defines array of tabs and variable to to keep track of current module:


Once menu item is loaded, we check first if same was opened already. If yes - we open it, instead of opening the same tab twice. Tab array is updated with new item and JET router navigates to current module, this is how module gets displayed, no additional magic:


When user selects tab, current module is loaded through JET router:


Tab removal is handled in delete method. Here calculation is done, which tab should be focused after current tab removal. Item is removed from array of tabs and JET router state is updated:


Sample JET code is available on my GitHub repository.

Recipe for JQuery Menu integration into ADF Faces

Thu, 2017-12-21 12:32
May be you are thinking to add some fresh looking element into your ADF app? I would propose JQuery menu. Advantage of such menu - it is purely client side implementation and it allows user to navigate through items really fast. Besides fast navigation, it looks great too and can be easily customized.

In this post I will describe how to achieve JQuery menu integration into ADF Faces and provide sample ADF application. I'm using JQuery menu from here - Animated Responsive Drop Down Menu. There is no need to implement JQuery menu component, we can use one of the existing implementations.

I have recorded short example, you can see how convenient is client side menu navigation. Each menu item loads ADF region:


JQuery menu integrated into my ADF sample:


It can display menu items for multiple levels:


Integration into ADF Faces is very simple, you need to copy JS and CSS files into ADF application:


Define JS and CSS resources in ADF main page, make sure to reference JQuery too:


Menu structure is defined in ADF fragment using HTML tags. Once menu item is selected, call to ADF is made through combination of client/server listeners (check WBClient.menuNavigation method). In my future posts I will describe how to construct similar menu structure dynamically. If your application is secured, make sure to define ADF Security permissions to TF's properly, any unprotected TF can be loaded from the menu:


Client listener method menuNavigation constructs params with target TF address and calls server listener, which in turn displays ADF dynamic region:


Menu fragment with HTML tags is integrated into ADF page template:


Download sample application - ADFJQueryMenuApp.zip.

Machine Learning with Oracle JET and TensorFlow

Sat, 2017-12-16 07:18
Oracle JET works with any kind of REST service, such service could be the one coming from TensorFlow (read more in my previous post - TensorFlow Linear Regression Model Access with Custom REST API using Flask). I have implemented linear regression model with gradient descent optimizer in TensorFlow (will describe this in more detail in my next post - machine learning is all about mathematics) and consuming it in JET UI:


There is option to define training steps (or data points) and learning rate. As outcome we get W and b values for linear equation y = Wx + b. After training is executed (so called machine learning process) - W and b parameters are identified, this allows to predict y value for any x. More about this in my next post, today will focus on JET.

To execute training, we call TensorFlow REST API endpoint and pass number of training steps/points and learning rate. I'm have create JET helper module to perform actual REST call, to separate logic from REST call invocation. Useful technique of custom callback is used to call method located in main module from helper module. We pass callback method name REST call helper function:


REST call is executed through JQuery and success callback is assigned with our method name (instead of processing results in REST call helper method, we can do processing in main module):


This is where results processing happens - in custom callback method. We prepare x/y training dataset to be displayed in chart. We get back estimated W and b parameters, this allows to calculate estimated y based on equation y = Wx + b for all x values used in training set (to compare how well model was trained):


One more nice component of Oracle JET - Form Layout. It makes very easy to create simple forms, specify number of max columns and you will get responsive UI for form components out of the box:


Make sure to add ojformlayout module:


To debug data returned by TensorFlow REST service, check Network history in browser developer tools:


By the way - it is completely up to you to decide about REST JSON data shape. With Python jsonify from Flask you can construct JSON from arrays of data and publish your own REST end point anywhere in Python script (you code in Python for TensorFlow):


Download sample JET application (make sure to use ojet restore, before ojet serve) from GitHub. TensorFlow linear regression model used for this sample is available as GitHubGist.

TensorFlow Linear Regression Model Access with Custom REST API using Flask

Mon, 2017-12-11 13:28
In my previous post - TensorFlow - Getting Started with Docker Container and Jupyter Notebook I have described basics about how to install and run TensorFlow using Docker. Today I will describe how to give access to the machine learning model from outside of TensorFlow with REST. This is particularly useful while building JS UIs on top of TensorFlow (for example with Oracle JET).

TensorFlow supports multiple languages, but most common one is Python. I have implemented linear regression model using Python and now would like to give access to this model from the outside. For this reason I'm going to use Flask, micro-framework for Python to allow simple REST annotations directly in Python code.

To install Flask, enter into TensorFlow container:

docker exec -it RedSamuraiTensorFlowUI bash

Run Flask install:

pip install flask

Run Flask CORS install:

pip install -U flask-cors

I'm going to call REST from JS, this means TensorFlow should support CORS, otherwise request will be blocked. No worries, we can import Flask CORS support.

REST is enabled for TensorFlow model with Flask in these two lines of code:


As soon as Flask is imported and enabled we can annotate a method with REST operations and endpoint URL:


There is option to check what kind of REST operation is executed and read input parameters from POST request. This is useful to control model learning steps, for example:


If we want to collect all x/y value pairs and return in REST response, we can do that by collecting values into array:


And construct JSON response directly out of the array structure using Flask jsonify:


After we run TensorFlow model in Jupyter, it will print URL endpoint for REST service. For URL to be accessible outside TensorFlow Docker container, make sure to run TensorFlow model with 0.0.0.0 as in the screenshot below:


Here is example of TensorFlow model REST call from Postman. POST operation is executed payload and response:


All REST calls are logged in TensorFlow:


Download TensorFlow model enabled with REST from my GitHub.

Pages