Andrejus Baranovski

Subscribe to Andrejus Baranovski feed
Blog about Oracle, Machine Learning and Cloud
Updated: 12 hours 28 min ago

ADF Performance Improvement with Nginx Compression

Fri, 2019-02-15 08:54
We are using Nginx web server for Oracle ADF WorkBetter hosted demo hosted on DigitalOcean cloud server. Nginx helps to serve web application content fast and offer improved performance. One of the important tuning options - content compression, Nginx does this job well and is simple to setup.

Content compression doesn't provide direct runtime performance, a browser would run the same code, doesn't matter it was compressed or not. But it brings improved perceived performance (which is very important), network time is way faster, because of reduced content size. Oracle ADF is a server-side framework, each request would bring content from the server - faster this content comes, means better application performance.

1. Content Compression = OFF

Let see stats, when no content compression applied (using our Oracle ADF WorkBetter hosted demo).

Page load size is 2.69 MB transferred. Finish time 1.55 s:


Navigation to the employee section generates 165.76 KB and finish time 924 ms:


Navigation to employee compensation generates 46.19 KB and finish time 494 ms:


2. Nginx compression

Compression is simple to setup in Nginx. Gzip settings are set in nginx.conf, make sure to list all content types which must be supported for compression. Restart nginx process after new settings are saved in nginx.conf:


3. Content Compression = ON

Page load size is 733.84 KB transferred. Finish time 1.48 s:


Navigation to the employee section generates 72.75 KB and finish time 917 ms:


Navigation to employee compensation generates 7.59 KB and finish time 498 ms:

Jupyter Notebook — Forget CSV, fetch data from DB with Python

Mon, 2019-02-11 14:11
If you read a book, article or blog about Machine Learning — high chances it will use training data from CSV file. Nothing wrong with CSV, but let’s think if it is really practical. Wouldn’t be better to read data directly from the DB? Often you can’t feed business data directly into ML training, it needs pre-processing — changing categorial data, calculating new data features, etc. Data preparation/transformation step can be done quite easily with SQL while fetching original business data. Another advantage of reading data directly from DB — when data changes, it is easier to automate ML model re-train process.

In this post I describe how to call Oracle DB from Jupyter notebook Python code.

Step 1 

Install cx_Oracle Python module:

python -m pip install cx_Oracle

This module helps to connect to Oracle DB from Python.

Step 2

cx_Oracle enables to execute SQL call from Python code. But to be able to call remote DB from Python script, we need to install and configure Oracle Instant Client on the machine where Python runs.

If you are using Ubuntu, install alien:

sudo apt-get update 
sudo apt-get install alien 

Download RPM files for Oracle Instant Client and install with alien:

alien -i oracle-instantclient18.3-basiclite-18.3.0.0.0–1.x86_64.rpm 
alien -i oracle-instantclient18.3-sqlplus-18.3.0.0.0–1.x86_64.rpm 
alien -i oracle-instantclient18.3-devel-18.3.0.0.0–1.x86_64.rpm 

Add environment variables:

export ORACLE_HOME=/usr/lib/oracle/18.3/client64 
export PATH=$PATH:$ORACLE_HOME/bin 

Read more here.

Step 3 

Install Magic SQL Python modules:

pip install jupyter-sql 
pip install ipython-sql 

Installation and configuration complete.

For today sample I’m using Pima Indians Diabetes Database. CSV data can be downloaded from here. I uploaded CSV data into the database table and will be fetching it through SQL directly in Jupyter notebook.

First of all, the connection is established to the DB and then SQL query is executed. Query result set is stored in a variable called result. Do you see %%sql — this magic SQL:


Username and password must be specified while establishing a connection. To avoid sharing a password, make sure to read password value from the external source (it could be simple JSON file as in this example or more advanced encoded token from keyring).

The beauty of this approach — data fetched through SQL query is out of the box available in Data Frame. Machine Learning engineer can work with the data in the same way as it would be loaded through CSV:

Sample Jupyter notebook available on GitHub. Sample credentials JSON file.

JDeveloper 12c IDE Performance Boost

Tue, 2019-02-05 02:14
There is a way to optimize JDeveloper 12c IDE performance by disabling some of the features you are not using.

I was positively surprised with improved JDeveloper responsiveness after turning off some of the features. ADF BC, Task Flow, and ADF Faces wizards started to respond in a noticeably faster way. Simple change and big performance gain, awesome.

One of the strongest JDeveloper performance improvements come from disabling TopLink feature. Ironically - TopLink is an abandoned product (12.1.3 was the last release). I remember back in 2006 TopLink was very promising and it was almost becoming the default platform for ADF Model. One of the old blog posts written by me related to TopLink - External Transaction Service in Oracle TopLink. But luckily it was overshadowed by ADF BC.

These are the features I disabled in my JDeveloper to get performance gain:

Cross Field Form Validation in Oracle JET

Mon, 2019-02-04 03:09
JET keeps evolving and in the latest versions  - toolkit provides improved support for form cross-field validation. It is much easier to implement validation than it was before. I will show it in this example.

Example of the data entry form. Validation logic:

- Invoice Date before Payment Due Date and Payment Date
- Payment Due Date before Payment Date


Example when two fields fail validation:


JET provides component called validation group. Form can be wrapped by this component to identify if any validation errors are reported there. For example, when calling JS function, before proceeding with the function code - we can check if validation group contains errors:


Input field can be assigned with custom validator function:


Example of validation function code where cross-field validation logic is implemented - we compare field value with other fields. If validation rule condition is false - validation error is thrown:


Example of function code, where validation group is checked for errors. If there are errors in the current validation group - errors are displayed and the first field with error is focused:


Download sample code from my GitHub repo.

Search Form in Oracle Visual Builder based on ADF BC REST

Sat, 2019-01-26 05:14
Oracle Visual Builder supports ADF BC REST out of the box. Build service connection using "Define by Specification" wizard:


Wizards support ADF as API type. Add describe at the end of the REST URL, this will bring metadata for exposed ADF BC REST service (information about attribute types, etc.):


List of endpoints will be populated automatically. You could select all endpoints to be supported for your connection or select only few:


The most typical thing you would do with endpoint - map it with the table to display collection data. You would drag and drop Oracle JET table into VBCS page and choose Add Data option to map it with the service connection:


In the wizard you would select previously defined service connection:


There is a way to switch wizard to detailed view and choose from multiple endpoints available for the connection:


In the next step, you would select service attributes to be displayed in table columns. All declarative, sweet:


In Visual Builder at any point you can quickly test application, it will load in separate browser tab (or you could switch app to Live mode and test page functionality directly in VBCS window):


Every action in Visual Builder is handled through events. For example, this event is mapped with Reset button (you can see it in structure tab on the left):


At any point, you can switch to source view and check (or edit) HTML/JET code which is generated for you by Visual Builder. So cool, imagine typing and copy-pasting all this text by hand, tough and time-consuming (you could do better things in your life than copy-pasting HTML code):


Let's explain how search form logic is done in this sample. I have defined page scope variable type, this type would hold search attribute name, type and operation:


Create as many variables based on this type, as many search criteria items you will expect to have. Make sure to provide attribute and operation names (leave value property empty, this will be assigned by user):


Map search form fields with variables:


Create an event for Search button, which calls search action chain:


In action chain we can define search logic. Before executing search criteria, we need to prepare search criteria array (normally this step could be skipped, but there is issue in current Visual Builder, it fails to execute criteria search, when at least one of the criteria items empty). Calling custom JavaScript function where search criteria array will be prepared:


Custom JavaScript function, it helps to prepare array to be based to criteria (if search item is not set, we are assigning empty value):


Result of the function is mapped with service connection criteria, search will be executed automatically:


Table pagination is handled automatically too. Make sure to specify scroll policy = loadMoreOnScroll and define fetch size:


Resources:

1. Sample source code on my GitHub
2. Blog from Shay - Filtering Data Providers with Compound Conditions in Visual Builder
3. Blog from Shay - Oracle JET UI on Top of Oracle ADF With Visual Builder
4. My previous post about query logic in Visual Builder - Query Logic Implementation in VBCS for ADF BC REST

Announcing Hosting for Oracle ADF Rich Client and Oracle ADF WorkBetter Demos

Mon, 2019-01-21 03:03
If you are curious about how Oracle ADF works or want to explore a rich set of ADF Faces components - welcome to access Oracle ADF demo apps hosted on our cloud server.

We launched a dedicated website Oracle ADF Components. Hosted demos:

1. ADF Faces Rich Client
2. ADF Work Better


These demo apps can be downloaded from Oracle, you could run them on your own environment too. But sometimes it is useful to have apps online for quick access.

Oracle ADF BC Reusing SQL from Statement Cache

Sat, 2019-01-19 12:14
Oracle ADF BC by default is trying to reuse prepared SQL query from statement cache. It works this way when ADF BC runs with DB pooling off (jbo.doconnectionpooling=false). Normally we tune ADF application to run with DB pooling on (jbo.doconnectionpooling=true), this allows to release unused DB connection back to the pool when a request is completed (and in this case, statement cache will not be used anyway). If View Object is re-executed multiple times during the same request - in this situation, it will use statement cache too.

However, there are cases when for specific View Object you would want to turn off statement cache usage. There could be multiple reasons for this - for example, you are getting Closed Statement error after it tries to execute SQL for statement obtained from statement cache. Normally you would be fine using statement cache, but as I said - there are those special cases.

We are lucky because there is a way to override statement cache usage behavior. This can be done in View Object implementation class either for particular View Object or in the generic class.

After View Object was executed, check the log. If this is not the first execution, you will see log message - "reusing defined prepared statement". This means SQL will be reused from statement cache:


To control this behavior, override getPreparedStatement method:


We create new prepared statement in this method, instead of reusing one from the cache.

As a result - each time View Object is executed, there is no statement cache usage:


Download sample application from GitHub repo.

On-Premise Machine Learning with XGBoost (Katana 19.1)

Sat, 2019-01-12 05:14
Happy to announce Katana 19.1 release with complete on-premise support for Machine Learning.



You can run Machine Learning (ML) models on Cloud (Amazon SageMaker, Google Cloud Machine Learning, etc.). I believe it is important to understand how to run Machine Learning in your own environment too. Without this knowledge ML skills set would not be complete. There are multiple reasons for this. Not everyone is using Cloud and you must provide on-premise solution. Without getting your hands dirty and configuring environment yourself, you would miss an exciting opportunity to learn more about ML.

Read more here.

Oracle Visual Builder 18.4.5 and JET 6 Support

Tue, 2019-01-08 10:12
Oracle Visual Builder 18.4.5 comes with very neat and polished UI. Also it brings Oracle JET 6 support (latest JET version to date). Read more about it - New Features in Oracle Visual Builder December Release.

I have upgraded our VBCS instance to 18.4.5:


I was curious how automatic upgrade would work for VBCS app implemented in the previous version (download source code for the upgraded app from my GitHub repo). Especially that now VBCS is using newer JET too. I must say I was pleased with the results - application was upgraded to JET 6 automatically without manual interference:


I did a quick check in the source on runtime - indeed our upgraded VBCS app is using JET 6:


Awesome work by VBCS team.

Knockout.js - Updating Single Array Element (Oracle JET)

Thu, 2018-12-27 02:40
If you implement tables and using Knockout.js to push data updates from JS to HTML - probably you experience a situation when it doesn't work to push an update for one of the columns. I mean you could replace the whole observable array element - this would cause full row refresh. But visually this doesn't look nice and why to refresh the whole row, if only one (or few) element (-s) from the row must be refreshed.

If you need to refresh a specific array element (or row column in other words) - you must define the value of that column to be observable.

Refresh will be happening much more smooth, instead of refreshing whole row. See how fast Risk column value is changed after clicking on Process button:


Table is implemented with Oracle JET table component. JET table allows to define template slots, this helps to create a better structure for table columns implementation:


Risk column - the one which is being refreshed is defined as an observable variable in the array:


A new value for Risk column is set directly - by iterating array elements. Refresh on UI happens automatically, through Knockout observable:


Sample application source code is available on my GitHub repo.

Tweet Escalation to Your Support Team — Sentiment Analysis with Machine Learning

Mon, 2018-12-24 03:06
I have published an article on Towards Data Science. I explain end-to-end technical solution which would help to streamline your company support process. With the focus on airline support requests received from Twitter. It could save a lot of time and money for the support department if they would know in advance which request is more critical and must be handled with higher priority.

Read the full article here - Solution to automate tweet sentiment processing for airline support request escalation.


Understanding Attributes Enum in ADF BC Row Class

Sun, 2018-12-23 03:59
Did you ever wonder why Attributes Enum is generated by JDeveloper in Entity or View Row class? Attributes Enum holds a collection of attribute names and there is a set of static variables with attribute indexes. These indexes are used to locate attribute in getter/setter. Attributes Enum is a structure which is required for JDeveloper on design time to generate Java code. On runtime Attributes Enum is needed only as long as you are using a static variable index in the getter/setter.

Attributes Enum and list of static indexes in View Row class:


Static index is used in the getter/setter to access attribute:


Attributes Enum is mimicking attributes order in the VO/EO. You can think about it as about attributes metadata. It is not mandatory to use index from Attributes Enum. In some use cases, you could get attribute index directly from VO/EO Def and use it to access attribute:


First name is fetched correctly using overridden getter:


Download sample code from GitHub

Off Canvas Menu in Oracle VBCS/JET Cloud

Sat, 2018-12-15 13:59
These days I'm actively working with VBCS/JET Cloud product from Oracle. The more I work with VBCS the more I like it. VBCS follows similar declarative development concepts as Oracle ADF, this makes it easy to get up to speed with VBCS development. VBCS with declarative JavaScript development approach brings unique solution for JavaScript systems implementation for enterprise.

I will share sample with off canvas menu implementation for VBCS app. Sample is based on step by step guide shared by Shay Shmeltzer. I don't describe steps how to build off canvas in VBCS from scratch, you should watch Shay's video for the instructions.

Off canvas menu rendered in VBCS app:


You should check how to build multiple flows in VBCS app in my previous post - Flow Navigation Menu Control in Oracle VBCS. I have defined three flows in my sample, this means there will be three menu items:


To render menu in off canvas block, I'm using JET navigation list component:


Sample app code which can be imported into your VBCS instance is available on GitHub.

Date Format Handling in Oracle JET

Tue, 2018-12-11 14:58
Oracle JET comes with out of the box support for date converter, check more about it in cookbook - Date Converter. This makes it very handy to format dates in JavaScript. Here is date picker field example with yyyy-MM-dd format applied:


When button Process is pressed, I take date value from date picker and add one day - result is printed in the log. This is just to test simple date operation in JavaScript.

Date picker is defined by JET tag. Format is assigned through converter property:


Current date is displayed from observable variable. This variable is initialized from current date converted to local ISO. Converter is configured with pattern. In the JS method, where tomorrow date is calculated - make sure to convert from ISO local date:


Hope this simple example helps you to work with dates in Oracle JET application. Source code is available on my GitHub directory.

API for Amazon SageMaker ML Sentiment Analysis

Thu, 2018-12-06 13:50
Assume you manage support department and want to automate some of the workload which comes from users requesting support through Twitter. Probably you already would be using chatbot to send back replies to users. Bu this is not enough - some of the support requests must be taken with special care and handled by humans. How to understand when tweet message should be escalated and when no? Machine Learning for Business book got an answer. I recommend to read this book, my today post is based on Chapter 4.

You can download source code for Chapter 4 from book website. Model is trained based on sample dataset from Kaggle - Customer Support on Twitter. Model is trained based on subset of available data, using around 500 000 Twitter messages. Book authors converted and prepared dataset to be suitable to feed into Amazon SageMaker (dataset can be downloaded together with the source code).

Model is trained in such way, that it doesn't check if tweet is simply positive or negative. Sentiment analysis is based on the fact if tweet should be escalated or not. It could be even positive tweet should be escalated.

I have followed instructions from the book and was able to train and host the model. I have created AWS Lambda function and API Gateway to be able to call model from the outside (this part is not described in the book, but you can check my previous post to get more info about it - Amazon SageMaker Model Endpoint Access from Oracle JET).

To test trained model, I took two random tweets addressed to Lufthansa account and passed them to predict function. I exposed model through AWS Lambda function and created API Gateway, this allows to initiate REST request from such tool as Postman. Response with __label__1 needs esacalation and __label__0 doesn't need. Second tweet is more direct and it refers immediate feedback, it was labeled for escalation by our model for sentiment analysis. First tweet is a bit abstract, for this tweet no escalation:


This is AWS Lambda function, it gets data from request, calls model endpoint and returns back prediction:

Let's have a quick look into training dataset. There are around 20% of tweets representing tweets marked for escalation. This shows - there is no need to have 50%/50% split in training dataset. In real life probably number of escalations is less than half of all requests, this realistic scenario is represented in the dataset:


ML model is built using Amazon SageMaker BlazingText algorithm:


Once ML model is built, we deploy it to the endpoint. Predict function is invoked through the endpoint:

Machine Learning - Date Feature Transformation Explained

Sat, 2018-12-01 09:46
Machine Learning is all about data. The way how you transform and feed data into ML algorithm - greatly depends training success. I will give you an example based on date type data. I will be using scenario described in my previous post - Machine Learning - Getting Data Into Right Shape. This scenario is focused around invoice risk, ML trains to recognize when invoice payment is at risk.

One of the key attributes in invoice data are dates - invoice date, payment due date and payment date. ML algorithm expects number as training feature, it can't operate with literals or dates. This is when data transformation comes in - out of original data we need to prepare data which can be understood by ML.

How we can transform dates into numbers? One of the ways is to split date value into multiple columns with numbers describing original date (year, quarter, month, week, day of year, day of month, day of week). This might work? To be sure - we need to run training and validate training success.

Resources:

1. Sample Jupyter notebooks and datasets are available on my GitHub repo
2. I would recommend to read this book - Machine Learning for Business

Two approaches:

1. Date feature transformation into multiple attributes

Example where date is split into multiple columns:


Correlation between decision column and features show many dependencies, but it doesn't pick up all columns for payment date feature. This is early sign training might not work well:


We need to create test (1/3 of remaining data), validation (2/3 of remaining data) and training (70% of all data) datasets to be able to train, validate and test ML model. Splitting original dataset into three parts:


Running training using XGBoost (Gradient boosting is currently one of the most popular techniques for efficient modeling of tabular datasets of all sizes). Read more about XGBoost parameters. We have validation dataset and this allows to use XGBoost early stopping functionality, if training quality would not improve in N (10 in our case) rounds - it will stop and pick best iteration as the one to be used for training result:


Result: training accuracy 93% and validation accuracy 74%. Validation accuracy is too low, this means training wasn't successful and we should try to transform dates in another way:


2. Date feature transformation into difference between dates

Instead of splitting date into multiple attributes, we should reduce number of attributes to two. We can use date difference as such:

- Day difference between Payment Due Date and Invoice Date
- Day difference between Payment Date and Invoice Date

This should bring clear pattern, when there is payment delay - difference between payment date/invoice date will be bigger than between payment date/invoice date. Sample data with date feature transformed into date difference:


Correlation is much better this time. Decision correlates well with date differences and total:


Test, validation and training data sets will be prepared in the same proportions as in previous test. But we will be using stratify option. This option helps to shuffle data and create test, validation and training data sets where decision attribute is well represented:


Training, validation and test datasets are prepared:


Using same XGBoost training parameters:


Result: This time we get 99% training accuracy and 97% validation accuracy. Great result. You can see how important is data preparation step for ML. It directly relates to ML training quality:

Notification Messages in Oracle JET

Wed, 2018-11-28 14:52
Let's take a look into cool component available in Oracle JET - notification messages (it is possible to display messages in different ways - inline or overlay. Check more about messages functionality in JET Cookbook example).

This is how notifications messages are showing up, very cool way to send information to the user:


Messages are implemented with oj-messages components. This component accepts observable array of messages to be displayed. We can specify how message is displayed (notification in this case), position information and close listener (where we can remove message info entry from messages array):


In sample processAction function I'm pushing new entries into messages array. To simulate delay for second message, pushing second entry with 1 second delay. Once message is closed after standard delay time - function closeMessageHandler is invoked, where we are removing entry from array:


Sample application code is available on my GitHub repo.

Our new product - Katana 18.1 (Machine Learning for Business Automation)

Mon, 2018-11-26 04:47
Big day. We announce our brand new product - Katana. Today is first release, which is called 18.1. While working with many enterprise customers we saw a need for a product which would help to integrate machine learning into business applications in more seamless and flexible way. Primary area for machine learning application in enterprise - business automation.


Katana offers and will continue to evolve in the following areas:

1. Collection of machine learning models tailored for business automation. This is the core part of Katana. Machine learning models can run on Cloud (AWS SageMaker, Google Cloud Machine Learning, Oracle Cloud, Azure) or on Docker container deployed On-Premise. Main focus is towards business automation with machine learning, including automation for business rules and processes. Goal is to reduce repetitive labor time and simplify complex, redundant business rules maintenance

2. API layer built to help to transform business data into the format which can be passed to machine learning model. This part provides API to simplify machine learning model usage in customer business applications

3. Monitoring UI designed to display various statistics related to machine learning model usage by customer business applications. UI which helps to transform business data to machine learning format is also implemented in this part

Katana architecture:


One of the business use cases, where we are using Katana - invoice payment risk calculation. UI which is calling Katana machine learning API to identify if invoice payment is at risk:


Get in touch for more information.

Oracle ADF + Jasper Visualize.js = Awesome

Sun, 2018-11-25 02:36
This week I was working on a task to integrate Jasper Visualize.js into Oracle ADF application JSF page fragment. I must say integration was successful and Jasper report renders very well in Oracle ADF screen with the help of Visualize.js. Great thing about Visualize.js - it renders report in ADF page through client side HTML/JS, there is no iFrame. Report HTML structure is included into HTML generated by ADF, this allows to use CSS to control report size and make it responsive.

To prove integration, I was using ADF application with multiple regions - ADF Multi Task Flow Binding and Tab Order. Each region is loaded with ADF Faces tab:


One of the tabs display region with Jasper report, rendered with Visualize.js:


Check client side generated code. You should see HTML from Visualize.js inside ADF generated HTML structure:


It is straightforward to render Jasper report with Visualize.js in Oracle ADF. Add JS resource reference to Visualize.js library, define DIV where report supposed to be rendered. Add Visualize.js function to render report from certain path, etc.:


Sample code is available on my GitHub repo.

Amazon SageMaker Model Endpoint Access from Oracle JET

Tue, 2018-11-13 10:54
If you are implementing machine learning model with Amazon SageMaker, obviously you would want to know how to access trained model from the outside. There is good article posted on AWS Machine Learning Blog related to this topic - Call an Amazon SageMaker model endpoint using Amazon API Gateway and AWS Lambda. I went through described steps and implemented REST API for my own module. I went one step further and tested API call from JavaScript application implemented with Oracle JET JavaScript free and open source toolkit.

I will not go deep into machine learning part in this post. I will focus exclusively on AWS SageMaker endpoint. I'm using Jupyter notebook from Chapter 2 of this book - Machine Learning for Business. At the end of the notebook, when machine learning model is created, we initialize AWS endpoint (name: order-approval). Think about it as about some sort of access point. Through this endpoint we can call prediction function:


Wait around 5 minutes until endpoint starts. Then you should see endpoint entry in SageMaker:


How to expose endpoint to be accessible outside? Through AWS Lambda and AWS API Gateway.

AWS Lambda

Go to AWS Lambda service and create new function. I already have function, with Python 3.6 set for runtime. AWS Lambda acts as proxy function between endpoint and API. This is the place where we can prepare input data and parse response, before returning it to API:


Function must be granted role to access SageMaker resources:


This is function implementation. Endpoint name is moved out into environment variable. Function gets input, calls SageMaker endpoint and does some minimal processing for the response:


We can test lambda function and provide test payload. This is test payload I'm using. This is encoded list of parameters for machine learning model. Parameters describe purchase order. Model decides if manual approval is required or not. Decision rule - if PO was raised by someone not from IT, but they order IT product - manual approval is required. Read more about it in the book mentioned above. Test payload data:


Run test execution, model responds - manual approval for PO is required:


AWS API Gateway

Final step is to define API Gateway. Client will be calling Lambda function through API:


I have defined REST resource and POST method for API gateway. Client request will go through API call and then will be directed to Lambda function, which will make call for SageMaker prediction based on client input data:


POST method is set to call Lambda function (function with this name was created above):


Once API is deployed, we get URL. Make sure to add REST resource name at the end. From Oracle JET we can use simple JQuery call to execute POST method. Once asynchronous response is received, we display notification message:


Oracle JET displays prediction received from SageMaker - manual review is required for current PO:


Download Oracle JET sample application with AWS SageMaker API call from my GitHub repo.

Pages