Skip navigation.

Steve Karam

Syndicate content
Steve Karam's Oracle Blog
Updated: 8 hours 55 min ago

Real Push Button Refresh with Raspberry Pi

Sun, 2015-04-19 16:19
Push Button

In this post, we’re going to perform a push button refresh of an Oracle Database, Application Express (APEX) installation, and Tomcat webserver.

“But Oracle Alchemist,” you’re probably thinking, “we know about that. You’ve told us about how Delphix can provision and refresh data.” And yes, you’d be right. But I wasn’t done yet.

We’re going to perform a refresh of an Oracle Database, APEX installation, and Tomcat running in Amazon Web Services, replicated from a local Delphix Engine, by pressing a physical button wired to a Raspberry Pi running a python app that communicates with the Delphix REST API in the cloud over wifi.

We’ll go over technical details and the Python code right after the video. Make sure you check it out so you can watch me excitedly press the button. I even did a refresh from Starbucks.

I Like Pi

I’ve wanted a Raspberry Pi for a long time now, and for whatever reason never got around to buying one. I finally did last week, and let me tell you this thing is such a beautiful little device, I nearly fried the logic board by weRaspberry Pi B+eping on it. The components, how tiny! The GPIO pins, how enticing! The Raspberry Pi really is the ultimate geek toy. I ended up going with the Raspberry Pi Model B. It has a 40 pin GPIO header (an I/O interface we’ll use for this article), four USB 2.0 ports, a push-button micro SD slot for the hard drive, and the standard HDMI port, ethernet port, and power via micro USB. I can power it using my iPhone charging block or even with a USB battery pack. The one I bought also came with NOOBS on an 8GB SD card, which was preloaded with Raspbian, ArchLinux, OpenELEC, Pidora, RaspBMC, and RiscOS. I opted for Raspbian.

Zebra CaseAs for extras, I also got a super sexy little case called a Zebra Case. It’s designed and built right here in the good ol’ US of A, is incredibly easy to assemble, has optional rubber feet, can be hung up on a wall, and has easy access to the GPIO pins. I also picked up an Edimax EW-7811Un 150Mbps 11n Wi-Fi USB Adapter because I don’t like being tied down.

Let’s Communicate

I have Delphix 4.2 (the latest version) set up on my laptop with a bunch of data sources: Oracle, an APEX app in Tomcat, Sybase, Postgres, MS SQL Server, and a Delphix Agile Masking repository. I also have Delphix 4.2 installed in Amazon Web Services and am replicating the Oracle Database and application stack to it via Delphix replication. In Amazon, the database, Tomcat, and APEX library replicas have all been provisioned to a target Linux system as Virtual Databases (VDBs) and Virtual Files (vFiles). Delphix has a powerful GUI that can handle refresh, rewind, etc. operations but instead we’re going to do it programmatically through the Delphix REST API in Python. Why? Because alchemy, that’s why. Raspbian has Python already loaded with the GPIO library built in. So let’s take a look at the code.

Here’s the delphix.py code that connects to my Amazon Delphix Engine and performs the operations.

import urllib2
import simplejson as json
from cookielib import CookieJar

# VDBControl accepts op = [ refresh | undo ]

def VDBControl(op):

  # Delphix Details
  url = "http://ec2-52-1-228-37.compute-1.amazonaws.com"
  username = "delphix_admin"
  password = "delphix"

  # urllib2 setup
  headers = {"Content-Type" : "application/json"}
  cj = CookieJar()
  opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

  # Get Delphix Session
  data = { "type" : "APISession", "version" : { "type" : "APIVersion", "major" : 1, "minor" : 1, "micro" : 0 } }
  opener.open(urllib2.Request(url + "/resources/json/delphix/session", data=json.dumps(data), headers=headers))

  # Login to Delphix
  data = { "type" : "LoginRequest", "username" : "delphix_admin", "password" : "delphix" }
  opener.open(urllib2.Request(url + "/resources/json/delphix/login", data=json.dumps(data), headers=headers))

  # Get Delphix Objects
  request = opener.open(urllib2.Request(url + "/resources/json/delphix/source", headers=headers))
  content = request.read()
  sourcelist = json.loads(content)

  # Loop through Delphix Objects

  for source in sourcelist["result"]:
    if source["virtual"] == True:
      # get container details including name and type
      virtualContainer = source["container"]
      virtualName = source["name"]
      request = opener.open(urllib2.Request(url + "/resources/json/delphix/database/" + virtualContainer, headers=headers))
      content = request.read()
      parentContainer = json.loads(content)["result"]["parentContainer"]
      objType = json.loads(content)["result"]["type"]
      refreshType = "OracleRefreshParameters" if objType == "OracleDatabaseContainer" else "RefreshParameters"

      if op == "refresh":
        # refresh each virtual object
        print "Refreshing " + virtualName + "..."
        data = { "type" : refreshType, "timeflowPointParameters" : { "type" : "TimeflowPointSemantic", "container" : parentContainer } }
        request = opener.open(urllib2.Request(url + "/resources/json/delphix/database/" + virtualContainer + "/refresh", data=json.dumps(data), headers=headers))
        content = request.read()
        print "Output: " + content
        print "Refresh Job Submitted."
      if op == "undo":
        print "Undoing Refresh of " + virtualName + " (if applicable)..."
        request = opener.open(urllib2.Request(url + "/resources/json/delphix/database/" + virtualContainer + "/undo", headers=headers))
        content = request.read()
        print "Output: " + content
        print "Undo Job Submitted."

In a nutshell, what this code does is:

  • Define a VDBControl function that accepts an operation, either “refresh” or “undo”
  • Declares variables for the Delphix Engine and the Python urllib2 classes
  • Get a Delphix session via the API and put a cookie in the cookie jar
  • Log into Delphix
  • Get a list of Delphix objects with a REST call to /resources/json/delphix/source on the Delphix Engine
  • Loop through the list and find the objects that are either Virtual Databases or Virtual Files
  • Grab some auxiliary data we need to do the refresh, like the source object we’re refreshing from.
  • If a refresh is being called, do the refresh by passing the proper data to Delphix using the /resources/json/delphix/database/VDBNAME/refresh REST method.
  • If an undo is being called, do the undo by passing the proper data using the undo method.

Now that we’ve seen the code that interfaces with Delphix, let’s take a look at the code that interfaces with the GPIO.

import RPi.GPIO as GPIO
import time
from delphix import VDBControl

GPIO.setmode(GPIO.BCM)
GPIO.setup(18, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(24, GPIO.IN, pull_up_down=GPIO.PUD_UP)

while True:
	input_state18 = GPIO.input(18)
	if input_state18 == False:
                VDBControl("refresh")
		time.sleep(0.2)

	input_state24 = GPIO.input(24)
	if input_state24 == False:
                VDBControl("undo")
		time.sleep(0.2)

Pi and BreadboardThis code must be run as root on the Raspberry Pi to interface with the GPIO. I actually have it appended in my rc.local so it starts up automatically when the Raspberry Pi comes online. The code is really quite simple as long as everything is plugged in properly. On my breadboard I have two buttons, each slotted in across the middle gap. The gap is there to divide up the breadboard so circuits can be built from both sides while only certain designated components “bridge” the gap. Two jumper cables connect each button to the Raspberry Pi GPIO. My refresh button has a positive connection to GPIO pin 18, and a ground connection. The undo button has a positive connection to GPIO pin 24, and a ground connection. Then the python code simply:

  • Sets up pins 18 and 24 for monitoring
  • Runs an endless loop
  • If GPIO pin 18 is clicked, the state changes to False, triggering a call to my delphix.VDBControl function with the “refresh” operation. It then sleeps for a brief moment to make sure the button press doesn’t rapid fire a bunch of presses.
  • If GPIO pin 24 is clicked, the same thing happens except an “undo” operation is passed.

And that’s it! A real, honest to goodness push-button refresh, courtesy of the cutest little computer I’ve ever seen and a hobby board, wires, and buttons that made me feel nervous carrying it around in public. It wouldn’t take much to make it presentable though. In fact, given the right top and bottom coverings, we could make quite the button. That was easy.®

So, what could you do with a Raspberry Pi and a couple buttons? Let me know in the comments!

Thank you to Simon Monk and the Raspberry Pi Cookbook for the button guide!

The post Real Push Button Refresh with Raspberry Pi appeared first on Oracle Alchemist.

Deploying Application Express with Delphix

Sat, 2015-01-24 14:31
VDBs

Seamless cloning of an application stack is an outstanding goal. Seamless cloning of an application stack including the full production database, application server, and webserver in a few minutes with next to zero disk space used or configuration required is the best goal since Alexander Graham Bell decided he wanted a better way tell Mr. Watson to “come here.”

So in the spirit of discovery, I’ve installed Oracle REST Data Services (ORDS) 2.0 and Oracle Application Express (APEX) 4.2 to a source Oracle database environment in my home Delphix setup. I’m going to:

  1. Sync the ORDS binaries with Delphix as a file source
  2. Sync the APEX binaries with Delphix as a file source
  3. Sync the ORCL database with Delphix as a database source
  4. Provision a clone of the ORCL database to a target linux system as DBDEV
  5. Provision a clone of the ORDS and APEX binaries to the target system

Some of you may be scratching your head right now thinking “What is Delphix?” I’ve written a few words on it in the past, and Kyle Hailey has quite a bit of information about it along with other links such as Jonathan Lewis explaining Delphix at OOW14.

If you’re into the whole brevity thing, here’s a short summation: Delphix is a technology you can sync nearly any kind of source data into and provision on demand from any point in time to any target, near instantly and at the click of a button, all without incurring additional disk space. What that means for your business is incredibly efficient development, faster time to market, and improved application quality. And if you want to see this in action, you can try it for yourself with Delphix Developer Edition.

Let’s use Delphix to deploy APEX to a target system.

Step 1. A look at the source

On the source environment (linuxsource, 172.16.180.11) I have an 11.2.0.1 database called “orcl”.

ORCL Source Database

In the /u01/app/oracle/product directory are ./apex and ./ords, holding the APEX and ORDS installations respectively.

Source Products Directory

When ORDS is started, I am able to see the APEX magic by browsing to http://172.16.180.11:8080/apex and logging in to my InvestPLUS workspace. Here’s the pre-packaged apps I have installed:

Source System APEX Apps

Sweet. Let’s check out what I have set up in Delphix.

Step 2. Check out the Delphix Sources

You can see that I have the ORCL database (named InvestPLUS DB Prod), Oracle REST Data Services, and APEX homes all loaded into Delphix here:

Delphix Sources

When I say they’re loaded into Delphix, I mean they’ve been synced. The ORCL database is synced over time with RMAN and archive logs and compressed about 3x on the base snapshot and 60x on the incremental changes. The /u01/app/oracle/product/apex and /u01/app/oracle/product/ords directories have also been synced with Delphix and are kept up to date over time. From these synced copies we can provision one or more Virtual Databases (VDBs) or Virtual Files (vFiles) to any target we choose.

Step 3. Deploy

Provisioning both VDBs and vFiles is very quick with Delphix and takes only a few button clicks. Just check out my awesomely dramatized video of the provisioning process. For this demo, first I provisioned a clone of the ORCL database to linuxtarget (172.16.180.12) with the name DBDEV.

Provisioning DBDEV to the target

Next I provisioned a copy of the ORDS home to the target at the same location as the source (/u01/app/oracle/product/ords) with the name ORDS Dev:

ORDS Dev on the target

And lastly I provisioned a copy of the APEX home to the target at the same location as the source (/u01/app/oracle/product/apex) with the name APEX Dev:

APEX Dev on target

In hindsight I probably could have just synced /u01/app/oracle/product and excluded the ./11.2.0 directory to get both ORDS and APEX, but hey, I like modularity. By having them separately synced, I can rewind or refresh either one on my target system.

Here’s the final provisioned set of clones on the target (you can see them under the “InvestPLUS Dev/QA” group on the left nav):

Provisioned Clones

Step 4. Check out the target system

Let’s see what all this looks like on the target system. Looking at the /u01/app/oracle/product directory on the target shows us the same directories as the source:

Target directories

I’ve also got the DBDEV database up on the target:

DBDEV on the target

To give you a glimpse of how Delphix provisioned the clone, check this out. Here’s a “df -h” on the linuxtarget environment:

Linux Target df command

What this is showing us is that the APEX Home, ORDS Home, and DBDEV clone are all being served over NFS from Delphix (172.16.180.3). This is how Delphix performs a clone operation, and why we call it virtual: data is synced and compressed from sources into Delphix, and when you provision a clone Delphix creates virtual sets of files that are presented over the wire to the target system. You can think of Delphix as a backup destination for source databases/filesystems, and as network attached storage for targets. The clever bit is that Delphix uses the same storage for both purposes, with no block copies at all unless data is changed on the target VDBs or vFiles. Cool, right? On a side note and for the curious, Delphix can use dNFS as well for your Oracle VDBs.

Step 5. Reconfigure ORDS

On the source environment, ORDS is configured to connect to the ORCL database. On the target we’re going to the DBDEV database. So the one quick change we’ll need to make is to change the SID in the /u01/app/oracle/product/ords/config/apex/defaults.xml file.

[delphix@linuxtarget ords]$ vi config/apex/defaults.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>Saved on Wed Jan 14 08:38:04 EST 2015</comment>
<entry key="cache.caching">false</entry>
<entry key="cache.directory">/tmp/apex/cache</entry>
<entry key="cache.duration">days</entry>
<entry key="cache.expiration">7</entry>
<entry key="cache.maxEntries">500</entry>
<entry key="cache.monitorInterval">60</entry>
<entry key="cache.procedureNameList"/>
<entry key="cache.type">lru</entry>
<entry key="db.hostname">localhost</entry>
<entry key="db.password">@050784E0F3307C86A62BF4C58EE984BC49</entry>
<entry key="db.port">1521</entry>
<entry key="db.sid">DBDEV</entry>
<entry key="debug.debugger">false</entry>
<entry key="debug.printDebugToScreen">false</entry>
<entry key="error.keepErrorMessages">true</entry>
<entry key="error.maxEntries">50</entry>
<entry key="jdbc.DriverType">thin</entry>
<entry key="jdbc.InactivityTimeout">1800</entry>
<entry key="jdbc.InitialLimit">3</entry>
<entry key="jdbc.MaxConnectionReuseCount">1000</entry>
<entry key="jdbc.MaxLimit">10</entry>
<entry key="jdbc.MaxStatementsLimit">10</entry>
<entry key="jdbc.MinLimit">1</entry>
<entry key="jdbc.statementTimeout">900</entry>
<entry key="log.logging">false</entry>
<entry key="log.maxEntries">50</entry>
<entry key="misc.compress"/>
<entry key="misc.defaultPage">apex</entry>
<entry key="security.disableDefaultExclusionList">false</entry>
<entry key="security.maxEntries">2000</entry>
</properties>

Note the only line I had to change was this one: <entry key=”db.sid”>DBDEV</entry>

After the config change, I just had to start ORDS on the target:

[delphix@linuxtarget ords]$ java -jar apex.war
Jan 21, 2015 1:18:22 PM oracle.dbtools.standalone.Standalone execute
INFO: NOTE:

Standalone mode is designed for use in development and test environments. It is not supported for use in production environments.

Jan 21, 2015 1:18:22 PM oracle.dbtools.standalone.Standalone execute
INFO: Starting standalone Web Container in: /u01/app/oracle/product/ords/config/apex
Jan 21, 2015 1:18:22 PM oracle.dbtools.standalone.Deployer deploy
INFO: Will deploy application path = /u01/app/oracle/product/ords/config/apex/apex/WEB-INF/web.xml
Jan 21, 2015 1:18:22 PM oracle.dbtools.standalone.Deployer deploy
INFO: Deployed application path = /u01/app/oracle/product/ords/config/apex/apex/WEB-INF/web.xml
Jan 21, 2015 1:18:22 PM oracle.dbtools.common.config.file.ConfigurationFolder logConfigFolder
INFO: Using configuration folder: /u01/app/oracle/product/ords/config/apex
Configuration properties for: apex
cache.caching=false
cache.directory=/tmp/apex/cache
cache.duration=days
cache.expiration=7
cache.maxEntries=500
cache.monitorInterval=60
cache.procedureNameList=
cache.type=lru
db.hostname=localhost
db.password=******
db.port=1521
db.sid=DBDEV
debug.debugger=false
debug.printDebugToScreen=false
error.keepErrorMessages=true
error.maxEntries=50
jdbc.DriverType=thin
jdbc.InactivityTimeout=1800
jdbc.InitialLimit=3
jdbc.MaxConnectionReuseCount=1000
jdbc.MaxLimit=10
jdbc.MaxStatementsLimit=10
jdbc.MinLimit=1
jdbc.statementTimeout=900
log.logging=false
log.maxEntries=50
misc.compress=
misc.defaultPage=apex
security.disableDefaultExclusionList=false
security.maxEntries=2000
db.username=APEX_PUBLIC_USER
Jan 21, 2015 1:18:58 PM oracle.dbtools.common.config.db.ConfigurationValues intValue
WARNING: *** jdbc.MaxLimit in configuration apex is using a value of 10, this setting may not be sized adequately for a production environment ***
Jan 21, 2015 1:18:58 PM oracle.dbtools.common.config.db.ConfigurationValues intValue
WARNING: *** jdbc.InitialLimit in configuration apex is using a value of 3, this setting may not be sized adequately for a production environment ***
Using JDBC driver: Oracle JDBC driver version: 11.2.0.3.0
Jan 21, 2015 1:18:59 PM oracle.dbtools.rt.web.SCListener contextInitialized
INFO: Oracle REST Data Services initialized
Oracle REST Data Services version : 2.0.10.289.08.09
Oracle REST Data Services server info: Grizzly/1.9.49

Jan 21, 2015 1:18:59 PM com.sun.grizzly.Controller logVersion
INFO: GRIZZLY0001: Starting Grizzly Framework 1.9.49 - 1/21/15 1:18 PM
Jan 21, 2015 1:18:59 PM oracle.dbtools.standalone.Standalone execute
INFO: http://localhost:8080/apex/ started.

Step 6. Victory

With ORDS started, I’m now able to access APEX on my target and log in to see my applications.

APEX Login on TargetAPEX Apps on Target

Conclusion (or Step 7. Celebrate)

The cloned ORDS and APEX homes on the target and the DBDEV database are 100% full clones of their respective sources; block for block copies if you will. No matter how big the source data, these clones are done with a few clicks and takes only a few minutes, barely any disk space (in the megabytes, not gigabytes), and the clones can be refreshed from the source or rewound in minutes.

Delphix is capable of deploying not just database clones, but the whole app stack. Because Delphix stores incremental data changes (based on a retention period you decide), applications can be provisioned from any point in time or multiple points in time. And you can provision as many clones as you want to as many targets as you want, CPU and RAM on the targets permitting. All in all a fairly powerful capability and one I’ll be experimenting on quite a bit to see how the process and benefits can be improved. I’m thinking multi-VDB development deployments and a rewindable QA suite next!

The post Deploying Application Express with Delphix appeared first on Oracle Alchemist.