Pat Shuff

Subscribe to Pat Shuff feed
Oracle Blogs
Updated: 8 min 27 sec ago

installing Tomcat on AWS

Wed, 2016-04-13 02:07
In our last two entries we installed Tomcat on the Oracle Compute Cloud. We first installed the application using oracle.bitnami.com and the second we installed Linux using Oracle Compute Cloud then downloading tomcat.apache.org and configuring the network and startup scripts. In this blog we will do the same thing for Amazon AWS. Note that there are a few blogs that do the same thing. We are going to cheat a little bit with AWS. Rather than configuring Linux, downloading Java and downloading Tomcat, we are going to go to the Amazon Marketplace and download an image that is already configured. This is similar to going through Bitnami but I thought it would be interesting to look at a different pre-configured instance and see how it differs from Bitnami. When we go to the marketplace we get the option of a community ami pool or a commercial ami pool. The selection is very diverse. I could not find anyone who pre-configured Tomcat on Oracle Enterprise Linux but did find Red Hat and Amazon Linux which are from the same codebase.

It is important to note that the commercial version does come with supplemental pricing on an hourly basis. This typically prices AWS as an option out of the running when compared to other cloud services.

We select an instance (the smallest since this is a demo of functionality) and go through the launch screens.

By default, the network configuration only opens up ssh and potentially port 80. We will need to add ports 8080 and 443. In hindsight we really don't need to add port 8080 because the commercial version remaps the catalina configuration file to port 80 but we did anyway for completeness.

Adding the new ports looks like

Note that this is different from the Oracle Compute network setup. Amazon sets this up during the instance configuration while Oracle allows you to add this after the instance is created. Neither are good or bad, just different. You do need to scroll to the far right to see the Security Group definition and follow the links to modify the rules to allow another port. My first assumption was to go to the instance configuration menu at the top but all the network options were greyed out. You need to scroll to the far right to change the ports using the security group link. I initially did not see this because my fonts were too large and I did not realize that I had to scroll to the far right to see this.

Once we have the network configured, we can review and launch the instance. Note that we can use our own ssh keys to attach to the instance.

When we finish and confirm everything we should get an initialization screen. If the startup takes too long we will get a waiting screen. Once the instance is created we should see that it is running in the EC2 console.

Once the instance is started we can connect to it. We do this by looking up the ip address and connecting with ssh.

It is important to note that Tomcat is installed in /opt/tomcat7 and the startup scripts in /etc/rc3.d/S80tomcat7_1 are already setup.

We restart the service just to test the startup script and test the instance locally by getting the html from the command line and confirm that everything works from our desktop browser.

In summary, we were able to install and configure everything using the marketplace in less than 15 minutes. The configuration was similar to the Bitnami instance but it is important to note that there is an extra cost associated with this instance on an hourly basis. The Bitnami economics are done on a per instance charge. I, for example, pay $30/month to allow me to deploy three instances across multiple cloud vendors. Note the model is on a per instance and not a per hour basis. We could have gone through the exact same configuration that we did with the Oracle Compute Cloud instance by installing Linux then using the Tomcat website to download the binaries and install. The same websites, same tar files, and same configurations work since both are Linux based installs.

installing Tomcat on Oracle

Tue, 2016-04-12 02:07
In our last entry we installed Tomcat onto the Oracle Compute Cloud using Bitnami. Just as a reminder, it was easy, it was simple, and it took 15 minutes. In this entry we are going to go through the manual process and show what has to happen on the server side and what has to happen on the cloud side. The steps that we will follow are
  • Install Oracle Enterprise Linux on the Oracle Compute Cloud Service
  • ssh into the box and install/update
    • Java 7
    • Tomcat
    • iptables to open port 80 and 443
  • Start the service and verify localhost:8080 works
  • Configure the ports on the cloud side so that 80 and 443 pass through

Installing Oracle Enterprise Linux has been done in a previous blog. We won't go through the screen shots for this other than to say that we called the box prsTomcat (as we did in the previous example) and requested OEL 6.6 with a 60 GB hard drive because this was the default installation and configuration. We selected the 60 GB hard drive because we had one preconfigured and it would reduce the creation time by not having to create and populate a new hard drive.

To ssh into the instance we need to go to the compute page and find the ip address. We need to login as opc so that we can execute sudo and install packages.

Once we have logged in we first need to verify that java is installed and configured. We do this with

java -version

This command verified that Java is properly installed. The next step is to download tomcat. To get the correct version and location to download it from we must go to tomcat.apache.org and figure out what version to install. This is a little confusing because there are numerous versions and numerous dot releases. We are looking for Tomcat 7 so we scroll down and download it from tar.gz binary distribution.

We look for Tomcat 7 and follow the download link.

Once we had downloaded the binary bundle we need to unzip this into a location that we want to run it from. In this example we are going to install it in /usr/local/apache-tomcat-version_number. This is done with

"w g e t " http://www-us.apache.org/dist/tomcat/tomcat-7/v7.0.68/bin/apache-tomcat-7.0.68.tar.gz
cd /usr/local
sudo tar xvzf /home/opc/apache-tomcat-7.0.68.tar.gz

Now that we have the binary downloaded, we have to start the server. This is done with the bin/startup.sh command. It is important to note that we will still need to install and configure an init script in /etc/rc3.d/S99tomcat to start and stop the service. This requires hand editing of the file to run the startup.sh script. Once we have Tomcat installed and running we can use "w g e t" to verify that the server is running

sudo /usr/local/apache-tomcat-7.0.68
"w g e t" http://localhost:8080
This should return the html page served by the Tomcat server. If it does not, we have an issue with the server starting or running.

Now that we have the server up and running, we need to update the iptables to add ports 80, 8080, and 443 as pass through ports. This is done by

sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT

sudo service iptables restart

Once we have the ports properly running at the operating system layer we need to go back to our compute cloud console and create the security rules for these ports. This is done by going into the instance and clicking on the network tab at the top of the screen.

We open up port 80 first from the public internet to the instance.

We then open up port 443 similarly

The final configuration should look like

We also need to add port 8080 which is by default not installed and configured. We do this by defining a new Security Application from the Networking tab. We add port 8080 then we have to add it as a security rule.

From this point we should be able to look from our desktop to the cloud instance and see port 80, 8080, and 443. We can test port 443 by logging into the management console as we did with the bitnami configuration. We can test port 8080 by going to the default link for our server ip address.

In summary, it took 5-6 minutes to get Linux installed. It took 5-10 minutes to do the y um install based on how many packages were out of sync and needed updating. It took 4-5 minutes to open up the ports and reconfigure the network access. To get the same configuration we would have to edit the catalina.conf file and redirect the browser from port 8080 to port 80 as well as create a startup script to initialize the server at boot time. Overall this method took us about 50% longer to install and configured the exact same thing as we did with Bitnami. The benefits to doing the configuration ourselves is that we could script it with tools like Puppet and Chef. We could automate this easily and make sure it is done the same way every time. Doing it by hand and creating the instance, logging in, and using a graphic interface to configure everything leads to error and divergence as time goes on.

Note that the "w g e t" should be one word but again, our blogging software does not allow me to use that word in the blog. Grrr!

installing Tomcat through bitnami

Mon, 2016-04-11 02:07
This week we are going to focus on installing Tomcat on to cloud servers. Today we are going to take the easy route. We are going to use bitnami and look at timings of how long everything takes as well as the automatic configurations that it sets up. In previous lesions we talked about linking your cloud accounts to bitnami and will not repeat that instructions. For those that are new to public domain software, Tomcat is a public domain software package that allows you to host java applications similar to WebLogic. I won't get in to a debate over which is better because we will be covering how to install WebLogic in a later blog. I will give you all the information that you need to make that decision of which is best for our company and implementation.

We login to our oracle.bitnami.com web site and verify our account credentials.

We want to launch a Tomcat server so we search for Tomcat and hover over the icon. When we hover over the icon the word Launch appears and we click on this button.

Once we click Launch we get the virtual machine configuration screen.

Things to note on this screen are

  • the name is what you want to add to identify the virtual macine
  • the cloud account identifies which data center, if it is metered or un-metered, and what shapes will be available to this virtual machine
  • the network is automatically configured for ports 80 and 443 and enabled not only in the cloud network security configuration but in the operating system as well
  • the operating system gives you the option but we default to OEL 6.7
  • we could increase the disk size and select the memory/cpu option but it does not show us the cost because bitnami does not know if your account is metered or un-metered which have different costing models.

After we click the create button we get an update that shows us how the installation is progressing. The installation took just under 15 minutes to finish everything, launch the instance, and show us the configuration.

Once everything finishes we get the ip address, the passwords, and the ssh keys that were used to create this virtual machine.

We are able to open the link to the Tomcat server by clicking on the Go To The Application on the top right of the screen. This allows us to see the splash screen as well as access the management console.

When you click on the Access my Application you get the detailed information about the Tomcat server. We can go to the management console and look at the configuration as well as bring the server up and down.

At this point we have a valid configuration that we can see across the internet. The whole process took 15 minutes and did not require any editing or configurations other than selecting the configuration and giving the virtual machine a name.

security diversion before going up the stack

Fri, 2016-04-08 02:07
This entry is going to be more of a Linux tutorial than a cloud discussion but it is relevant. One of the questions and issues that admins are faced is creation and deletion of accounts. With cloud access being something relatively new the last thing that you want is to generate a password with telnet access to a server in the cloud. Telnet is inherently insecure and any script kiddy with a desire to break into accounts can run ettercat and look for clear text passwords flying across an open wifi or wired internet connection. What you really want to do is login via secure ssh or secure putty is you are on windows. This is done with a public/private key exchange.

There are many good explanations of ssh key exchange, generating ssh keys, and using ssh keys. My favorite is a digitalocean.com writeup. The net, net of the writeup is that you generate a public and private key using ssh-keygen or putty-gen and upload the public file to the ~user/.ssh/authorized_keys location for that user. The following scripts should work on an Azure, Amazon, and Oracle Linux instance created in the compute shapes. The idea is that we initially created a virtual machine with the cloud vendor and the account that we created with the VM is not our end user but our cloud administrator. The next level of security is to create a new user and give them permissions to execute what they want to execute on this machine. For example, in the Oracle Database as a Service images there are two users created by default; oracle and opc. The oracle user has the rights to execute everything related to sqlplus, access the file systems where the database and backups are located, and everything related to the ora user. The opc user has sudo rights so that they can execute root scripts, add software packages, apply patches, and other things. The two users have different access rights and administration privileges. In this blog we are going to look at creating a third user so that we can have someone like a backup administrator login and copy backups to tape or a disk at another data center. To do this you need to execute the following instructions.

sudo useradd backupadmin -g dba
sudo mkdir ~backupadmin/.ssh
sudo cp ~oracle/.ssh/authorized_keys ~backupadmin/.ssh
sudo chown -R backupadmin:dba ~backupadmin
sudo chmod 700 ~backupadmin/.ssh

Let's walk through what we did. First we create a new user called backupadmin. We add this user to the dba group so that they can perform dba functions that are given to the dba group. If the oracle user is part of a different group then they need to be added to that group and not the dba group. Next we create a hidden directory in the backupadmin directory called .ssh. The dot in front of the file denotes that we don't want this listed with the typical ls command. The sshd program will by default look in this directory for authorized keys and known hosts. Next we copy a known authorized_keys file into the new backupadmin .ssh directory so that we can present a private key to the operating system as the backupadmin to login. The last two commands are setting the ownership and permissions on the new .ssh directory and all files under it so that backupadmin can read and write this directory and no one else can. The chown sets ownership to backupadmin and the -R says do everything from that directory down to the same ownership. While we are doing this we also set the group permissions on all files to the group dba. The final command sets permissions on the .ssh directory to read, write, and execute for the owner of the directory only. The zeros remove permissions for the group and world.

In our example we are going to show how to access a Linux server from Azure and modify the permissions. First we go to the portal.azure.com site and login. We then look at the virtual machines that we have created and access the Linux VM that we want to change permissions for. When we created the initial virtual machine we selected ssh access and uploaded a public key. In this example we created the account pshuff as the initial login. This account is created automatically for us and is given sudo rights. This would be our cloud admin account. We present the same ssh keys for all virtual machines that we create and can copy these keys or upload other keys for other users. Best practice would be to upload new keys and not replicate the cloud admin keys to new users as we showed above.

From the portal we get the ip address of the Linux server. In this example it is 13.92.235.160. We open up putty from Windows, load the 2016.ppk key that corresponds to the 2016.pub key that we initialized the pshuff account with. When asked for a user to authenticate with we login as pshuff. If this were an Oracle Compute Service instance we would login as opc since this is the default account created and we want sudo access. To login as backupadmin we open putty and load the ppk associated with this account.

When asked for what account to login as we type in backupadmin and can connect to the Linux system using the public/private key that we initialized.

If we examine the public key it is a series of randomly generated text values. To revoke the users access to the system we change the authorized_keys file to a different key. The pub file looks like

if we open it in wordpad on Windows. This is the file that we uploaded when we created the virtual machine.

To deny access to backupadmin (in the case of someone leaving the organization or moving to another group) all we have to do is edit the authorized_keys file as root and delete this public key. We can insert a different key with a copy and paste operation allowing us to rotate keys. Commercial software like key vaults and key management systems allow you to do this from a central control point and update/rotate keys on a regular basis.

In summary, best practices are to upload a key per user and rotate them on a regular basis. Accounts should be created with ssh keys and not password access. Rather than copying the keys from an existing account it would be an upload and an edit. Access can be revoked by the root user by removing the keys or from an automated key management system.

next generation of compute services

Thu, 2016-04-07 02:07
Years ago I was a systems administrator at a couple of universities and struggled making sure that systems were operational and supportable. The one thing that frustrated me more than anything else was how long it took to figure out how something was configured. We had over 100 servers in the data center and on each of these server we had departmental web servers, mail servers, and various other servers to serve the student and faculty users. We standardized on an Apache web server but there were different versions, different configurations, and different additions to each one. This was before virtualization and golden masters became a trendy topic and things were built from scratch. We would put together Linux server with Apache web servers, PHP servers, and MySQL. These later became called LAMP servers. Again, one frustration was the differences between the different versions, how they were compiled, and how they were customized to handle a department. It was bad enough that we had different Linux versions but we had different versions of every other software combination. Debugging became a huge issue because you first had to figure out how things were configure then you had to figure out where the logs were stored and then could start looking at what the issue was.

We have been talking about cloud compute services. In the past blogs we have talked about how to deploying an Oracle Linux 6.4 server onto compute clouds in Amazon, Azure, and Oracle. All three look relatively simple. All three are relatively robust. All three have advantages and disadvantages. In this blog we are going to look at using public domain pre-compiled bundles to deploy our LAMP server. Note that we could download all of these modules into out Linux compute services using a yum install command. We could figure out how to do this or look at web sites like digitalocean.com that go through tutorials on how to do this. It is interesting buy I have to ask why. It took about 15 minutes to provision our Linux server. Doing a yum update takes anywhere from 2-20 minutes based on how old you installation is and how many patches have been released. We then take an additional 10-20 minutes to download all of the other modules, edit the configuration files, open up the security ports, and get everything started. We are 60 minutes into something that should take 10-15 minutes.

Enter stage left, bitnami.com. This company does exactly what we are talking about. They take public domain code and common configurations that go a step beyond your basic compute server and provision these configurations into cloud accounts. In this blog we will look at provisioning a LAMP server. We could have just as easily have configured a wiki server, tomcat server, distance education moodle server, or any other of 100+ public domain configurations that bitmai supports.

The first complexity is linking your cloud accounts into the bitnami service. Unfortunately, the accounts are split into three different accounts; oracle.bitnami.com, aws.bitnami.com, and azure.bitnami.com. The Oracle and Azure account linkages are simple. For Oracle you need to look up the rest endpoint for the cloud service. First, you go to the top right, click the drop down to do account management.

From this you need to look up the rest endpoint from the Oracle Cloud Console by clicking on the Details link from the main cloud portal.

Finally, you enter the identity domain, username, password, and endpoint. With this you have linked the Oracle Compute Cloud Services to Bitnami.

Adding the Azure account is a little simpler. You go to the Account - Subscriptions pull down and add account.

To add the account you download a certificate from the Azure portal as described on the bitnami.com site and import it into the azure.bitnami.com site.

The Amazon linkage is a little more difficult. To start with you have to change your Amazon account according to Bitnami Instructions. You need to add a custom policy that allows bitnami to create new EC2 instances. This is a little difficult to initially understand but once you create the custom policy it becomes easy.

Again, you click on the Account - Cloud Accounts to create a new AWS linkage.

When you click on the create new account you get an option to enter the account name, shared key, and secret key to your AWS account.

I personally am a little uncomfortable providing my secret key to a third party because it opens up access to my data. I understand the need to do this but I prefer using a public/private ssh key to access services and data rather than a vendor provided key and giving that to a third party seems even stranger.

We are going to use AWS as the example for provisioning our LAMP server. To start this we go to http://aws.bitnami.com and click on the Library link at the top right. We could just as easily have selected azure.bitnami.com or oracle.bitnami.com and followed this exact same path. The library list is the same and our search for a LAMP server returns the same image.

Note that we can select the processor core count, disk size, and data center that we will provision into. We don't get much else to choose from but it does the configuration for us and provisions the service in 10-15 minutes. When you click the create key you get an updated screen that shows progress on what is being done to create the VM.

When the creation is complete you get a list of status as well as password access to the application if there were a web interface to the application (in this case apache/php) and an ssh key for authentication as the bitnami user.

If you click on the ppk link at the bottom right you will download the private ssh key that bitnami generates for you. Unfortunately, there is not a way of uploading your own keys but you can change that after the fact for the users that you will log in as.

Once you have the private key, you get the ip address of the service and enter it into putty for Windows and ssh for Linux/Mac. We will be logging in as the user bitnami. We load the ssh key into the SSH - Auth option in the bottom right of the menu system.

When we connect we will initially get a warning but can connect and execute common commands like uname and df to see how the system is configured.

The only differences between the three interfaces is the shapes that you can choose from. The Azure interface looks similar. Azure has fewer options for processor configuration so it is shown as a list rather than a sliding scale that changes the processor options and price.

The oracle.bitnami.com create virtual machine interface does not look much different. The server selection is a set of checkboxes rather than a radio checkbox or a sliding bar. You don't get to check which data center that you get deployed into because this is tied to your account. You can select a different identity domain which will list a different data center but you don't get a choice of data centers as you do with the other services. You are also not shown how much the service will cost through Oracle. The account might be tied to an un-metered service which comes in at $75/OCPU/month or might be tied to a metered service which comes in at $0.10/OCPU/hour. It is difficult to show this from the bitnami provisioning interface so I think that they decided to not show the cost as they do with the other services.

In summary, using a service like bitnami for pre-configured and pre-compiled software packages is the future because it has time and cost advantages. All three cloud providers have marketplace vendors that allow you to purchase commercial packages or deploy commercial configurations where you bring your own license for the software. More on that later. Up next, we will move up the stack and look at what it takes to deploy a the Oracle database on all three of these cloud services.

Oracle Linux on Amazon AWS

Wed, 2016-04-06 02:07
In this entry we are going to create a Linux 6.4 virtual machine on Amazon AWS EC2. In our last entry we did this on the Microsoft Azure using a single processor instance and 1.75 GB of RAM. The installation took a few steps and was relatively easy to install. We will not look at how to create an Amazon account but assume that you already have an account. The basic AWS console looks like the image below

When we click on the EC2 console instance it allows us to look at our existing instances as well as create new ones.

Clicking on the "Launch Instance" button allows us to start the virtual machine instance creation. We are given a choice of sources for the virtual machine. The default screen does not offer Oracle Linux as an option so we have to go to the commercial or community screens to get OEL 6.x as an option.

It is important to note that the commercial version has a surcharge on an hourly basis. If we search on Oracle Linux we get a list of different operating system versions and database and WebLogic installations. The Orbitera version in the commercial version adds a hefty surcharge of $0.06 per hour for our instance and gets more expensive on an hourly basis as the compute shapes get larger. This brings the cost to 7x times that of the Oracle Compute Service and 5x the times of the Microsoft Azure instance.

The community version allows us to use the same operating system configuration without the surcharge. The drawback to this option is trustability on the configuration as well as repeatability. The key advantage over the commercial version is that it has version control and will be there a year from now. The community version might or might not be there in a year and if you need to create a new virtual machine based on something that you did a year ago might or might not be there. On the flip side, you can find significantly older versions of the operating system in the community version that you can not in the commercial version.

Given that I am cheap (and funding this out of my own pocket) we will go through the community version to reduce the hourly cost. The main problem with this option is that we installed Oracle Linux 6.4 when installing on Oracle Compute Cloud Service and Microsoft Azure. On Amazon AWS we have to select Oracle Linux 6.5 since the 6.4 version is not available. We could select 6.6 and 6.3 but I wanted to get as close to 6.4 as possible. Once we select the OS version, we then have to select a processor shape.

Note that the smaller memory options are not available for our selection. We have to scroll down to the m3.medium shape with 1 virtual processor and 3.75 GB of RAM as the smallest configuration.

The configuration screen allows us to launch the virtual machine into a virtual network configuration as well as different availability zones. We are going to accept the defaults on this screen.

The disk selection page allows us to configure the root disk size as well as alternate disks to attach to the services. By default the disk selection for our operating system is 40 GB and traditional spinning disk. You can select a higher speed SSD configuration but there are additional hourly charges for this option.

The tags screen is used to help you identify this virtual machine with projects, programs, or geographical affiliations. We are not going to do anything on this screen and skip to the port configuration screen.

The port screen allows us to open up security ports to communicate with the operating system. Note that this is an open interface that allows us to open any ports that we desire and provide access to ports like 80 and 443 to provide access to web services. We can create white lists or protected networks when we create access points or leave everything open to the internet.

We are going to leave port 22 as the only port open. If we did open other ports we would need to change the iptables configuration on the os instance. We can review the configuration and launch the instance on the next screen.

When we create the instance we have to select a public and private key to access the virtual machine. You had to previously create this instance through the AWS console.

Once we select the key we get a status update of the virtual machine creation.

If we go to the EC2 instance we can follow the status of our virtual machine. In this screen shot we see that the instance is initializing.

We can now connect using putty or ssh to attach to the virtual machine. It is important to note that Amazon uses a different version of the private key. They use the pem extension which is just a different version of the ppk extension. There are tools to convert the two back and forth but we do need to select a different format when loading the private key using putty on Windows. By default the key extension that it looks for is ppk. We need to select all files to find the pem keys. If you follow the guidelines from Amazon you can convert the pem key to a ppk key and access the instance as was done previously.

It is important to note that you can not login as oracle but have to login as root. To enable logging in as oracle you will need to copy the public key into the .ssh directory in the /home/oracle directory. This is a little troubling having the default login as root and having to enable and edit files to disable this. A security model that allows you to login as oracle or opc and sudo to root is much preferable.

In summary, the virtual machine creation is very similar to the Oracle Compute Cloud Service and Microsoft Azure Cloud Service. The Amazon instance was a little more difficult to find. Oracle installations are not the sweet spot in AWS and other Linux instances are preferred. The ssh keys are a little unusual in that the EC2 instance wants a different format of the ssh keys and if Amazon generates them for you it requires a conversion utility to get it into standard format. The cost of the commercial implementation drives the price almost to cost prohibitive. The processor and memory configuration are similar to the other two cloud providers but I was able to try a 1 processor and 1 GB instance and it failed due to insufficient resources. We had to fall back to a much larger memory footprint for the operating system to boot.

All three cloud vendors need to work on operating system selection. When you search for Oracle Linux you not only get various versions of the operating system but database and weblogic server configurations as well. The management consoles are vastly different between the three cloud vendors as well. It is obvious what the background and focus is of the three companies. Up next, using bitnami to automate service installations on top of a base operating system.

Oracle Linux on Microsoft Azure

Tue, 2016-04-05 02:07
In this entry we are going to create a Linux 6.4 virtual machine on Microsoft Azure. In our last entry we did this on the Oracle Compute Cloud using a single processor instance and 15 GB of RAM. The installation took a few steps and was relatively easy to install. We will not look at how to create an Azure account but assume that you already have an account. The basic Azure console looks like the image below

From this console we can either create a Virtual Machine or Virtual Machine (classic). From the main console click on the "Virtual Machine" button on the left side of the screen. We will walk down this path rather than the classic mode for this installation. After you click on the virtual machine menu item and the "+ Add" button at the top left we can select the operating system type.

From this screen we can search for an installation type. If we type "oracle" in the search field we get over 20 entries provided by the Oracle Corporation. The first few are Linux only installations. The next few are Database and Java/WebLogic installations on Linux or Windows.

For our test, we will select the Oracle Linux 6.4 to match what we did in the previous blog. With this selection we get another screen that provides links to an informational page and more information. Notice that the deployment model only allows us to create the virtual machine in the classic mode. Had we gone down the Virtual machine (classic) menu item at the start we would end up in the same place with this selection. Clicking on the "create" button takes us to the next screen where we define the properties of the virtual machine. The basic information that we need are the virtual machine name, a user name to log in as, basic security access information (password or ssh key access), and compute size.

Rather than providing a username and password to access the virtual machine, we are going to select an ssh key. We will use the same key that we used in the last blog and copy the public ssh key that we created with puttygen and upload a text copy of the public ssh key. It is important to note that we can create the virtual machine with a password but to be honest we don't recommend it. You can do this but security becomes a huge issue. The password that you enter does not check for viability or security. I was able to type in the traditional "Welcome1" password and the system accepted it as a viable password. Again, it is not recommended to do this but I was testing to see if I could enter a simple password that is easily found in the dictionary.

When we click on the Pricing Tier we can select the compute shape. When we first clock on this we get three recommended choices. These choices are all single core options with a small memory footprint. It is important to note that all are IO limited at 500 IOs per second and all have the option for a load balancer to be put in front of the virtual machine. The key difference is the processor type. The A processor is a lower speed, older processor that does not have as much power. The D processor is a higher speed, newer processor. Both options are lower clock speeds than the Oracle compute shape which is a 3.0 Ghz Xeon processor. The memory configuration is significantly lower with 1.75 or 3.5 GB when compared to 15 GB or 30 GB offered by the Oracle Compute Service.

If we want to explore more options we can click on the "View All" option at the top right. This allows us to look at over 60 different configurations that have different core counts, memory configurations, and disk options.

For our exercise we are going to go with the recommended A1 Standard configuration with 1 virtual processor and 1.75 GB of RAM.

The final step that we need to look at are the network, disk, and availability zone configurations. To be honest, we could accept the defaults and not configure these options.

If we look at the optional configurations, we can configure an availability zone. This allows us to replicate services between user defined zones. For this instance we are going to use the standalone virtual machine configuration.

The next configuration option allows us to define the local network configurations, which subnet it will belong to, and the server name. We recommend not changing the subnet information because this could cause issues if you enter the wrong network or enter an invalid subnet that does not have a dhcp server.

We can select a reserved ip address rather than a dynamically allocated ip address. It is important to enter this information correctly because you could step on an existing server on the internet and not be able to get to your virtual machine. It is also important to map the static ip address to the domain name that you have reserved through naming servers on the internet. We will use the default dhcp rather than use a reserved ip address.

We could attach alternate disks to this instance. For example, if we wanted to pre-load the Oracle database binary, we could mount the disk as a secondary disk and attach it to our instance. We will not do this as part of our exercise but go with the default boot disk to show how to create a basic virtual image.

We can also configure the ports that are open to the virtual machine. It is important to note that by default ssh is open and available. We could open port 80 or port 443 if we wanted to provide web access to this machine. We would also have to change the iptables configuration on the operating system to gain access to these services.

Finally, we can add options to the Linux operating system. This would be similar to selecting the Orchestration option on the Oracle Compute Service. My recommendation is to not select this but do this with the apt-get or yum installation method using post configuration utilities and methods.

The final options that we have deal with how we pay for the service and which data center we drop the virtual machine into. We will accept the defaults of "Pay as you go" and "US East" data center for our exercise.

When we click "Create" we are put back on the main portal screen with an update window showing progress. The create takes a few minutes and will be dropped into the Virtual Machine window when finished.

You can click on the bell shaped icon on the top right to see the progress bar and click on the progress bar to look at the ongoing status of creation. Note that the status is "creating" and will change when the virtual machine is finished creating.

Once the status turns to "running" we can click on the machine name and get more detailed information like the ip address assigned and more detailed data on the final configuration.

Once everything is running we can login via putty on Windows or ssh on Linux or Mac. We get the ip address from the status page on the virtual machine. We use the username that we entered when we created the Linux instance. We use the private ssh keys to connect to the instance just like the Oracle Compute Service.

Once we accept the keys we can login and verify the os version and disk shape created.

In summary, the difference between the Oracle Compute Cloud and the Microsoft Azure Cloud is not that different. The selection of the operating system is much more of a GUI experience with Microsoft and the Oracle shapes are much larger when it comes to memory. The Microsoft options have more options on the low end and high end but the Oracle solution is designed for the Oracle Database and WebLogic servers. It took about the same amount of time to create the two virtual machines. Security is a little tighter with Oracle but can be made the same between the two. Azure gives you the option of using a username and password and allows you to open any port that you want into the virtual machine. Given that these instances are on the public internet we recommend a tighter security configuration.

Oracle Linux on Oracle Compute Cloud

Mon, 2016-04-04 02:07
In this blog we are going to look at creation of an Infrastructure as a Service foundation using Compute as a Service and Storage as a Service to create an Oracle Linux instance. To start with we begin by logging into http://cloud.oracle.com and entering our Identity Domain, user name, and password. In this example we are connecting to metcsgse00026 as cloud.admin for the username.

If we look to the right of the Compute Cloud Service header we see a "Service Console" tab. Clicking on this allows us to create a new virtual machine by clicking on the "Create Instance" button. Not all accounts will have the create instance button. Your account needs to have the funding behind generic compute and the ability to consume either metered or un-metered IaaS options.

Note that we have two virtual machines that have previously been created. The first listed is a database service that was created. The compute infrastructure is automatically created when we create a database as a service instance. The second listed is a Java service that was created through the Java Service console. The compute infrastructure was also created for the JaaS option. We can drill into these compute services to look at security, networking, and ip addresses assigned.

To create a virtual machine we click on the "Create Instance" button which takes us to the next screen. On this screen we enter the name of the virtual machine that we are creating, a description label, the operating system instance and type, the shape of the instance. By shape we mean the number of processors and memory since this is how compute shapes are priced.

To select the different types of operating systems, we can enter a "*" into the Image type and it lists a pull down of operating system types. You can scroll down to select different configurations and instances. In the screen shot below we see that we are selecting OEL 6.4 with a 5 GB root directory. The majority of the images are generic Linux instances with different disk configurations, different software packages installed, and different OS configurations.

The next step is to select the processor and memory size. Again, this is a pull down menu with a pre-configured set of options. We can select from 1, 2, 4, 8, and 16 virtual processors and either 15 GB of RAM or 30 GB of RAM per processor. These options might be a bit limiting for some applications or operations but are optimized and configured for database and java servers.

In this example we selected a 1 virtual processor, 15 GB of RAM, 5 GB of disk for the operating system, and Oracle Linux 6.4 as the operating system. We can enter tags so that we can associate this virtual machine with a target, production environment, system, or geographic location consuming the resources.

At this time we are not selecting any custom attributes and not using Orchestration to provision services, user accounts, passwords, or other services into our virtual machine. We click the "Next" button at the top of the screen to go to network configurations.

In the network configuration we can accept the defaults and have an ip address assigned to us. If we have an ip address on reserve we can consume that reserved address and even assign a name to it to resolve to linux6.mydomain.net if we wanted to map this to an internet name. In this example we just accept the defaults and try not to get too fancy on our first try. This will create an ip address for our server, open port 22 for ssh access, and allow us to network it to other cloud services inside our instance domain with local network routing.

The next step is to configure a disk to boot from. We are presented with the option of using a pre-configured existing disk or creating a new one. The list that we see here is a list of disks for the database and java servers that we previously created. We are going to click on the create new check box and have the system create the disk for us.

The storage property pull down allows us to select the type of disk that we are going to use. If we are trying to boot from this disk, we should select the default option. If we were installing a database we would select something like the iSCSI option to attach as the data or fast recovery disk options.

The final step is to upload the public key of our ssh key pair. This is probably the biggest differential between the three services. Amazon requires that you use their shared and secret key that they generate for you. Microsoft allows you to create a service without an ssh key and use a username and password to connect to the virtual machine. Oracle requires that you use the ssh public-private key that you generate with puttygen or ssh-keygen. The public key is uploaded during this creation time (or selected if you have previously uploaded the key). The private key is presented via putty or ssh connection to the server once it is created. The default users that are created in the Oracle instances are the oracle account that is part of the orainst group and the opc account that has sudo privileges.

Once we have everything entered, we click on next and review the screen. Clicking on the "Create" button will create a compute instance with the configuration and specifications that we requested. In this example we are creating a Linix 6.4 instance onto a 1 OCPU machine with 15 GB of memory and attaching a 5 GB disk drive for the operating system.

As the system is provisioning, it updates us on progress

When everything is done we can view the details of the virtual machine and see the ip address, which key was used, and how the service is configured.

Before we can attach to the server, we need to open up the ssh port (port 22). This is done by going into the Network tab and adding a "Security Rule". This rule needs to be defined as public internet access to our default security rule since we accepted the default network rules as our network protocol in the creation of the virtual machine.

Note in this example we selected ssh as the protocol, public internet as the source, and default as the destination. With this we can now connect using putty from our Windows desktop. We need to configure putty with the ip address of our virtual machine and our private key as the ssh protocol for connecting. We could create a tunnel for other services but in this example we are just trying to get a command line login to the operating system.

Note that we can confirm that we are running Linux 6.4 and have a 5 GB hard drive for the operating system. The whole process takes a few minutes. This is relatively fast and can be scripted with curl commands. More on that later.

compute as a service

Fri, 2016-04-01 14:42
In an ongoing learning journey of trying to understand cloud services, I got accounts on the Amazon cloud, Azure cloud, and Oracle cloud. I thought I would start with the basics and grow from there. As an exercise, let's create a Linux server with no software installed in each three platforms. Apart from creating an account on all three platforms (which was non-trivial) creation of a compute server on each platform was relatively simple.

Amazon Web Services

The initial look and feel of the console starts the experience. It does show what the three companies are focused on. Let's start with Amazon (it is first in the alphabet and I had to pick something). The console lists a wide variety of services and things that you can purchase. Without doing research I would not have known that S3 stands for storage and EC2 stands for compute.

I get what a virtual server in the cloud is but how does that differ from a docker container and why should I care? Why should I care about managing Web Apps if I am just looking for raw compute? Why do I want to run code outside of a virtual machine? Which one should I choose? We are not going to go into depth on any of these subjects. If we are just looking at running a Linux instance, the simple EC2 should be adequate. We can install Docker as a package in our Linux instance to help us control how much of a processor is allocated to a service or program. We can install applications like Tomcat or WebLogic to run Web Apps. Linux gives us the foundation to do all of this with packages. Lambda is a totally different beast in that I can run code snippets to do things like voice command interpretation for an Amazon Echo or asynchronous events from devices and launch web sites or REST apis without having to install, manage, and configure an operating system. The rest of the world calls this a Node.js function and offers it as a separate service as well. I realize that I am oversimplifying this but you have to know what you are trying to accomplish before you start to create your first compute instance in the cloud.

Microsoft Azure Services

The Azure services are a little different in that they focus more on the user creation of virtual machines, SQL server, and some app services. Creation of a virtual image is relatively easy and it makes sense what you are doing. The console is relatively simple and clean with more options on the second page instead of the first page as is done with Amazon.

As you click on the Add button for Virtual Image you get an expanded set of operation system options and configurations.

Note that you can search for Oracle Linux and get a listing of various versions of the database. The virtual machine is easy to configure and create using the portal. If, however, you want to configure and create this via a command line, you need to download the PowerShell and run everything inside the application. The command line is Microsoft specific and difficult to port and migrate to other services. With Amazon and Oracle you can easily use RESTapi calls to provision and create services. Microsoft makes it a little more difficult to generically script but easily do this in their shell and language.

Oracle Cloud Services

The Oracle cloud compute services are new to the market. In the past compute services have been sold in bundles of 500 processors but have recently been made available in single processor consumption models. The cloud console has a different look and feel because the focus of the cloud services are more on the PaaS layer and less on the compute and storage layers.

Note that the screen shot starts with the storage and compute services but scrolling down shows database, java, SOA, and more PaaS layers.

To create a virtual image, you need to click on the compute cloud service - Service Console and it will allow you create an instance. The operating system selection is not as graphical or user friendly as the Microsoft interface but does list a variety of operating system options and configurations.

In conclusion, all three of these cloud consoles allow you to create a virtual image. In the next blog entry we will walk through the steps needed to create a Linux 6 instance on each of the three cloud platforms. We will not talk about how to create accounts. We will assume that you can find account setup and creation on your own. All three sites offer "try me" services that give at least 30 days evaluations. The eventual recommendation will be to use services like bitnami.com that takes public domain services like LAMP servers, Wiki engines, blog servers, and other public domain tools. The Bitnami site allows you to select a pre-configured instance and provision it into all three of these cloud services along with a few other cloud providers.

printing from apex using pl/sql

Mon, 2016-03-28 15:55
As part of our exercise to convert an excel spreadsheet to APEX, we ran into some simple logic that required more logic and less database select statements. The question comes up how to do an if - then - else statement in APEX. In this example we are trying to advise between using compute on demand or dedicated compute resources. With dedicated compute resources, we can purchase 50 OCPUs as an aggregate. With dedicated we can go as low as 1 OCPU. If we request 14 systems with 4 OCPUs it might be better to request 50 OCPUs.

A typical question form would look like the following image allowing us to ask processor shape as well as quantity. If the total mount exceeds, 50 processors, we output a message suggesting dedicated compute rather than compute on demand.

To get this message on the screen, we first had to pull in the questions that we ask using variables. In this example, we read in the UNMETERED_COMPUTE_SHAPE which is a pick list that allows you to select (1, 2, 4, 8, or 16) OCPU shapes. You can also type in a quantity number into UNMETERED_COMPUTE_QUANTITY. The product of these two values allows us to suggest dedicated or compute on demand for economic reasons.

To execute pl/sql commands, we have to change the content type. To create this area we first create a sub-region. We change the name of the sub-region to represent the question that we are trying to answer. For this example we use the title "Compute on Demand vs Dedicated Compute" as the sub-region header. We then change the type to "PL/SQL Dynamic Content". Under this we can then enter our dynamic code. The sub-region looks like

If you click on the expand button it opens a full editor allowing you to edit the code. In our example we are going to read the variables :UNMETERED_COMPUTE_SHAPE and :UNMETERED_COMPUTE_QUANTITY. Notice the colon in front of these names. This is how we treat the values as variables read from APEX. The code is very simple. It starts with a begin statement followed by an if statement. The if statements looks to see if we are allocating more than 50 processors. We then output a statement suggesting dedicated or compute on demand using the htp.p function call. This call prints what is passed to it to the screen. The code should look like .

Overall, this is a simple way of outputting code that requires control flow. In the previous example we used a select statement to output calculations. In this example we are outputting different sections and different recommendations based on our selections. We could also set variables that would expose or hide different sub-regions below this section. This is done by setting :OUTPUT_VARIABLE = desired_value. If we set the value inside the pl/sql code loop, we can hide or expose sections as we did in a previous blog by setting a value from a pull down menu.

The code used to output the recommendation is as follows

BEGIN
if (:UNMETERED_COMPUTE_SHAPE * :UNMETERED_COMPUTE_QUANTITY > 50) THEN
    htp.p('You might consider dedicated compute since you have ' 
     || :UNMETERED_COMPUTE_SHAPE * :UNMETERED_COMPUTE_QUANTITY 
     || ' OCPUs which is greater than the smallest dedicated compute of 50 OCPUs');
else
    htp.p('Compute on Demand for a total of ' 
     || :UNMETERED_COMPUTE_SHAPE * :UNMETERED_COMPUTE_QUANTITY || ' OCPUs');
end if;
END;

converting excel to apex

Thu, 2016-03-24 11:00
I am trying to go through the exercise of converting an excel spreadsheet into apex and have stumbled across a few interesting tricks and tidbits.

One thing that I have noted is that stuff done in a spreadsheet can be automated via navigation menus in apex. I talk about this in another blog on how to create a navigation system based on parts of a service that you want to get you to the calculation that you need. This is much better if you don't really know what you want and need to be lead through a menu system to help you decide on the service that you are looking for.

To create a calculator for metered and un-metered services in a spreadsheet requires two workbooks. You can tab between the two and enter data into each spreadsheet. If something like a pricelist is entered into a unique spreadsheet, static references and dynamic calculations can be easily. For example, we can create a workbook for archive - metered storage services and a workbook for archive - unmetered services which will be blank since this is not a service that is offered. If we create a third workbook called pricelist, we can enter the pricing for archive services into the pricelist spreadsheet and reference it from the other sheets. For archive cloud services you need to answer four basic questions; how many months, how much you will start archiving, how much you will end up with, and how much do we expect to read back during that period. We should see the following as questions How Many Months?cell F6 Initial Storage Capacitycell F7 Final Storage CapacityCell F8 Retrieval FactorCell F9

The cost will be calculated as Storage Capacity((F8+F7+((F8-F7)/F6))*F6/2*price_of_archive_per_month)/F6((F8+F7+((F8-F7)/F6))*F6/2*price_of_archive_per_month) Retrieval Cost(((F8+F7+((F8-F7)/F6)/2)*(F9/100))*price_of_archive_retrieval/F6(((F8+F7+((F8-F7)/F6)/2)*(F9/100))*price_of_archive_retrieval Outbound Data Transfersumifs(table lookup, table lookup, ...)sumifs(table lookup, table lookup,...*F6 In Apex, this is done a little differently with a sequence of select statements and formatting statements to get the right answer

select 
'   sub-part: ' || PRICELIST.PART_NUMBER ||
    ' - Archive Storage Capacity           ' as Description,
to_char(PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2, '$999,990') as PRICE 
 from PRICELIST PRICELIST 
 where PRICELIST.PART_NUMBER = 'B82623'
UNION
select 
'   sub-part: ' || PRICELIST.PART_NUMBER ||
    ' - Archive Retrieval           ' as Description,
to_char(PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:RETRIEVE_ARCHIVE/100), '$999,990') as PRICE 
 from PRICELIST PRICELIST 
 where PRICELIST.PART_NUMBER = 'B82624'
UNION
select 
'   sub-part: ' || PRICELIST.PART_NUMBER ||
    ' - Archive Deletes           ' as Description,
to_char(PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:DELETE_ARCHIVE/100), '$999,990') as PRICE 
 from PRICELIST PRICELIST 
 where PRICELIST.PART_NUMBER = 'B82629'
UNION
select 
'   sub-part: ' || PRICELIST.PART_NUMBER ||
    ' - Archive Small Files           ' as Description,
to_char(:SMALL_ARCHIVE, '$999,990') as PRICE 
 from PRICELIST PRICELIST 
 where PRICELIST.PART_NUMBER = 'B82630'
UNION
select 
'   sub-part: ' || PRICELIST.PART_NUMBER ||
    ' - Outbound Data Transfer           ' as Description,
to_char(PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:RETRIEVE_ARCHIVE/100), '$999,990') as PRICE 
 from PRICELIST PRICELIST 
 where PRICELIST.PART_NUMBER = '123456'
UNION
select
'   Total:' as Description,
to_char(sum(price), '$999,990') as Price 
from (
  select   PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2 as price from PRICELIST 
    where pricelist.part_number = 'B82623'
  UNION
  select PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:RETRIEVE_ARCHIVE/100) as price from PRICELIST
    where pricelist.part_number = 'B82624'
  UNION
  select PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:DELETE_ARCHIVE/100) as price from PRICELIST
    where pricelist.part_number = 'B82629'
  UNION
  select PRICELIST.PRICE*1000*(:FINAL_ARCHIVE+:INITIAL_ARCHIVE+((:FINAL_ARCHIVE-:INITIAL_ARCHIVE)/12))*12/2*(:RETRIEVE_ARCHIVE/100) as price from PRICELIST
    where pricelist.part_number = '123456'
    );
The variables :INITIAL_ARCHIVE replaces F7, :FINAL_ARCHIVE replaces F8, and :RETRIEVE_ARCHIVE replaces F9. Rather than referring to the pricelist spreadsheet, we enter the pricing information into a database and do a select statement with the part_number being the key for the lookup. This allows for a much more dynamic pricebook and allows us to update and add items without risk of breaking the spreadsheet linkages. We can also use REST apis to create and update pricing using an outside program to keep our price calculator up to date and current. Using a spreadsheet allows users to have out of date versions and there really is not any way of communicating to users who have downloaded the spreadsheet that there are updates unless we are all using the same document control system.

Note that we can do running totals by doing a sum from a select ... union statement. This allows us to compare two different services like Amazon Glacier and Oracle Archive easily on the same page. The only thing that we need to add is the cost of Glacier in the database and generate the select statements for each of the Glacier components. We can do this and use a REST api service nightly or weekly to verify the pricing of the services to keep the information up to date.

The select statements that we are use are relatively simple. The difficult part is the calculation and formatting out the output. For the bulk of the select statements we are passing in variables entered into a form and adding or multiplying values to get quantities of objects that cost money. We then look up the price from the database and print out dollar or quantity amounts of what needs to be ordered. The total calculation is probably the most complex because it uses a sum statement that takes the results of a grouping of select statements and reformats it into a dollar or quantity amount.

An example of the interfaces would look like

a traditional spreadsheet

and in Application Express 5.0

pulling X-Auth-Token from login

Wed, 2016-03-23 11:00
I am a big scripting guy. I believe in automating as much as possible and having a program do as much as possible and me typing as little as possible. I find it easier to use command lines than drag and drop interfaces. I have been struggling with how to script the REST apis for Oracle Cloud Services and wanted to get some feedback on different ways of doing this. I wanted to script creation of a database for some workshops that I typically give. The first step is creating the storage containers for the database backup.

Realize that the blogging software that is used does not allow me to type in "c url" without the space. If you see "c url" somewhere in this text, take out the space.

Most of the information that I got is from an online tutorial around creating storage containers. I basically boiled this information down and customized it a little to script everything.

First, authentication can be obfuscated by hiding the username and password in environment variables. I typically use a Mac so everything works well in a Terminal Window. On Windows 7 I use CygWin-64 which includes Unix like commands that are good for scripting. The firs tsetp is to hide the username, identity domain, and password in environment variables.

  • export OPASS=password
  • export OUID=username
  • export ODOMAIN=identity_domain
In my case, the identity domain is metcsgse00026. The username is cloud.admin. The password is given to me when I log into the demo.oracle.com system corresponding to this identity domain. What I would type in is
  • export OPASS=password
  • export OUID=cloud.admin
  • export ODOMAIN=metcsgse00026
The first step required is authentication. You need to log into the cloud service using the RESTapi to generate an X-Auth-Token. This is done with a GET command using the "c url" command.

c url -v -X GET -H "X-Storage-User: Storage-$ODOMAIN:$OUID" -H "X-Storage-Pass: $OPASS" https://$ODOMAIN.storage.oraclecloud.com/auth/v1.0

Note the -v is for verbose and displays everything. If you drop the -v you don't get back the return headers. Passing the -i might be a better option since the -v echos the user password and the -i only replies back with the tokens that you are interested in.

c url -i -X GET -H "X-Storage-User: Storage-$ODOMAIN:$OUID" -H "X-Storage-Pass: $OPASS" https://$ODOMAIN.storage.oraclecloud.com/auth/v1.0 In our example, this returned

HTTP/1.1 200 OK
date: 1458658839620
X-Auth-Token: AUTH_tkf4e26780c9e6b1d171f3dbeafa194cac
X-Storage-Token: AUTH_tkf4e26780c9e6b1d171f3dbeafa194cac
X-Storage-Url: https://storage.us2.oraclecloud.com/v1/Storage-metcsgse00026
Content-Length: 0
Server: Oracle-Storage-Cloud-Service

When you take this output and try to strip the X-Auth-Token from the header you get a strange output and need to add -is to the command to suppress timing of the outputs.

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

If you add the grep "X-Auth-Token followed by awk '{print $2]' you get back just the AUTH_string which is what we are looking for.

c url -is -X GET -H "X-Storage-User: Storage-metcsgse00026:cloud.admin" -H "X-Storage-Pass: $OPASS" https://metcsgse00026.storage.oraclecloud.com/auth/v1.0 | grep -s "X-Auth-Token" | awk '{print $2}'

AUTH_tkf4e26780c9e6b1d171f3dbeafa194cac

accessing oracle cloud storage from command line

Tue, 2016-03-22 11:00
Note for the purposes of this blog entry, the world "c url" should be interpreted as one word and not two. Unfortunately, the blog editing software that we have prohibits this work and kicks the blog entry out if it is placed in the blog without the space. Can everyone say a collective "Good Grief" and move on. Unfortunately, you will need to delete the space to make everything work properly.

Now that we have the cost and use out of the way, let's talk about how to consume these services. Unfortunately, consuming raw blocks, either tape or spinning disk, is difficult in the cloud. Amazon offers you an S3 interface and exposes the cloud services as an iSCSi interface through a downloadable object or via REST api services. Azure offers something similar with REST api services but offers SMB downloadable objects to access the cloud storage. Oracle offers REST api services but offers NFS downloadable objects to access the cloud storage. Let's look at three different ways of consuming the Oracle Cloud services.

The first way is to use the rest API. You can consume the services by accessing the client libraries using Postman from Chrome or RESTClient from Firefox. You can also access the service from the c url command line.

c url -v -X GET -H "X-Storage-User: Storage-metcsgse00026:cloud.admin" -H "X-Storage-Pass: $OPASS" https://metcsgse00026.storage.oraclecloud.com/auth/v1.0

In this example we are connecting to the identity domain metcsgse00026. The username that we are using is cloud.admin. We store the password in an environment variable OPASS and pull in the password when we execute the c url command. On Linux or a Mac, this is done from the pre-installed c url command. On Windows we had to install cygwin-64 to get the c url command working. When we execute this c url command we get back and AUTH header that can be passed in to the cloud service to create and consume storage services. In our example above we received back X-Auth-Token: AUTH_tk928cf3e4d59ddaa1c0a02a66e8078008 which is valid for 30 minutes. The next step would be to create a storage container

c url -v -s -X PUT -H "X-Auth-Token: AUTH_tk928cf3e4d59ddaa1c0a02a66e8078008" https://storage.us2.oraclecloud.com/v1/Storage- metcsgse00026/myFirstContainer

This will create myFirstContainer and allow us to store data either with more REST api commands or tools like CloudBerry or NFS. More information about how to use the REST api services can be found in an online tutorial

The second way of accessing the storage services is through a program tool that takes file requests on Windows and translates them to REST api commands on the cloud storage. CloudBerry has an explorer that allows us to do this. The user interface looks like and is setup with the File -> Edit or New Accounts menu item. You need to fill out the access to look like . Note that the username is a combination of the identity domain (metcsgse00026) and the username (cloud.admin). We could do something similar with PostMan or RESTClient extensions to browsers. Internet Explorer does not have plug ins that allow for REST api calls.

The third, and final way to access the storage services is through NFS. Unfortunately, Windows does not offer NFS client software on desktop machines so it is a little difficult to show this as a consumable service. Mac and Linux offer these services as mounting an nfs server as a network mount. Oracle currently does not offer SMB file shares to their cloud services but it is on the roadmap in the future. We will not dive deep into the Oracle Storage Cloud Appliance in this blog because it gets a little complex with setting up a VM and installing the appliance software. The documentation for this serviceM is a good place to start.

In summary, there are a variety of ways to consume storage services from Oracle. They are typically program interfaces and not file interfaces. The service is cost advantageous when compared to purchasing spinning disks from companies like Oracle, NetApp, or EMC. Using the storage appliance gets rid of the latency issues that you typically face and difficulty in accessing data from a user perspective. Overall, this service provides higher reliability than on-premise storage, lower cost, and less administration overhead.

accessing cloud storage

Mon, 2016-03-21 15:02
Oracle cloud storage is not the first product that performs basic block storage in the cloud. The name is a little confusing as well. When you think of cloud storage, the first thing that you think of is dropbox, box.com, google docs, or some other file storage service. Oracle Cloud Storage is a different kind of storage. This storage is more like Amazon S3 storage and less like file storage in that it provides the storage foundation for other services like compute, backup, or database. If you are looking for file storage you need to look Document Cloud Storage Services which is more tied to processes and less tied to raw cloud storage. In this blog we will look at different ways of attaching to block storage in the cloud and look at the different ways of creating and consuming services. To start off with, there are two ways to consume storage in the Oracle Cloud, metered and un-metered. Metered is charged on a per-hourly/monthly basis and you pay for what you consume. If you plan on starting with 1 TB and growing to 120 TB over a 12 month period, you will pay on average for 60 TB over the year. If you consume this same service as an un-metered service you will pay for 120 TB of storage for 12 months since you eventually cross the 1 TB boundary some time over the year. With the metered services you also pay for the data that you pull back across the internet to your computer or data center but not the initial load of data to the Oracle Cloud. This differs from Amazon and other cloud services that charge both for upload and download of data. If you consume the resources in the Oracle Cloud by other cloud services like compute or database in the same data center, there is no charge for reading the data from the cloud storage. For example, if I use a backup software package to copy operating system or database backups to the Oracle Cloud Storage and restore these services into compute servers in the Oracle Cloud, there is no charge for restoring the data to the compute or database servers.

To calculate the cost of cloud storage from Oracle, look at the pricing information on the cloud web page. for metered pricing and for un-metered pricing.

If we do a quick calculation of the pricing for our example previously where we start with 1 TB and grow to 120 TB over a year we can see the price difference between the two solutions but also note how much reading back will eventually cost. This is something that Amazon hides when you purchase their services because you get charged for the upload and the download. for un-metered pricing and for metered pricing. Looking at this example we see that 120 TB of storage will cost us $43K per year with un-metered services but $36K per year for metered services assuming a 20% reading of the data once it is uploaded. If the read back number doubles, so does the cost and the price jumps to $50K. If we compare this cost to a $3K-$4K/TB cost of on-site storage, we are looking at $360K-$480K plus $40K-$50K in annual maintenance. It turns out it is significantly cheaper to grow storage into the cloud rather than purchasing a rack of disks and running them in your own data center.

The second way to consume storage cloud services is by using tape in the cloud rather than spinning disk in the cloud. Spinning disk on average costs $30/TB/month whereas tape averages $1/TB/month. Tape is not offered in an un-metered service so you do need to look at how much you read back because there is a charge of $5/TB to read the data back. This compares to $7/TB/month with Amazon plus the $5/TB upload and download charges.

Pages