Pat Shuff

Subscribe to Pat Shuff feed
Oracle Blogs
Updated: 57 min 21 sec ago

Instance and storage snapshot

Fri, 2016-08-26 07:51
Yesterday we went through and created an E-Business Suite 12.5.5 instance on three servers in the Oracle Public Cloud. On previous days we talked about how to protect these instances by hiding the database and removing access from the public internet and only allowing the application server to connect to the database instance. Today we are going to assume that our work as an architect is done and we need to backup our work. We could go through the standard ufsdump and backup our system to network storage. This only solves half the problem in the cloud. We can restore our data but things are a little different in the cloud world. We need to backup our Security Lists, Security Rules, and instance configurations. We might want to replicate this environment for a secondary dev/test or QA environment so creating a golden master would be a nice thing to do.

With the Oracle Cloud we have the option of doing an instance snapshot as well as storage snapshot. This is equivalent to cloning our existing instance and having it ready to provision when we want. This is different from a backup. A backup traditionally assumes a fixed computer architecture and we can restore our operating system bits and application code onto a disk. If we suddenly change and add a virtual private network for communications with our on premise data center the backup might or might not have that configuration as part of the bits on the network disk. Many customers found that this was the case with VMWare. When you can redefine the network through software defined networks, create virtual disks and virtual network interfaces, these additions are not part of a ufsdump or OS level backup. You really need to clone the virtual disk as well as the configurations.

Oracle released snapshots of storage as well as snapshots of instances in the May/June update of cloud services. There really are no restrictions on the storage snapshots but there are a few on the instance snapshots. For the instance snapshot you need to make sure that the boot disk is non-persistent. This means that you don't pre-create the disk, attach it to the instance and boot from it. The disk needs to have the characteristic of delete upon termination. This sounds very dangerous up front. If you create customizations like adding user accounts to /etc and init files to the /etc/init directory these get deleted on termination. The key is that you create an instance, customize it, and create a snapshot of it. You then boot from a clone of this snapshot rather than a vanilla image of the operating system.

First, let's look at storage snapshots. We can find more information in the online documentation for the console or the online documentation for the REST API and command line interface. There are some features in the REST API that are worth diving a little deeper into. According to the REST API documentation you can create a snapshot in the same server to allow for faster restores by specifying /oracle/private/storage/snapshot/collocated as a property when you create the snapshot. From this you can create a storage volume from a snapshot. We can do most of these functions through the compute console. We select the storage volume and select the Create Snapshot menu item.

We can now restore this snapshot as a bootable disk and can create a new instance based on this volume. We restore by going to the storage snapshot tab, selecting the snapshot, and selecting Restore Volume from the menu. We can see the restored volume in the storage list.

We can create an instance snapshot as well. The key limitation to creating a snapshot from an instance is that the disk needs to be non-persistent. This means that we have a disk that is deleted on termination rather than created and mounted as part of the instance. This is a little confusing at first. If you follow the default instance creation it creates a storage volume for you. You need to delete this storage volume and have it replaced by a ROOT disk that is deleted upon termination. If we walk through an instance creation we have to change our behavior when we get to the storage creation. The default creates a storage instance. We want to remove it and it will be automatically replaced by a nonpersistent volume.

Once we have this hurdle removed, we can create an instance snapshot. We select the instance and click on the Create Snapshot from the menu item. If the menu item is greyed out we have a persistent storage volume as our boot image.

We can create a bootable image from this snapshot by clicking on the menu for the snapshot and Associate Image with this snapshot. This allows us to create an instance from our image.

The key to using instance snapshots is we create a bootable instance, configure it the way that we want and then create a snapshot of this instance. This gives us a golden master of not only the boot disk but of the network and customizations that we have done to the instance. You have to think a little differently when it comes to instance snapshots. It is a little strange not having a persistent root disk. It is a little strange knowing that any customizations will be lost on reboot. It is a little strange knowing that default log files will be wiped out on reboot. You need to plan a little differently and potentially reconfigure your logs, configurations, and customizations to go to another disk rather than a default root disk. If you think about it, this is not a bad thing. The root disk should be protected and not customized. Once you have the customized it should be frozen in time. One key advantage of this methodology is that you can't really insert a root kit into the kernel. These types of intrusions typically need to reboot to load the malware. Rebooting reverts you back to a safe and secure kernel and default libraries. This does mean that any packages or customizations will require a new snapshot for this customization to be persistent.

In summary, snapshots are a good way of freezing storage and an instance in time. This is good for development and test allowing you to create a golden master that you can easily clone. It also adds a new level of security by freezing your boot disk with packages that you want and locks out malware that requires reboot. It does add a new layer of thought that is needed in that any package or root file customization requires a new golden image with a new snapshot. Hopefully this helps you think of how to use snapshots and create a best practice methodology for using snapshots.

Instance and storage snapshot

Fri, 2016-08-26 07:51
Yesterday we went through and created an E-Business Suite 12.5.5 instance on three servers in the Oracle Public Cloud. On previous days we talked about how to protect these instances by hiding the database and removing access from the public internet and only allowing the application server to connect to the database instance. Today we are going to assume that our work as an architect is done and we need to backup our work. We could go through the standard ufsdump and backup our system to network storage. This only solves half the problem in the cloud. We can restore our data but things are a little different in the cloud world. We need to backup our Security Lists, Security Rules, and instance configurations. We might want to replicate this environment for a secondary dev/test or QA environment so creating a golden master would be a nice thing to do.

With the Oracle Cloud we have the option of doing an instance snapshot as well as storage snapshot. This is equivalent to cloning our existing instance and having it ready to provision when we want. This is different from a backup. A backup traditionally assumes a fixed computer architecture and we can restore our operating system bits and application code onto a disk. If we suddenly change and add a virtual private network for communications with our on premise data center the backup might or might not have that configuration as part of the bits on the network disk. Many customers found that this was the case with VMWare. When you can redefine the network through software defined networks, create virtual disks and virtual network interfaces, these additions are not part of a ufsdump or OS level backup. You really need to clone the virtual disk as well as the configurations.

Oracle released snapshots of storage as well as snapshots of instances in the May/June update of cloud services. There really are no restrictions on the storage snapshots but there are a few on the instance snapshots. For the instance snapshot you need to make sure that the boot disk is non-persistent. This means that you don't pre-create the disk, attach it to the instance and boot from it. The disk needs to have the characteristic of delete upon termination. This sounds very dangerous up front. If you create customizations like adding user accounts to /etc and init files to the /etc/init directory these get deleted on termination. The key is that you create an instance, customize it, and create a snapshot of it. You then boot from a clone of this snapshot rather than a vanilla image of the operating system.

First, let's look at storage snapshots. We can find more information in the online documentation for the console or the online documentation for the REST API and command line interface. There are some features in the REST API that are worth diving a little deeper into. According to the REST API documentation you can create a snapshot in the same server to allow for faster restores by specifying /oracle/private/storage/snapshot/collocated as a property when you create the snapshot. From this you can create a storage volume from a snapshot. We can do most of these functions through the compute console. We select the storage volume and select the Create Snapshot menu item.

We can now restore this snapshot as a bootable disk and can create a new instance based on this volume. We restore by going to the storage snapshot tab, selecting the snapshot, and selecting Restore Volume from the menu. We can see the restored volume in the storage list.

We can create an instance snapshot as well. The key limitation to creating a snapshot from an instance is that the disk needs to be non-persistent. This means that we have a disk that is deleted on termination rather than created and mounted as part of the instance. This is a little confusing at first. If you follow the default instance creation it creates a storage volume for you. You need to delete this storage volume and have it replaced by a ROOT disk that is deleted upon termination. If we walk through an instance creation we have to change our behavior when we get to the storage creation. The default creates a storage instance. We want to remove it and it will be automatically replaced by a nonpersistent volume.

Once we have this hurdle removed, we can create an instance snapshot. We select the instance and click on the Create Snapshot from the menu item. If the menu item is greyed out we have a persistent storage volume as our boot image.

We can create a bootable image from this snapshot by clicking on the menu for the snapshot and Associate Image with this snapshot. This allows us to create an instance from our image.

The key to using instance snapshots is we create a bootable instance, configure it the way that we want and then create a snapshot of this instance. This gives us a golden master of not only the boot disk but of the network and customizations that we have done to the instance. You have to think a little differently when it comes to instance snapshots. It is a little strange not having a persistent root disk. It is a little strange knowing that any customizations will be lost on reboot. It is a little strange knowing that default log files will be wiped out on reboot. You need to plan a little differently and potentially reconfigure your logs, configurations, and customizations to go to another disk rather than a default root disk. If you think about it, this is not a bad thing. The root disk should be protected and not customized. Once you have the customized it should be frozen in time. One key advantage of this methodology is that you can't really insert a root kit into the kernel. These types of intrusions typically need to reboot to load the malware. Rebooting reverts you back to a safe and secure kernel and default libraries. This does mean that any packages or customizations will require a new snapshot for this customization to be persistent.

In summary, snapshots are a good way of freezing storage and an instance in time. This is good for development and test allowing you to create a golden master that you can easily clone. It also adds a new level of security by freezing your boot disk with packages that you want and locks out malware that requires reboot. It does add a new layer of thought that is needed in that any package or root file customization requires a new golden image with a new snapshot. Hopefully this helps you think of how to use snapshots and create a best practice methodology for using snapshots.

E-Business Suite in the Oracle Cloud

Thu, 2016-08-25 09:00
For the last three days we talked about deploying multiple servers and securing the different layers by hiding services in the cloud. Today we are going to go through the installation processes and procedure to install E-Business Suite 12.2.5 (EBS) into a multiple compute cloud instance. We could use this instance for development and test, quality assurance, or even production work if we wanted to. Note that there are many ways to install EBS. We could install everything into Oracle Compute Cloud Services (IaaS) and install WebLogic and the Oracle Database onto Linux servers. We could install the database component into Oracle Database Cloud Services (DBaaS) and the rest on IaaS. We could also install the database into DBaaS, the application into Oracle Java Cloud Services (JaaS), and the rest in IaaS. The current recommended deployment scenario is to deploy everything into IaaS and bring your own licenses for EBS, Database, WebLogic, and Identity Servers.

We are going to go through the tutorial for installing EBS on IaaS for this blog. We are going to go down the multi-node install which first requires installing the provisioning tools to boot all of the other images into standalone instances. We will need at least four compute instances with 500 GB of disk storage to deploy our test. The individual requirements are shown in the diagram below.

Before we can start deploying we must first go to the Oracle Cloud Marketplace and download five EBS bootable images. We start by going to the marketplace and searching for "e-business" images. A list of the images that we need are shown in the diagram below.

Step 1:Download EBS 12.2.5 Fresh Install DB Tier Image. This is done by selecting the image that is returned from the search. When we get to the image page we click on "Get App". This brings up a usage terms screen that we need to click on and click OK. Once we have accepted the terms we are presented with a list of cloud instances that we can deploy into. If you don't see a list of servers you need to go into your preferences for your instance and click the checkbox that allows you to provision the marketplace apps into your instance. You will also need Compute_Admin roles to provision these boot images. You don't need to go to the compute instance after you download the image. You are mainly trying to copy the DB Tier Image into your private images.

Step 2:Download EBS 12.2.5 Demo DB Tier Image. Unfortunately there is no go back feature so you need to go to the marketplace page, search again for e-business, and select the Demo DB Tier Image.

Step 3:Download EBS 12.2.5 Application Tier Image.

Step 4:Download EBS OS-Only Image

Step 5:Download EBS Provisioning Tools Image

Step 6:Verify that all of the images are ready. You should get an email confirmation that the image is ready. You should also be able to create a new instance and see the images in the private images area. You should have five images available and we could create a bootable instance for all of them.

Step 7:Create a compute instance using the Provisioning Tool image. We are going to go with an OC3 instance and accept the default. We will create a new security list and rule that allows http access. We do have to select the boot image from the private image list.

You get to review this before it is provisioned.

This will create an Orchestration that will create the bootable disk and boot the instance. It will take a few minutes to do this and once it is done we should have all of the provisioning tools ready to execute and deploy our multi-node EBS instance.

Step 8:Connect to the server via ssh using opc. Get the ip address from the previous screen. When I first tried to connect I had to add default to the Security List otherwise the connection timed out. Once I added the ssh rule, everything worked as expected.

Step 9:change user to oracle and execute

knife oc image list

You will need the compute endpoint of the compute service because you will be prompted for it. To find this you need to go to the Compute Dashboard and look at the Compute Detail. The RESTapi Endpoint is shown but for our instance we need to change it a little bit. We have two zones associated with this domain. We want to connect to the z16 instead of the z17 zone. Once we enter the endpoint, identity domain, account id, and account password, we get a list of images that we can boot from. At the bottom of the list we see the EBS images and should be good to go. It is important to repeat that using the z17 zone will not show the images so we had to change over to the z16 zone. This is due to a Marketplace configuration that always deploys images into the lowest numbered zone for your instance.

Step 10:Edit /u01/install/APPS/apps-unlimited-ebs/ProvisionEBS.xml and replace the id-domain and user name with the output of the knife command. It is important to note that your substitute command will be a little different from the screen shot below. I also had to change the OS-Image to include the date otherwise the perl script that we are about to execute will fail as well. The file name should be /Compute-obpm44952/pat.shuff@oracle.com/Oracle-E-Business-Suite-OS-Image-12032015 but your instance and user will be different.

Step 11:Run perl /u01/install/APPS/apps-unlimited-ebs/ProvisionEBS.pl to start the install. This will ingest the xml file from the previous section and present you a menu system to install the other instances. The system will again ask for the restAPI Endpoint for the compute server, your restAPI Endpoint for storage (go to Dashboard and click on Storage to get this), your identity domain, account, and password again. For our test installation we selected option 3 for a multi-node single application server installation. The perl script then installs chef, pulls cookbooks, and installs the database, app server, and forms server instances into compute instances. This step will take a while. I recommend playing around with all of the options and configurations until you get comfortable with what you are installing. We were going for the demo installation rather than a dev/test installation. We went for a single app node and a single database node. We could have gone for multiple app nodes and gone with demo or dev deployments. Some of the screen shots from this process are below. We called our installation prsEBS so if you see references to this it relates to our installation. The process deploys orchestrations to the cloud services then starts these services in the Oracle Cloud.

We can confirm that this is doing what is expected by looking at the Orchestration page under the compute console.

When it is complete we will see four instances are running in compute.

In summary, we are able to provision multiple instances that comprise a single application, E-Business Suite. This process is well documented and well scripted. Hopefully these screen shots and steps help you follow the online tutorial mentioned earlier. What is needed next is to apply the security principles that we talked about in the past few days to secure the database and hide it from the public.

E-Business Suite in the Oracle Cloud

Thu, 2016-08-25 09:00
For the last three days we talked about deploying multiple servers and securing the different layers by hiding services in the cloud. Today we are going to go through the installation processes and procedure to install E-Business Suite 12.2.5 (EBS) into a multiple compute cloud instance. We could use this instance for development and test, quality assurance, or even production work if we wanted to. Note that there are many ways to install EBS. We could install everything into Oracle Compute Cloud Services (IaaS) and install WebLogic and the Oracle Database onto Linux servers. We could install the database component into Oracle Database Cloud Services (DBaaS) and the rest on IaaS. We could also install the database into DBaaS, the application into Oracle Java Cloud Services (JaaS), and the rest in IaaS. The current recommended deployment scenario is to deploy everything into IaaS and bring your own licenses for EBS, Database, WebLogic, and Identity Servers.

We are going to go through the tutorial for installing EBS on IaaS for this blog. We are going to go down the multi-node install which first requires installing the provisioning tools to boot all of the other images into standalone instances. We will need at least four compute instances with 500 GB of disk storage to deploy our test. The individual requirements are shown in the diagram below.

Before we can start deploying we must first go to the Oracle Cloud Marketplace and download five EBS bootable images. We start by going to the marketplace and searching for "e-business" images. A list of the images that we need are shown in the diagram below.

Step 1:Download EBS 12.2.5 Fresh Install DB Tier Image. This is done by selecting the image that is returned from the search. When we get to the image page we click on "Get App". This brings up a usage terms screen that we need to click on and click OK. Once we have accepted the terms we are presented with a list of cloud instances that we can deploy into. If you don't see a list of servers you need to go into your preferences for your instance and click the checkbox that allows you to provision the marketplace apps into your instance. You will also need Compute_Admin roles to provision these boot images. You don't need to go to the compute instance after you download the image. You are mainly trying to copy the DB Tier Image into your private images.

Step 2:Download EBS 12.2.5 Demo DB Tier Image. Unfortunately there is no go back feature so you need to go to the marketplace page, search again for e-business, and select the Demo DB Tier Image.

Step 3:Download EBS 12.2.5 Application Tier Image.

Step 4:Download EBS OS-Only Image

Step 5:Download EBS Provisioning Tools Image

Step 6:Verify that all of the images are ready. You should get an email confirmation that the image is ready. You should also be able to create a new instance and see the images in the private images area. You should have five images available and we could create a bootable instance for all of them.

Step 7:Create a compute instance using the Provisioning Tool image. We are going to go with an OC3 instance and accept the default. We will create a new security list and rule that allows http access. We do have to select the boot image from the private image list.

You get to review this before it is provisioned.

This will create an Orchestration that will create the bootable disk and boot the instance. It will take a few minutes to do this and once it is done we should have all of the provisioning tools ready to execute and deploy our multi-node EBS instance.

Step 8:Connect to the server via ssh using opc. Get the ip address from the previous screen. When I first tried to connect I had to add default to the Security List otherwise the connection timed out. Once I added the ssh rule, everything worked as expected.

Step 9:change user to oracle and execute

knife oc image list

You will need the compute endpoint of the compute service because you will be prompted for it. To find this you need to go to the Compute Dashboard and look at the Compute Detail. The RESTapi Endpoint is shown but for our instance we need to change it a little bit. We have two zones associated with this domain. We want to connect to the z16 instead of the z17 zone. Once we enter the endpoint, identity domain, account id, and account password, we get a list of images that we can boot from. At the bottom of the list we see the EBS images and should be good to go. It is important to repeat that using the z17 zone will not show the images so we had to change over to the z16 zone. This is due to a Marketplace configuration that always deploys images into the lowest numbered zone for your instance.

Step 10:Edit /u01/install/APPS/apps-unlimited-ebs/ProvisionEBS.xml and replace the id-domain and user name with the output of the knife command. It is important to note that your substitute command will be a little different from the screen shot below. I also had to change the OS-Image to include the date otherwise the perl script that we are about to execute will fail as well. The file name should be /Compute-obpm44952/pat.shuff@oracle.com/Oracle-E-Business-Suite-OS-Image-12032015 but your instance and user will be different.

Step 11:Run perl /u01/install/APPS/apps-unlimited-ebs/ProvisionEBS.pl to start the install. This will ingest the xml file from the previous section and present you a menu system to install the other instances. The system will again ask for the restAPI Endpoint for the compute server, your restAPI Endpoint for storage (go to Dashboard and click on Storage to get this), your identity domain, account, and password again. For our test installation we selected option 3 for a multi-node single application server installation. The perl script then installs chef, pulls cookbooks, and installs the database, app server, and forms server instances into compute instances. This step will take a while. I recommend playing around with all of the options and configurations until you get comfortable with what you are installing. We were going for the demo installation rather than a dev/test installation. We went for a single app node and a single database node. We could have gone for multiple app nodes and gone with demo or dev deployments. Some of the screen shots from this process are below. We called our installation prsEBS so if you see references to this it relates to our installation. The process deploys orchestrations to the cloud services then starts these services in the Oracle Cloud.

We can confirm that this is doing what is expected by looking at the Orchestration page under the compute console.

When it is complete we will see four instances are running in compute.

In summary, we are able to provision multiple instances that comprise a single application, E-Business Suite. This process is well documented and well scripted. Hopefully these screen shots and steps help you follow the online tutorial mentioned earlier. What is needed next is to apply the security principles that we talked about in the past few days to secure the database and hide it from the public.

hiding a server in the cloud

Wed, 2016-08-24 09:00
There was a question the other day about securing a server and not letting anyone see it from the public internet. Yesterday we talked about enabling a web server to talk on the private network and not be visible from the public internet. The crux of the question was can we hide the console and shell access and only access the system from another machine in the Oracle Cloud.

To review, we can configure ports into and out of a server by defining a Security Rule and Security List. The options that we have are to allow ports to communicate between the public-internet, sites, or instances. You can find out more about Security Lists from the online documentation. You must have the Compute_Operations role to be able to define a new Security List. With a Security List you can drop inbound packets without acknowledge or reject packets with acknowledgement. The recommended configuration is to Drop with no reply. The outbound policy allows you to permit, drop without acknowledgement or reject the packet with acknowledgement. The outbound policy allows you to have your program communicate with the outside world or lock down the instance. By default everything is configured to allow outbound and deny inbound.

Once you have a Security List defined, you create exceptions to the list through Security Rules. You can find out more about Security Rules from the online documentation. You must have the Compute_Operations role to manage Security Rules. With rules you create a name for the rule and either enable or disable communications on a specific port. For example the defaultPublicSSHAccess is setup to allow communications on port 22 with traffic from the public-internet to your instance. This is mapped to the default Security List which allows console and command line login to Linux instances. For our discussion today we are going to create a new Security List that allows local instances to communicate via ssh and disable public access. We will create a Security Rule that creates the routing locally on port 22. We define a port by selecting a Security Application. In this example we want to allow ssh which corresponds to port 22. We additionally need to define the source and destination. We have the choice of connecting to a Security List or to Security IP List. The Security IP List is either to or from an instance, the pubilc internet, or a site. We can add other options using the Security IP List tab on the left side of the screen. If we look at the default definitions we see that instance is mapped to the instances that have been allocated into this administrative domain (AD). In our example this maps to 10.196.96.0/19, 10.2.0.0/26, 10.196.128.0/19 because these three private ip address ranges can be provisioned into our AD. The public internet is mapped to 0.0.0.0/0. The site is mapped to 10.110.239.128/26, 10.110.239.0/26, 10.110.239.192/26. Note that the netmask is the key difference between the site and instance definitions.

Our exercise for today is to take our WebServer1 (or Instance 1 in the diagram below) and disable ssh access from the public internet. We also want to enable ssh from WebServer2 (or Instance 2) so that we can access the console and shell on this computer. We effectively want to hide WebServer1 from all public internet access and only allow proxy login to this server from WebServer2. The network topology will look like

Step 1:Go through the configuration steps (all 9 of them) from two days ago and configure one compute instance with an httpd server, ports open, and firewall disabled. We will call this instance WebServer1 and go with the default Security List that allows ssh from the public internet.

Step 2:Repeat step 1 and call this instance WebServer2 and go with the default Security List that allows ssh from the public internet.

Step 3:The first thing that we need to do is define a new Security List. For this security list we want to allow ssh on the private network and not on the public network. We will call this list privateSSH.

Step 4:Now we need to define a Security Rule for port 22 and allow communication from the instance to our privateSSH Security List that we just created. We are allowing ssh on port 22 on the 10.x.x.x network but not the public network.

Step 5:We now need to update the instance network options for WebServer1 and add the privateSSH Security List item and remove the default Security List. Before we make this change we have to setup a couple of things. We first copy the ssh keys from our desktop to the ~opc/.ssh directory to allow WebServer2 to ssh into WebServer1. We then test the ssh by logging into WebServer2 then ssh from WebServer2 to WebServer1. We currently can ssh into WebServer1 from our desktop. We can do this as opc just to test connectivity.

Step 6:We add the privateSSH, remove default, and verify the Security List is configured properly for WebServer1.

Step 7:Verify that we can still ssh from WebServer2 to WebServer1 but can not access WebServer1 from our desktop across the public internet. In this example we connect to WebServer1 as opc from our desktop. We then execute step 6 and try to connect again. We expect the second connection to fail.

In summary, we have taken two web servers and hidden one from the public internet. We can log into a shell from the other web server but not from the public internet. We used web servers for this example because they are easy to test and play with. We could do something more complex like deploy PeopleSoft, JDE, E-Business Suite, or Primavera. Removing ssh access is the same and we can open up more ports for database or identity communication between the hidden and exposed services.

The basic answer to the question of "can we hide a server from public internet access" the answer is yes. We can easily hide a server with Security Lists, Security Rules, Security IP Lists, and Security Applications. We can script these in Orchestrations or CLI scripts if we wanted to. In this blog we went through how to do this from the compute console and provided links to additional documentation to learn more about using the compute console to customize this for different applications.

hiding a server in the cloud

Wed, 2016-08-24 09:00
There was a question the other day about securing a server and not letting anyone see it from the public internet. Yesterday we talked about enabling a web server to talk on the private network and not be visible from the public internet. The crux of the question was can we hide the console and shell access and only access the system from another machine in the Oracle Cloud.

To review, we can configure ports into and out of a server by defining a Security Rule and Security List. The options that we have are to allow ports to communicate between the public-internet, sites, or instances. You can find out more about Security Lists from the online documentation. You must have the Compute_Operations role to be able to define a new Security List. With a Security List you can drop inbound packets without acknowledge or reject packets with acknowledgement. The recommended configuration is to Drop with no reply. The outbound policy allows you to permit, drop without acknowledgement or reject the packet with acknowledgement. The outbound policy allows you to have your program communicate with the outside world or lock down the instance. By default everything is configured to allow outbound and deny inbound.

Once you have a Security List defined, you create exceptions to the list through Security Rules. You can find out more about Security Rules from the online documentation. You must have the Compute_Operations role to manage Security Rules. With rules you create a name for the rule and either enable or disable communications on a specific port. For example the defaultPublicSSHAccess is setup to allow communications on port 22 with traffic from the public-internet to your instance. This is mapped to the default Security List which allows console and command line login to Linux instances. For our discussion today we are going to create a new Security List that allows local instances to communicate via ssh and disable public access. We will create a Security Rule that creates the routing locally on port 22. We define a port by selecting a Security Application. In this example we want to allow ssh which corresponds to port 22. We additionally need to define the source and destination. We have the choice of connecting to a Security List or to Security IP List. The Security IP List is either to or from an instance, the pubilc internet, or a site. We can add other options using the Security IP List tab on the left side of the screen. If we look at the default definitions we see that instance is mapped to the instances that have been allocated into this administrative domain (AD). In our example this maps to 10.196.96.0/19, 10.2.0.0/26, 10.196.128.0/19 because these three private ip address ranges can be provisioned into our AD. The public internet is mapped to 0.0.0.0/0. The site is mapped to 10.110.239.128/26, 10.110.239.0/26, 10.110.239.192/26. Note that the netmask is the key difference between the site and instance definitions.

Our exercise for today is to take our WebServer1 (or Instance 1 in the diagram below) and disable ssh access from the public internet. We also want to enable ssh from WebServer2 (or Instance 2) so that we can access the console and shell on this computer. We effectively want to hide WebServer1 from all public internet access and only allow proxy login to this server from WebServer2. The network topology will look like

Step 1:Go through the configuration steps (all 9 of them) from two days ago and configure one compute instance with an httpd server, ports open, and firewall disabled. We will call this instance WebServer1 and go with the default Security List that allows ssh from the public internet.

Step 2:Repeat step 1 and call this instance WebServer2 and go with the default Security List that allows ssh from the public internet.

Step 3:The first thing that we need to do is define a new Security List. For this security list we want to allow ssh on the private network and not on the public network. We will call this list privateSSH.

Step 4:Now we need to define a Security Rule for port 22 and allow communication from the instance to our privateSSH Security List that we just created. We are allowing ssh on port 22 on the 10.x.x.x network but not the public network.

Step 5:We now need to update the instance network options for WebServer1 and add the privateSSH Security List item and remove the default Security List. Before we make this change we have to setup a couple of things. We first copy the ssh keys from our desktop to the ~opc/.ssh directory to allow WebServer2 to ssh into WebServer1. We then test the ssh by logging into WebServer2 then ssh from WebServer2 to WebServer1. We currently can ssh into WebServer1 from our desktop. We can do this as opc just to test connectivity.

Step 6:We add the privateSSH, remove default, and verify the Security List is configured properly for WebServer1.

Step 7:Verify that we can still ssh from WebServer2 to WebServer1 but can not access WebServer1 from our desktop across the public internet. In this example we connect to WebServer1 as opc from our desktop. We then execute step 6 and try to connect again. We expect the second connection to fail.

In summary, we have taken two web servers and hidden one from the public internet. We can log into a shell from the other web server but not from the public internet. We used web servers for this example because they are easy to test and play with. We could do something more complex like deploy PeopleSoft, JDE, E-Business Suite, or Primavera. Removing ssh access is the same and we can open up more ports for database or identity communication between the hidden and exposed services.

The basic answer to the question of "can we hide a server from public internet access" the answer is yes. We can easily hide a server with Security Lists, Security Rules, Security IP Lists, and Security Applications. We can script these in Orchestrations or CLI scripts if we wanted to. In this blog we went through how to do this from the compute console and provided links to additional documentation to learn more about using the compute console to customize this for different applications.

Networking 102 - part 2

Tue, 2016-08-23 11:37
Yesterday we looked at what it takes to start an Apache Web Server on a Linux instance in the Oracle Cloud. We had to create a security rule, security list, and associate it with a running instance as well as configure the firewall on the operating system. We took a castle defense strategy and turned off the firewall on the operating system and are trusting that the external cloud firewall into our server is good enough. Today we are going to drop that assumption and spin up a second server in the same compute zone and configure a secure communication between the two servers. The idea is if we have a database server as a back end to a shopping cart, we want to hide the database from public access. We want to be able to store credit card information, customers addresses and phone numbers and do so securely. We don't want to expose port 1521 to the public internet but only expose it to our Web/Application server and keep it secure in our cloud instance from anyone hacking into it.

To achieve this level of security we will again look at our network diagram and realize that we can communicate on the private network interface rather than the public interface that is facing the public internet. What we want to do is change the network configuration on instance 1 to only listen for traffic from instance 2 and have instance 2 open to the public internet. We will do this with an Apache Web Server because it is easy to configure and test.

Step 1:Go through the configuration steps (all 9 of them) from yesterday and configure one compute instance with an httpd server, ports open, and firewall disabled. We are going to fix the security issues in the upcoming steps and make them only accessible from the second instance that we are about to spin up.

Step 2:Create a second Oracle Linux instance on the same compute cloud. This is done by going into the Create Instance, selecting Oracle Linux 6.6 and accepting the default configurations. A few minutes later we should have a second Linux instance that we can play with. Note that we could have cheated by creating a snapshot of our first instance and spinning up a new instance based on the first instance. This would save us a few steps and configuration options if our installation were more complex. We will save that for another day. Today we will provision an Oracle Linux 6.6 instance with WebServer as the security list which opens up port 80 and 22 for the instance. We accept all of the other defaults.

Step 3:Log into our instance as opc by getting the public ip address from the compute console and using ssh on a Mac or putty on Windows. Once we log in we install w get as a package so that we can read web pages from WebServer1.

Step 4:We can now read the web page from WebServer1 by getting the index.html page from the public ip address as well as the private ip address. We find these ip addresses from the compute console.

Step 5:Now that we can read from the private ip address, we can turn off the public ip address from WebServer1 and communicate on the 10.196 network. This is done by changing the Security List from WebServer back to default for WebServer1. We add the default in the security list and remove the WebServer in the security list.

Step 6:We can test the interface by repeating the read from the http server. We will get a timeout on the public ip address and timeout on the private ip address as well. We will need to create a new security list that allows network communications from the 10.196 network on port 80 to get to the server.

Step 7:We need to define a new Security List that allows port 80 on network 10.196. This is done by going to the Network tab on the compute console and defining a new security list. We will call it privateHttp. Once we have this defined we will allow http on port 80 on the private network but not the public network. We create a security rule for privateWebServer that allows us to go from an instance using port 80 to local instances. Once we have this defined we need to add the privateHttp in the security list for the WebServer1 instance.

Step 7a - add privateHttp to security list

Step 7b - add privateWebServer to security rule

Step 7c - associate new security list with instance

Step 8:Verify that we have connectivity on private network but lack of connectivity on public network

In summary, we created a new compute instance and reconfigured the network for our two compute instances. The goal was to setup our WebServer2 instance so that we could server the public internet with an Apache Web Server. Note that we did not go through these steps because we did this yesterday. We wanted to have WebServer2 talk to WebServer1 but do it on the private network and not have WebServer1 accessible from the public internet. We used an Apache Web Server as the example because it is easy to configure. We could have made this an identity server, a database, a file server, or any service that we want. The key difference would be the port that we create for the communication and security rule/list. Think of running EBusiness Suite or JD Edwards. We really don't want port 1521 of our database exposed to the public internet but we do want the http server exposed. If we run the ERP database on a separate server we need a secure way of communicating with our WebLogic server that is running the ERP logic without exposing drivers license numbers, credit cards, or private information to the public cloud. Hopefully this example allows you to take this concept with web servers and deploy more complex systems into the public cloud securely. It is important to note that we didn't fix the iptables issue and have the firewall turned off for the Linux instance on WebServer1. This is not best practice but we will leave that for another day.

Networking 102 - part 2

Tue, 2016-08-23 11:37
Yesterday we looked at what it takes to start an Apache Web Server on a Linux instance in the Oracle Cloud. We had to create a security rule, security list, and associate it with a running instance as well as configure the firewall on the operating system. We took a castle defense strategy and turned off the firewall on the operating system and are trusting that the external cloud firewall into our server is good enough. Today we are going to drop that assumption and spin up a second server in the same compute zone and configure a secure communication between the two servers. The idea is if we have a database server as a back end to a shopping cart, we want to hide the database from public access. We want to be able to store credit card information, customers addresses and phone numbers and do so securely. We don't want to expose port 1521 to the public internet but only expose it to our Web/Application server and keep it secure in our cloud instance from anyone hacking into it.

To achieve this level of security we will again look at our network diagram and realize that we can communicate on the private network interface rather than the public interface that is facing the public internet. What we want to do is change the network configuration on instance 1 to only listen for traffic from instance 2 and have instance 2 open to the public internet. We will do this with an Apache Web Server because it is easy to configure and test.

Step 1:Go through the configuration steps (all 9 of them) from yesterday and configure one compute instance with an httpd server, ports open, and firewall disabled. We are going to fix the security issues in the upcoming steps and make them only accessible from the second instance that we are about to spin up.

Step 2:Create a second Oracle Linux instance on the same compute cloud. This is done by going into the Create Instance, selecting Oracle Linux 6.6 and accepting the default configurations. A few minutes later we should have a second Linux instance that we can play with. Note that we could have cheated by creating a snapshot of our first instance and spinning up a new instance based on the first instance. This would save us a few steps and configuration options if our installation were more complex. We will save that for another day. Today we will provision an Oracle Linux 6.6 instance with WebServer as the security list which opens up port 80 and 22 for the instance. We accept all of the other defaults.

Step 3:Log into our instance as opc by getting the public ip address from the compute console and using ssh on a Mac or putty on Windows. Once we log in we install w get as a package so that we can read web pages from WebServer1.

Step 4:We can now read the web page from WebServer1 by getting the index.html page from the public ip address as well as the private ip address. We find these ip addresses from the compute console.

Step 5:Now that we can read from the private ip address, we can turn off the public ip address from WebServer1 and communicate on the 10.196 network. This is done by changing the Security List from WebServer back to default for WebServer1. We add the default in the security list and remove the WebServer in the security list.

Step 6:We can test the interface by repeating the read from the http server. We will get a timeout on the public ip address and timeout on the private ip address as well. We will need to create a new security list that allows network communications from the 10.196 network on port 80 to get to the server.

Step 7:We need to define a new Security List that allows port 80 on network 10.196. This is done by going to the Network tab on the compute console and defining a new security list. We will call it privateHttp. Once we have this defined we will allow http on port 80 on the private network but not the public network. We create a security rule for privateWebServer that allows us to go from an instance using port 80 to local instances. Once we have this defined we need to add the privateHttp in the security list for the WebServer1 instance.

Step 7a - add privateHttp to security list

Step 7b - add privateWebServer to security rule

Step 7c - associate new security list with instance

Step 8:Verify that we have connectivity on private network but lack of connectivity on public network

In summary, we created a new compute instance and reconfigured the network for our two compute instances. The goal was to setup our WebServer2 instance so that we could server the public internet with an Apache Web Server. Note that we did not go through these steps because we did this yesterday. We wanted to have WebServer2 talk to WebServer1 but do it on the private network and not have WebServer1 accessible from the public internet. We used an Apache Web Server as the example because it is easy to configure. We could have made this an identity server, a database, a file server, or any service that we want. The key difference would be the port that we create for the communication and security rule/list. Think of running EBusiness Suite or JD Edwards. We really don't want port 1521 of our database exposed to the public internet but we do want the http server exposed. If we run the ERP database on a separate server we need a secure way of communicating with our WebLogic server that is running the ERP logic without exposing drivers license numbers, credit cards, or private information to the public cloud. Hopefully this example allows you to take this concept with web servers and deploy more complex systems into the public cloud securely. It is important to note that we didn't fix the iptables issue and have the firewall turned off for the Linux instance on WebServer1. This is not best practice but we will leave that for another day.

Networking 102

Mon, 2016-08-22 13:08
This week we are going to go through some basic networking tutorials. It is important to understand how to do some simple stuff before we can do more complex stuff. We are going to start out by deploying an Oracle Linux instance, installing the Apache httpd service, and opening up port 80 to the world. This is a simple task but it helps understand how to find the ip address of your instance, how to open a port in the operating system as well as the cloud compute console, and how to connect to the instance from your desktop and from a second instance that we will spin up. Later this week we will spin up two Oracle Linux instances and configure an Apache httpd service on one but configure it only to talk to the other instance in the cloud.

Step 1: Open the Oracle Compute Console and Create Instance. We are going to create an Oracle Linux 6.6 instance. We are not going to do anything special but accept the defaults. We will call our server WebServer1 and give it the default network connection so that we can ssh into the instance.

After a few minutes we should have a Linux instance that has just port 22 open and we can ssh into the server. We don't have an Apache Web Server installed and if we did port 80 is locked down in the operating system and cloud networking interfaces.

Step 2: Connect to our instance with ssh to verify that we have command line access. We connect as opc so that we can execute commands as root. In this example we do this with the ssh command in a terminal window from MacOS. We could have just as easily used putty from a Windows box to make the connection.

Step 3: Add the Apache httpd software to the Linux instance with yum. We could just as easily have downloaded the software from apache.org and installed it that way but yum allows us to do this quickly and easily in one step. You need to make sure that you logged in as opc in the previous step because to sudo command will not work if you logged in as oracle. Note that the first time that you run this command it will take a while because you have to download all of the manifests for the different kernel versions and check for dependencies. The httpd package does not need many extras so the install is relatively clean. It will take a while to download the manifests but the actual install should not take long.

Step 4:Configure the httpd software to run by editing the index.html file and starting the service. Note that this will not allow us to see the service anywhere other than on this computer because we need to enable port 80 in the operating system and in the cloud service to pass the requests from the client to the operating system.

Step 5:Configure the cloud service to pass port 80 from the public internet to our instance. This is done in the Compute Console by clicking on the Networking tab and creating a new Security List. In this example we are going to create a new list that includes http and ssh as the protocols that we will pass through. We first create a Security List. We will call it WebServer.

Step 6:Configure port 80 as a Security Rule for the Security List that we just created. We create a rule for http and a rule for ssh. We then verify that the new rule has been created. Note that our instance is associated with the default rule, We need to change that in the next step.

Step 7:Associate our new rule with our instance. This is done by going into the Instance tab and clicking on View instance. We want to see what Security List is associated with our instance and change it. We are initially connected to the default list which only contains ssh. We want to add WebServer list and then delete the default list. The resulting list should only contain our WebServer list which enables ssh and http. We can easily now add https or sftp if we wanted to to help maintain our web server and not effect any other instances that are using the default rule/list.

Step 8:We now need to open up the ports in the operating system. This is done by modifying the SELINUX interface and iptables interface. We want to let traffic come into the server on port 80 so we can either turn off these services or add an iptables rule to allow everything on port 80 to pass through. We can disable all firewall rules by turning off the SELINUX services and iptables as shown below. It is not recommended to do this because it opens up all ports and makes your operating system vulnerable to attacks if other ports are open to this machine or other machines inside the same rack that you are running in. You can either watch a video or execute the commands shown on a tutorial web site that disables SELINUX and iptables. The important thing is to set SELINUX=disabled and turn off the iptables services for all of this to work.

Step 9:To test the changes, open a browser and try to attach to the Apache server. We should be able to go to the public ip address with a simple web client and get the index.html file. We should get back the message "I am here!" on the web page. Again, this is the insecure way of doing this. We really want to customize iptables to allows port 80 to pass and deny everything else that is not ssh.

In summary, we configured a Linux server, installed the Apache httpd, and configured the network rules at the cloud console and at the operating system to allow traffic to pass from the public internet into our compute instance. We are blocking all traffic at the cloud interface other than ports 80 and 22. Even though it is poor practice we disabled the firewall on the compute operating system and are allowing all traffic in and using our cloud instance as a firewall. This is not good practice because other compute services in the data center can access these open ports. We will dive deeper into that tomorrow and look at turning the operating system firewall back on and configuring it properly. We will also look at inter server communications inside the data center to allow hiding services from public access but allowing our front end public facing server to access the services securely.

Networking 102

Mon, 2016-08-22 13:08
This week we are going to go through some basic networking tutorials. It is important to understand how to do some simple stuff before we can do more complex stuff. We are going to start out by deploying an Oracle Linux instance, installing the Apache httpd service, and opening up port 80 to the world. This is a simple task but it helps understand how to find the ip address of your instance, how to open a port in the operating system as well as the cloud compute console, and how to connect to the instance from your desktop and from a second instance that we will spin up. Later this week we will spin up two Oracle Linux instances and configure an Apache httpd service on one but configure it only to talk to the other instance in the cloud.

Step 1: Open the Oracle Compute Console and Create Instance. We are going to create an Oracle Linux 6.6 instance. We are not going to do anything special but accept the defaults. We will call our server WebServer1 and give it the default network connection so that we can ssh into the instance.

After a few minutes we should have a Linux instance that has just port 22 open and we can ssh into the server. We don't have an Apache Web Server installed and if we did port 80 is locked down in the operating system and cloud networking interfaces.

Step 2: Connect to our instance with ssh to verify that we have command line access. We connect as opc so that we can execute commands as root. In this example we do this with the ssh command in a terminal window from MacOS. We could have just as easily used putty from a Windows box to make the connection.

Step 3: Add the Apache httpd software to the Linux instance with yum. We could just as easily have downloaded the software from apache.org and installed it that way but yum allows us to do this quickly and easily in one step. You need to make sure that you logged in as opc in the previous step because to sudo command will not work if you logged in as oracle. Note that the first time that you run this command it will take a while because you have to download all of the manifests for the different kernel versions and check for dependencies. The httpd package does not need many extras so the install is relatively clean. It will take a while to download the manifests but the actual install should not take long.

Step 4:Configure the httpd software to run by editing the index.html file and starting the service. Note that this will not allow us to see the service anywhere other than on this computer because we need to enable port 80 in the operating system and in the cloud service to pass the requests from the client to the operating system.

Step 5:Configure the cloud service to pass port 80 from the public internet to our instance. This is done in the Compute Console by clicking on the Networking tab and creating a new Security List. In this example we are going to create a new list that includes http and ssh as the protocols that we will pass through. We first create a Security List. We will call it WebServer.

Step 6:Configure port 80 as a Security Rule for the Security List that we just created. We create a rule for http and a rule for ssh. We then verify that the new rule has been created. Note that our instance is associated with the default rule, We need to change that in the next step.

Step 7:Associate our new rule with our instance. This is done by going into the Instance tab and clicking on View instance. We want to see what Security List is associated with our instance and change it. We are initially connected to the default list which only contains ssh. We want to add WebServer list and then delete the default list. The resulting list should only contain our WebServer list which enables ssh and http. We can easily now add https or sftp if we wanted to to help maintain our web server and not effect any other instances that are using the default rule/list.

Step 8:We now need to open up the ports in the operating system. This is done by modifying the SELINUX interface and iptables interface. We want to let traffic come into the server on port 80 so we can either turn off these services or add an iptables rule to allow everything on port 80 to pass through. We can disable all firewall rules by turning off the SELINUX services and iptables as shown below. It is not recommended to do this because it opens up all ports and makes your operating system vulnerable to attacks if other ports are open to this machine or other machines inside the same rack that you are running in. You can either watch a video or execute the commands shown on a tutorial web site that disables SELINUX and iptables. The important thing is to set SELINUX=disabled and turn off the iptables services for all of this to work.

Step 9:To test the changes, open a browser and try to attach to the Apache server. We should be able to go to the public ip address with a simple web client and get the index.html file. We should get back the message "I am here!" on the web page. Again, this is the insecure way of doing this. We really want to customize iptables to allows port 80 to pass and deny everything else that is not ssh.

In summary, we configured a Linux server, installed the Apache httpd, and configured the network rules at the cloud console and at the operating system to allow traffic to pass from the public internet into our compute instance. We are blocking all traffic at the cloud interface other than ports 80 and 22. Even though it is poor practice we disabled the firewall on the compute operating system and are allowing all traffic in and using our cloud instance as a firewall. This is not good practice because other compute services in the data center can access these open ports. We will dive deeper into that tomorrow and look at turning the operating system firewall back on and configuring it properly. We will also look at inter server communications inside the data center to allow hiding services from public access but allowing our front end public facing server to access the services securely.

New features in the Oracle Compute Cloud

Fri, 2016-08-19 07:00
Today Oracle updated the Oracle Compute Cloud Service by adding three new features.
  • Integration of the Oracle Marketplace into the Compute Cloud Console to make it easier to deploy custom solutions
  • Expanding the functionality of Backup Services to snapshot Compute Instances and clone them from snapshots
  • Making it easier to import existing virtual machines into the Oracle Cloud and making these images available to the Public and Private Marketplace
We talked earlier this week on pulling an image from the Oracle Marketplace. Previous to today you had to go to cloud.oracle.com/marketplace, setup preferences to link your account to your compute account, get an app from the marketplace, and provision the instance through the compute cloud. Today we just need to go into the create instance menu system from the Compute Console and select an image from the Marketplace to provision into a compute instance. This modification reduces the number of steps required to use the Marketplace as well as making it easier to provision preconfigured solutions into a compute instance or set of compute instances.

Note the Marketplace tab below the Private Images. A list of instances in the Marketplace along with a search engine are integrated into the provisioning of a new compute instance.

Compute instances now also have a backup tab similar to the monitor tab that was introduced a few weeks ago. This allows you to create snapshots of whole instances, save the snapshot as a bootable image, and provision new instances based on this snapshot.

This allows you to provision a predefined instance from a default OS or Marketplace instance, customize it, take a snapshot, then provision new instances from the customized installation.

The third new feature release is for users to have the ability to import VMWare instances directly into the Compute Cloud private images. The goal of this release is to allow users to import VMDK formatted images into the Oracle Cloud and run them with little or no modifications. This includes defining multiple network interfaces at import time rather than having to go back and configure multiple interfaces after the fact. The import does not require the users to modify the network drivers before importing but leverages the experience of the Ravello team for translating VMWare definitions into Oracle Compute Cloud definitions using Orchestration to create the network definition and provision it as the compute instance is started.

In summary, three new features were released today to make it easier to use the Oracle Compute Cloud Service. This is an ongoing improvement of services to help allow for frictionless migration of services from your data center into the cloud. These improvements along with those that will be announced between now and Oracle OpenWorld in September will help users treat the Oracle Public Cloud as an extension of their own data center for capacity expansion, disaster recovery, and development and test.

Ravello cloud virtualization

Thu, 2016-08-18 02:07
Yesterday we talked about what it would take to go from a bare metal solution or virtualized solution in your data center to a cloud vendor. We found out that it is not only difficult but it requires some work to make it happen. There are tools to convert your VMDK code to AMI with Amazon or VHD with Microsoft or tar.gz format with Oracle. That's the fundamental problem. There are tools to convert. You can't just simply pull a backup of your bare metal install or the VMDK code and upload and run it. Java ran into this problem during the early years. You could not take a C or C++ bundle of code, take the binary and run in on a Mac or Linux or Windows. You had to recompile your code and hopefully the libc or libc++ library was compatible from operating system to operating system. A simple recompile should solve the problem but the majority of the time it required a conditional compile or different library on a different operating system to make things work. The basic problem was things like network connections or reading and writing from a disk was radically different. On Windows you use a forward slash and on Linux and MacOS you use a backslash. File names and length are different and can or can't use different characters. Unfortunately, the same is true in cloud world. A virtual network interface is not the same between all of the vendors. Network storage might be accessible through an iSCSI mount, an NFS mount, or only a REST API. The virtual compute definition changes from cloud vendor to cloud vendor thus creating a need for a virtualization shim similar to a programming shim as Java did a few decades ago. Ravello stepped in and filled this gap for the four major cloud vendors.

Ravello Systems stepped in a few years ago and took the VMDK disk image proposed by VMWare and wrote three components to virtualize a cloud vendor to look like a VMWare system. The three components are nested virtualization, software defined networking, and virtual storage interfaces. The idea was to take not only a single system that made up a solution but a group of VMWare instances and import them into a cloud vendor unchanged. The user took a graphical user interface and mapped the network relationships between the instances and deployed these virtual images into a cloud vendor. The basis of the solution was to deploy the Ravello HVX hypervisor emulator onto a compute instance in the cloud vendor for each instance then deploy the VMWare VMDK on top of the HVX instance. Once this was done the storage and network interfaces were mapped according to the graphical user interface connections and the VMDK code could run unchanged.

Running a virtual instance unchanged was a radical concept. So radical that Oracle purchased Ravello Systems early this spring and expanded the sales force of the organization. The three key challenges faced by Ravello was that 50% of the workloads that run in customer data centers do not port well to the cloud, many of these applications utilize layer 2 IP protocols which are typically not available in most cloud environments, and VMWare implementations on different hardware vendors generate different virtual code and configurations enough to make it difficult to map it to any cloud vendor. The first solution was to virtualize the VMWare ESX and ESXi environment and layer it on top of multiple cloud vendor solutions. When an admin allocates a processor does this mean a thread as it does in AWS or a core as it does in Azure and Oracle? When a network is allocated and given a NAT configuration, can this be done on the cloud infrastructure or does it need to be emulated in the HVX?

The nested virtualization engine was designed to run VMWare saved code natively without change. Devices from the cloud vendor were exposed to the code as VMWare devices and virtual devices. The concept was to minimize the differences between different cloud solutions and make the processor and hypervisor look as much like ESX and ESXi as possible. HVX employs a technology called Binary Translation to implement high-performance virtualization that does not require these virtualization extensions. When virtualization extensions are available, the easiest way to implement the illusion is using "trap and emulate" .Trap and emulate works as follows. The hypervisor configures the processor so that any instruction that can potentially "break the illusion" (e.g., accessing the memory of the hypervisor itself) will generate a "trap". This trap will interrupt the guest and will transfer control to the hypervisor. The hypervisor then examines the offending instruction, emulates it in a safe way, and then it will allow the guest to continue executing. HVX, the Ravello hypervisor, uses a technology called binary translation. Unlike the trap-and-emulate method, binary translation does work when virtualization extensions are not available.

Pure L2 access is difficult and VLANs, span ports, broadcast/multicasting usually do not work. Ravello allows you to run existing multi-VM applications unmodified in the cloud, not just single virtual machines. To make this possible, Ravello provides a software-defined network that virtualizes the connectivity between the virtual machines in an application. The virtual network is completely user-defined and can include multiple subnets, routers, and supplemental services such as DHCP, DNS servers and firewalls. The virtual network can be made to look exactly like a datacenter network. The data plane of the virtual network is formed by a fully distributed virtual switch and virtual router software component that resides within HVX. Network packets that are sent by a VM are intercepted and injected into the switch. The switch operates very similar to a regular network switch. For each virtual network device, the virtual switch creates a virtual port that handles incoming and outgoing packets from the connected virtual NIC device.

Ravello’s storage overlay solution focuses on performance, persistence and security. It abstracts native cloud storage primitives such as object storage and various types of block devices into local block devices exposed directly to the guest VMs. Everything from the device type and controller type to the location on the PCI bus remains the same. Hence it appears to the guest as-if it was running in its original data-centre infrastructure. This allows the guest VM to run exactly as is with its storage configuration as if it was running on premises. Cloud storage abstraction (and presentation as a local block device), coupled with the HVX overlay networking capabilities allows for running various NAS appliances and their consumption over network based protocols such as iSCSI, NFS, CIFS and SMB. These block devices are backed by a high performance copy-on-write filesystem which allows us to implement our multi-VM incremental snapshot feature.

We could walk through a hands on lab developed by the Ravello team to show how to import a Primavera on site deployment into the Oracle Compute Cloud. The block diagram looks like the picture shown below. We import all of the VMDK files and connect the instances together using the GUI based application configuration tool.

Once we have the instances imported we can configure the network interfaces by adding a virtual switch, virtual gateway, virtual nic, assigning public IP addresses, and adding a VLAN to the configuration.

Ravello allows us to define features that are not supported with cloud vendors. For example, Amazon and Microsoft don't allow layer 2 routing and multicast broadcasting. VMWare allows for both. The HVX layer traps these calls and emulates these features by doing things like ping over TCP or multicast broadcasts by opening connections to all hosts on the network and sending packets to each host. In summary, Ravello allows you to take your existing virtualization engine from VMWare and deploys it to virtually any cloud compute engine. The HVX hypervisor provides the shim and even expands some of the features and functions that VMWare provides to cloud vendors. Functions like layer 2 routing, VLAN tagging, and multicast/broadcast packets are supported through the HVX layer between instances.

uploading custom boot image for compute cloud

Wed, 2016-08-17 12:45
One of the things that we talked about yesterday was uploading an image for a bundled solution into the Oracle Cloud Marketplace. Partners can register to bundle and sell solutions on the Oracle Cloud by providing a bootable image and have that image run in the Oracle Compute Cloud. Some examples of this are All of these solutions started with a base operating system and layered other products onto the operating system. They then took the resulting virtual machine configuration and bundled it into a tar.gz file and uploaded it to the Marketplace. Once this is done it can be offered as a product through the search engine and tracked, marketed, and sold by the partner. The resulting boot image is loaded into your private images and allows you to boot these instances as a compute service where you select the processor, memory, disk, and network configurations. The beauty of this interface is that it is the same that you would use if you wanted to create a custom boot image of your own. This enables features like snapshots of instances so that you can clone instances, replicate instances for scaling, and backup instances to a common storage location for fast disaster recovery.

Before we dive into how to create a bootable image for the Oracle Compute Cloud, let's review some basics so that we have the same terminology and a common language to discuss how to take an instance from your data center and run it in the Oracle Compute Cloud. We will also look at the steps needed for AWS and Azure and what it takes to upload an image to their compute clouds. Some of the different ways that you can run an application in your data center are

  • Bare Metal - load an operating system on a computer and load applications on the operating system. Backup the instance and restore into cloud
  • Oracle VirtualBox - create a virtual instance on your desktop/laptop/server and convert the resulting OVA/VDI into an uploadable format
  • VMWare ESX - create a virtual instance on your VMWare cluster and convert the resulting VMDK images into an uploadable format
  • Citrix or public domain Xen Server - create a virtual instance on your Xen server and convert the VMDK images into an uploadable format
  • Microsoft HyperV - create a virtual instance on your HyperV server and convert the VHD image into an uploadable format
The key difference between all of these steps are the conversion into an uploadable format for the target cloud provider. Oracle Compute Cloud currently requires that you create a tar.gz binary from a supported operating system and load it into your private image space. Documentation on creating your own image is available from Oracle and goes into detail on what operating systems are currently supported and references a Tutorial on building a LAMP Stack instance. In this example the tutorial uses VirtualBox to boot from an iso that is downloaded from edelivery.oracle.com which is the traditional way of building an image on bare metal or a virtual server.

We will not go through the installation process with images as we usually do because the tutorial does a very good job of showing you how to do this. The process takes a while (budget a half day) to setup VirtualBox, download the iso image, load and boot the Linux instance, patch the instance, install the Apache Web Server, MySQL, and PHP. The most difficult part of this is configuration of the network. The tutorial suggests that you turn off selinux and iptables. We suggest that you go to the extra effort to enable these services and open up the ports needed to communicate to your services. Leaving these services open inside any cloud providers internal network and relying upon the external firewall and routing rules is a slippery slope to insecurity. We suggest multiple layers of security at the cloud admin layer, at the network configuration layer with whitelist routing, and at the operating system layer. You might need to first allow connection to port 80 from anywhere at the operating system layer and tighten up these rules once you know where your clients are coming from but turning off security in the operating system is not recommended.

Microsoft provides a writeup on how to import a Linux image which could be configured with the LAMP stack. It is important to note that the article assumes that you are starting with a VHD file or have converted your existing file format into VHD for upload. The writeup also assumes that you have the Azure Command Line or PowerShell configured and installed which again assumes that you are starting with a Windows desktop to do the heavy lifting. There are few blogs that detail how to do this. Most recommend using Bitnami to load images or load a Linux image and rebuild the instance in Azure rather than building one of your own and uploading it. Most of the blog entries talk about having the para-virtual drivers installed to enable HyperV to properly work once the image is running.

Amazon provides a good documentation and tutorials on importing images from VMWare, Windows, and other formats. Their tools allow for either command line import or web based imports and converts them to Amazon AMI format. It is important to note that once you have imported the images you do need to reconfigure networking and security as must be done with almost all of the other cloud vendor solutions. The only true exception to this is Ravello Systems which is available in the Amazon, Google, Azure, and the Oracle Cloud. Ravello allows you to import your VMWare images unchanged and run them in all four cloud providers. Note that this is different from converting it to a runnable image in the cloud provider format. Ravello uses a hypervisor emulator that uploads a shim to translate VMWare calls into cloud provider calls and interfaces.

In summary, all of the cloud providers allow you to import existing in house images into their cloud. All use a format translator to translate the original format into the cloud provider format. The single exception is using the Ravello import utility that takes VMWare format and imports it unchanged and runs it in the cloud provider of your choice. The key difference between the different import mechanisms are what tools are needed. Do you need to start with a Windows desktop and use PowerShell to import the image? Do you need to install a command line utility that runs on most operating systems and convert your image to a custom image? Can you upload and download the image from the cloud provider to maintain version control in your data center? Does the cloud provider have the ability to run these images in your data center as if they were cloud services on a machine under your security umbrella? The more we dive into topics the less we seem to get answers but more and more questions. Hopefully today's blog gives you more insight into running existing applications on different cloud vendor platforms and what control you give up, what options you have, and what it takes to import virtual images from your data center into a cloud solution.

Cloud Marpetplace part 2

Tue, 2016-08-16 09:31
The Oracle Cloud Marketplace is a location where customers can discover partner provided applications and services that complement the Oracle Public Cloud services. It is also a location where partners can list and promote cloud based applications and services that extend, integrate with, or build on Oracles Public Cloud services. This is an interesting concept that has various ways of joining customers to partners and giving partners view into what customers are looking at, downloading, and executing. There are over 1000 apps, both commercial and public domain compiled source, available from over 1500 partners and system integrators. Some applications are helper apps like CloudBerry Explorer that assists you in consuming cloud storage services. Other applications are Oracle provided deployments of E-Business Suite running on infrastructure as a service for development and test. The Marketplace has system integrators listed that specialize in cloud services. Deloitte is an example of a Cloud Elite Systems Integrator who provides a variety of consulting services to help build custom cloud solutions for customers.

For partners, there are tools that allow you to develop, market, and sell applications. The development allows you to build binaries that you can download to use as tools to access cloud services, bundles to upload to the cloud, or images that can be launched in the cloud. The tools also exist to help you market your application once it is built and uploaded. You can look at page views, who downloaded your application, and geographic data about the customers to help target marketing campaigns. There are also tools to help with revenue capture and making money from your application either through the Marketplace page or redirection to your own page. There are lead generation integrations into the partner portal to help with follow up and calling campaigns for customers that are downloading and using applications. Partners must be gold level or above and get this service for free by signing up through the Oracle Partner Portal. Partners also have access to BI reports to look at trending, usage, and utilization of the applications that they have listed. These reports are designed with partners like Bitnami who lists hundreds of public domain compiled images on the Oracle Cloud for compute resources. The reports help them look at the most popular download, packages that are trending, feedback from customers both positive and negative, as well as the age of packages and ones that might need updating.

Customers can search for applications based on key words, categories of applications, and company names. You do need to enable the Compute_Operations role before you deploy an image into the Oracle Cloud as well as going into the Properties tab at the top right of the Cloud Console and creating a linkage between the Marketplace and your cloud account.

In summary, tools exist for customers and partners to help link people who want to create applications with people who want to use applications. There is a review mechanism to help with feedback as well as notifications of updates and changes to applications. This tool is constantly being updated and changing so don't be surprised if the screen shots that you see here are a little different when you visit the pages.

Cloud Marketplace

Mon, 2016-08-15 02:07
At OpenWorld 2013, Oracle announced the Cloud Marketplace. This is a place where customers can download or purchase solutions that are not stock out of the box images to boot from. For example, if you want a LAMP stack (Linux, Apache, MySQL, PHP) you could boot a Linux image, install the three other components as well as update the operating system with the latest patches. Alternatively you could search for LAMP in the Marketplace and get an image in a few minutes rather than building one.

The logic behind the Marketplace is that you have an account that is authorized to create a Compute Instance and give the account permission to upload bootable images from the Marketplace.

You also have to go into the Preferences menu and check the box that allows you to use this account to upload images.

Once you have the user that has permissions to create compute instances and a user that has permission to upload images, you can select an image and upload it to a cloud account. This is done easily by searching for an image. There are free images and images that you pay for. The pay images are typically bundled solutions that consist of one or more service with production quality software to solve a specific business problem. Oracle has many partners that provide both pay and free services for customers to use.

If, for example, we wanted to run a LAMP image, we would search for LAMP in the Marketplace.

We have the option of selecting a variety of Linux implementations (Oracle Linux, Ubuntu, etc) and loading this selection as a boot image. We also get applications that are based on the LAMP stack (Taleo add ons, WittyParrot, etc) that we could load. We will select an Oracle Linux implementation as an example.

When we select Get App it asks us to agree to legal terms and select the instance that we want the app to run in. It is important to note that the instance will default drop into the first zone available in your list. If you account has multiple zones associated with it, the bootable image will be dropped into the first zone and might or might not be available in the other zones. This is a known bug with the Marketplace that should be fixed if it has not already been fixes at the time of this posting.

Once you are done uploading the bootable disk you get a link to launch the compute console.

We can either continue with the provisioning of the instance by following the wizard or exiting out and launching a new instance. It is important to note that the new boot image will be listed as a private image since we uploaded it from the Marketplace. This allows us to load public or private images from a public or private Marketplace.

In June the Marketplace was changed to take advantage of Orchestrations. We talked about how these work two weeks ago and went through the configuration files. This week we will just accept the fast that these files will spin up our instance with an Apache Web Server, MySQL, and PHP. The entire process took just a few minutes and we have a running server. We can spin up new instances by copying the json Orchestration files and launch as many instances that we would like.

In summary, we used the Cloud Marketplace to load a pre-configure instance and launch it through either a compute wizard or by launching a new instance. You can learn more about the Marketplace in the online Cloud Marketplace documentation. Tomorrow we will talk about how to register as a partner to create images that you can sell or use as marketing tools to promote the name of your company.

networking wrap up

Fri, 2016-08-12 02:07
Today we are going to wrap up our detailed discussion with networking. We will probably revisit network performance at a later time but we are going to get out of the weeds today and talk about things at a slightly higher level. Yesterday we looked at screen shots of the Oracle, AWS, and Azure security lists and rules. It is important to note that Azure only offers TCP and UDP as rules. You can't define firewall and routing rules for other protocols. Amazon allows for ICMP firewall and routine rules which Oracle also allows. The big question in all of this becomes "so what". So what if you don't support ICMP? What functionality do you loose if you don't provide this packet header on top of IP?

First, let's review what ICMP is. ICMP stands for Internet Control Message Protocol and is defined in RFC 792. It is primarily used for diagnostic and control as well as discovery of services on a network. ICMP packets are treated differently from normal IP packets because they typically require a response to a query or an error code to be returned as part of the response. These packets are good for testing latency and connectivity between machines that need to traverse complex networks. Oracle uses this protocol as part of the keep alive heartbeat of a Database RAC configuration. The networking requirements of RAC also require ARP support as well as multicast within the same subnet. This basically means that RAC will never work on an Azure compute cluster because ICMP and multicast are not supported. Amazon has written a whitepaper on running RAC on AWS but it is not recommended to run RAC with a multi-host configuration that simulates shared storage. This looks like a good science experiment but not a production solution. Amazon basically engineered around the multicast requirements needed for shared storage by creating a message protocol at the operating system layer.

ICMP is also good for network discovery and monitoring network integrity. Oracle also uses this protocol with the Oracle Advanced Support Gateway. The support gateway is a tool that Oracle uses to manage services inside a customer data center. The Oracle Cloud Machine and Exadata/SuperCluster Managed Services products also use ICMP to verify the integrity of the network and report timing and connectivity issues when something happens on the network. The typical structures that are used with the gateway are looking for a message 0 request (standard ping echo request) and a message 8 request (ping echo reply and trace route data) to show network viability. Error codes returned in message 3 are also inspected to see if network configurations have changed or been modified.

Oracle Enterprise Manager also uses ICMP packets as part of a beacon communication and network discovery operations. This does imply that you can not use Enterprise Manager to manage host targets on Azure but you can on AWS. I have successfully configured Linux hosts as well as database instances in Amazon RDS and connected them to Enterprise Manager. This is a very powerful feature that allows you to schedule backups, replicate data from on premise to the cloud, and examine changes in a dev/test environment in the cloud and apply them to your production environment. Without ICMP you loose the ability to connect Enterprise Manager and these higher level functions. According to some Microsoft msdn blogs you can add ICMP to a Windows VM inside their firewall so you can ping between VMs but going across the internet is not necessarily supported. A second msdn blog suggests using alternate applications like TCPing and NMap to work around not having ICMP. This does not solve connecting with Enterprise Manager unless you run an OEM instance in Azure to monitor and measure all servers running in Azure.

The second protocol that Oracle supports that Amazon and Microsoft do not support is the GRE protocol. GRE stands for Generic Routing Encapsulation and is defined in RFC 2784 and RFC 2890. This protocol is used for point to point tunneling and IPSec for passing routing information between connected networks. Oracle currently uses this protocol to create Corente VPN services. The protocol was designed by Cisco Systems so connecting a Cisco router for a VPN connection is relatively easy with this protocol. You can connect with other protocols for a VPN connection but this layer has the mechanisms to keep routing tables in sync without having to run applications to talk to routers and update maps. This is more of an efficiency of networking than a functionality of networking that we saw with the ICMP support. We will talk about VPN services in a later blog. The general concept behind VPN is you would like the computers in your data center to talk to computers in the cloud as if they were on your corporate network. If you create a virtual private network the ip addresses that the computers in your data center use are the same ip addresses that are used in the cloud. The VPN creates a routing protocol that translates the virtual ip addresses which are typically a non routable address to an actual address that gets routed across the internet to the cloud provider. A VPN server on the cloud side then translates these packets to the non routable ip address and it looks like the request came from a machine on the local network. This simplifies network topology and configurations if we can extend our corporate network into a cloud network and have them operates as if they looked like they were on the same network. The tricks with with solution is that network changes need to be constantly updated and latency between your data center and the cloud data center can kill performance. The GRE protocol helps solve route table updates. Products like FastConnect help reduce latency by providing a fast path across the internet that you pay for on a monthly basis.

In summary, the protocols that cloud vendors support have important considerations in architecting a solution. Going with Azure basically prohibits you from using all of Enterprise Manager to manage servers in the Microsoft cloud. You can look at the database by connecting to port 1521 (with firewall rules set properly) but you will get host down when looking for operating system and host information. You can also see higher level services like WebLogic servers or App services because these protocols all run on TCP and connect to ports. The basic host information will not be available. Not supporting the GRE protocol is less of a functionality issue and more of a performance issue. Many Oracle customers are looking at FastConnect as a way of getting 1 GigE or 10 GigE connectivity but for simple workloads like database backup to storage operate good enough without having to go with the additional network cost. Again, this blog is not intended to say that one cloud vendor is superior to another. It is intended to help you decide which cloud provider will give you the service that you want. Feedback and comments are welcome.

networking - practical

Thu, 2016-08-11 02:07
Today we are going to explore how to configure and setup the basics of networking for a Linux compute instance in the Oracle Cloud. If you would like to read more about network configurations you should refer to the Oracle Compute Cloud Services (IaaS) documentation. We are specifically interested in chapter 7, Configuring Network Settings. Terms like security list, security rules, and roles play a part of the configuration. By default security is locked down and no traffic can be received from outside the host. It is important to note that the demo accounts that you get when you click the Try Me button on http://cloud.oracle.com do not allow you to create a list of valid ip addresses but allow you to either share ports with the public internet or not. This is mainly for simplicity when running and configuring the demo accounts. When you get a commercial paid account you get full access to restrict access by ip address, ip range, or list of computers.

If we log into our compute console we can see a list of instances that exist for an account. In our example we have four servers defined. One is Oracle Linux, one is CentOS7, one is a database server, and the fourth is a WebLogic server. If we click on the Network tab we see the Security Rules, Security Lists, Security Applicaitons, and Security IP Lists.

It is important to realize that Oracle takes a different approach when provisioning servers. The server is first provisioned with only SSH or RDP as the default rule or a security list that you create. In this example we see four different lists. The bitnami-moodle definition on the list opens up port 80 and 443 for a web server. The database definition prs12cHP opens up port 1521. The prsJava definitions open up administration ports as well as ports 80 and 443. The default security list only opens up port 22 for ssh connections.

If we look at the default security list the default operation is to deny all inbound traffic for computers not in the security list and drop the packets with no reply. We could configure reject with reply but this might lead to a denial of service attack with someone constantly sending TCP/IP requests to our server just to overload the server and network with TCP ack packets. By default the configuration is to drop packets and this typically happens at the border gateway rather than at our compute server. The outbound definition gives you the option of allowing packets, rejecting packets with an ack, and dropping packets with no ack. It is important to communicate to your users how you configure your server. If you configure outbound for deny with no reply they might end up troubleshooting network connection issues when it is by design dropping packets and it is not a router or connection issue.

Note that the concept of security list is a little misleading. For all of our instances we have an inbound policy of deny and an outbound policy of permit. Why not go with one security list and map all instances to this configuration? The key is in the security rules definition. We create a definition of a rule that maps security applications to a source and destination. By application we really mean a port number for an application. The source is where the packet is coming from and the destination is where the packet is going to. Since we have a permit all outbound traffic we only need to define the exceptions to the rule for inbound traffic. If, for example, we defined a deny inbound and deny outbound we would need to define the exception for both directions. If you look at the security rule definitions we are defining the source as the public-internet and the destination as each of our servers.

Security rules are essentially firewall rules. This permits traffic from your compute instance and can be used in different security lists as well as specific definitions between instances and external hosts. Yesterday we talked about turning off public ssh for a database server and only allowing ssh into the database server from our Java server. We would do this by turning off public-internet access over port 22 into the database server and allowing port 22 from our Java server to our database server. To access the database we would have to have public access of port 22 into the Java server, require the user to log in to the command line then ssh across to the database server using port 22 from the Java server to the database server. With this we can hide our database instance from the public internet but still allow access to the console to manage it. We will need to define an outbound rule that allows the database server to reach out and pull down patches if we want or require staging patches from the Java server to the database server by turning off all outbound traffic and only allowing port 1521 to and from the Java server.

Note that we create a rule association by defining the security application and associating it with a source and destination. When we create a security rule we define if it is enable or disable as well as the port or port ranges that we want to open. We can identify the source either with a security list or specific ip lists. If we go with a Security IP List we can define a specific instance, a subnet (site), or the public internet. We can do the same for the destination and specify a security list or specific ip lists. This effectively creates a virtual software defined network that maps packet routing to and from your instance.

If we look at the moodle server that we have running, for example, we have three security applications open. The first is ssh which allows us to connect to a shell and configure services. The second is http which maps to port 80 if we look at the Security Applications. The third is https which maps to port 443. These three ports are the only ports that are open and they are open to the public-internet as the source. We have a permit outbound rule so that the moodle server can pull in images from our storage servers, get updates with command line tools from other web servers, and download updates to the moodle server as needed from bitnami. We could just as easily have set the outbound policy to deny and only allow http, https, and ssh connections to this server inbound and outbound.

Note that this process and procedure is very similar to the way that Amazon AWS and Microsoft Azure define network rules. With AWS you go through the VPC Dashboard and define a VPN Connection. You create Security Groups that defines the ports and access rights. For example the bch-shared-firewall-34877 opens up ports 22, 80, and 443 to the public internet. The source of 0.0.0.0/0 is associated with the public internet. Note that we also have another rule that maps us to the 184.72.221.134 server for management. Once we define the inbound rules we can associate it with a VPN connection or gateway and define the inbound and outbound rules as we do on the Oracle Compute Cloud.

Azure does something similar and allows you to define ports or sets of ports when you create the instance. Note that TCP and UDP are the protocols that are allowed. This tends to imply the ICMP and other protocols are restricted in the Microsoft network. This typically is not a big deal but does have implications on how and what you can deploy in the Microsoft network. Amazon appears to allow ICMP as a rule definition as well as Oracle.

In summary, it appears that all three cloud vendors provide basic inbound and outbound rules. Microsoft limits the protocols to TCP and UDP and does not allow ICMP rules. This might or might not matter when selecting a cloud vendor. Once you have the rules defined you effectively have a secure system and flexibility to define subnets, netmasks, router tables, and layers of security with software defined networks. All three vendors appear to address this basic networking issue the same with one small difference with Azure. Now that we know how to configure networks it might be important to talk about speed, blocking, and throttling of networks. More tomorrow.

TCP, UDP, and IP

Wed, 2016-08-10 02:07
For the last two days we have been going through TCP/IP Illustrated Volume I. We are going to shift gears a little bit and look at the OSI stack from the perspective of another book. Today we are going to look at VPN Illustrated: Tunnels, VPNs, and IPSec by Jon C. Snader. We are shifting to another book because the TCP/IP Illustrated is an excellent book for Computer Scientists who want to know the nuts and bolts of how computers talk to each other. Things have changed since this book was published. We are no longer bound to a computer with an ethernet connection or two and network connections have become more of a virtual connection and less of a physical connection. When we talk about a network connection in the cloud we are not talking about a physical wire connected to a physical computer connected to a physical router. We are typically talking about a software defined network (SDN) where we have a virtual network connected to a virtual router that goes through a boarder router and gets us to the internet. We have covered the IP protocol in a pervious blog where we are concerned with a source and destination address and talked about different classes of networks, subnets, and netmasks. We skipped what it takes to figure out a routing map and shortest hop connection between the two computers. For most deployments we point to a default router and the default router deals with this complexity.

If we look at the layer communication (Figure 2.2 from VPN Illustrated) we see the different layers of the OSI layer represented

Today we are going to be talking about the Transport Layer or Layer 3. An example of an application would be a web browser communicating to a web server. The web browser would connect to the ip address of the web server and make an http request for a file. The http request is an example of an application layer request. At the TCP layer we have to define the handshake mechanism to request and receive the file as well as the port used for the request. Ports are a new concept where we not only talk to the ip address of a server but we specifically talk to it through a specific protocol that the server has a listener ready and available for requests. In our web browser example we read clear text web pages on port 80 typically and secure web pages on port 443. The secure web page not only can accept a file download request but does it without anyone else on the network knowing what is being asked because the communication between the web browser and web server is encrypted and encoded to prevent anyone from snooping traffic that is being exchanged. This is needed if you want to transmit secure information like credit card numbers, social security numbers, or any other financial related keys that assist in doing commerce across the internet.

Tools like

allow you to bring up and down network interfaces. You can go to a higher level with commands like ifup and ifdown on Linux to do more than just bring an ether connection up or down by reading the configuration files for netmasks, firewall, and network services to start. Other tools like

We just mentioned a new term here, a firewall. A firewall is a program that runs on a server and either allows traffic through or disables traffic on a specific port. For example, if we want to allow anyone on our subnet access to our web page, we open up port 80 to the same subnet that we are on. If our corporate subnet consists of more than just one subnet we might want to define an ip address range that we want to accept requests from. A firewall takes these connection requests at the TCP layer and opens up the TCP header, inspects it looking at the source and destination address as well as the port that is used for communications. If the port is open and allowing traffic from a subnet or ip address range, the request is then passed to the web server software. If the port is open but the traffic is coming outside of the ip address range, the request is dropped and an error is returned or the tcp/ip packet is dropped based on our firewall rules. The same is true for all ports that attach to compute engines on the internet. By default most cloud vendors open up port 22 which is the ssh port that allows you to connect to a command line on a Linux or Unix server. Microsoft Azure typically opens up port 3389 which is the remote desktop connection port. This allows you to connect to a Windows desktop using the RDP application on Windows desktops. It is typically a good idea to restrict the ip address that you can connect to your compute cloud server from an ip address rather than from any address.

We could consider a router to be an implementation of a firewall. A router between your subnet and the corporate network would be a wide open firewall. It might not pass UDP headers and most likely does not pass multicast broadcasts. It will not typically pass non routable addresses that we talked about yesterday. If we have a 192.168.1.xxx address we typically don't route this outside of our local network by definition since these are local private addresses. A router can block specific addresses and ports by design and act like a firewall. For example, Oracle does not allow ftp access from inside of the corporate network to outside servers. The ftp protocol transmits user names and passwords in the clear which means that anyone using tools like tcpdump, ettercap, and ethereal can capture and display the passwords. There are more secure programs like sftp that performs the same function but not only encrypts the username and password but each data byte transmitted to and from the server.

Many routers like wifi routers that most people have in their homes allow for network address translation (NAT) so that you are not presenting the 192.168.1.xxx address to the public internet but the address of your router/modem that connects you to the internet. Your desktop computer is at address 192.168.1.100, for example, but resolves to 66.29.12.122 with your internet provider. When you connect to port 80 at address 157.166.226.25 which correlates to http://cnn.com you connect to port 80 with a TCP/IP header source address of 66.29.12.122 and a destination address of 157.166.226.25. When your router gets a response back it knows that it needs to forward the response to 192.168.1.100 because you connected with a header value that said you were using a NAT connection. The router bridges this information back to you so that you don't need to consume more ip addresses on the internet for each device that you connect with from your home. The router/modem translates these requests using NAT, Bridge links, or actual IP addresses if you configure your back end server to request a direct mapping.

If we put all of this together along with the route command on Windows or Linux, we can define a default router that will take our IP packets and forward them to the right path. It might take a hop or two to get us to our eventual destination but we should be able to use something like Figure 2.9 from VPN Illustrated to represent out access to the Internet and use tool like traceroute to look at the hops and hop cost for us to get to the different cloud servers.

Note in this diagram if we are on Host 4 we set our default router to be router 2. We then trust that router 2 will know how to get to router 1 and router 1 will take us to our desire to look at cnn.com or whatever web site we are trying to connect to. All cloud vendors provide a default router configuration. All cloud vendors will give you a way of connecting to the internet. All cloud vendors will give you a way of configuring a firewall and subnet definitions. We might want to create a database server that does not have an internet connection and we need to connect to our application server through ssh then ssh into our database server through a private network. We might not have a public internet connection for our database but hide it in a subnet to keep it secure. In our routing map from VPN Illustrated we might want to put our database on host 4 and disable any connection to the internet. We might only want to allow traffic from the 200.10.4.xxx network to connect to the database. We might want to allow ssh, port 80, and port 443 connection to host 1 and allow only host 1 to connect ssh to host 4. All could vendors allow you to do this and configure virtual networks, subnets, firewalls, and netmasks.

We recommend that you get an IaaS account on AWS, Azure, and Oracle IaaS and play. See what works. See what you can configure from the command line. See what requires console configuration and your options when you provision a new operating system. See what you can automate with Orchestration scripts or python scripts or chef/puppet configurations. Automation is the key to a successful deployment of a service. If something breaks it is important to be able to automate restarting, sizing up, and sizing down services and this begins at the compute layer. It is also important to see if you can find a language or platform that allows you to change from one cloud vendor to another. Vendor lock in at this level can cause you to stick with a vendor despite price increases. Going with something like bitnami allows you to select which vendor is cheapest, has the best network speeds and options, has the fastest chips and servers, as well as the best SLAs and uptime history.

We didn't dive much into UDP. The key difference between TCP and UDP is the acknowledgement process when a packet is sent. TCP is a stateful transmission. When a web request is asked for by a browser the client computer sends a TCP/IP packet. The web server responds that it got the request and sends an acknowledgement packet that it received the request. The web server then takes the file that was requested, typically something like index.html, and sends it back in another TCP/IP packet. The web browser responds that it received the file with an acknowledgement packet. This is done because at times the Internet gets busy and there is a chance for collision of packets and the packet might never get delivered to the destination address. If this happens and the sender does not receive an acknowledgement it resends the request again. With a UDP packet the handshake does not happen. The sender sends out a packet and assumes that it was received. If there was a collision and the packet got dropped it is never retransmitted. Applications like Internet Radio and Skype use this type of protocol because you don't need a retransmission of audio signals if the time to listen to it has passed. The packet is dropped and the audio is skipped and picked up at the next packet transmitted. Most cloud vendors support UDP routing and transmission. This is optional and typically a firewall configuration. It might or might not make sense for a database to send and receive using UDP so it might not be an option when you get the Platform as a Service. Most Infrastructure as a Service vendors provide configuration tools to allow or block UDP.

In summary, we have covered basic addressing, routine, firewalls, and touched briefly on the TCP and UDP headers. We don't really need to get into the depths of TCP and how packets are transmitted, how congestion is handled, and how collisions are compensated for. In a cloud vendor you typically need to ask if the network is oversubscribed or bandwidth limited. You also need to ask if you have configuration limitations and restrictions on what you can and can not transmit. One of the risks to an unlimited network is noisy neighbor and getting congestion from another virtual machine that you are provisioned with. On the other hand if your network is oversubscribed you have to be bandwidth limited and accessing your storage can limit your application speed. Our advice is know your application, know if you are network limited, and know the security model and network configuration that you want ahead of time. Every cloud vendor differentiates their services but few offer service level agreements on bandwidth and compute resources. Read the fine details and play with all options.

link layers 2 and 3

Tue, 2016-08-09 02:07
We are going through the OSI 7 layer stack and looking at the different layers. Yesterday we stared the discussion by looking at Kevin Fall and Richard Steven's book TCP/IP Illustrated Volume 1. In this book they describe the different layers and look at the how, what, and why of the design. Today we will focus on layers 2 and 3 the link layer and network layer.

Alternate sources of information about these layers can be found at

Layer 2 is basically a way of communicating between two neighbors. How many milliseconds a bit of data is kept on the wire, physical addressing, and aggregation of data packets are defined here. If you have ever wondered what a MAC Address is, this is where it is defined. Vendors are given a sequence of bits that indicate the address of a device that they create. Note that this is not your ip address but a physical sequence of bits as defined by the Institute of Electrical and Electronic Engineers (IEEE) 802 definition. The data packet consists of six octets of data with the first three octets identifying a corporation or manufacturer and the second three octets representing a unique sequence number of a device that the vendor manufactured. An example of this would be the MAC address on my MacBook Pro, 00:26:b0:da:c8:10. Apple is assigned 00:26:b0 as the identifier for their products. My specific laptop gets the identifier da:c8:10. When a data packet is placed on the internet through a hard wired cable or wifi it is placed there with the unique MAC Address of my laptop. When data was generated and consumed by physical hardware these addresses meant something. With virtualization and containers the MAC Address has become somewhat meaningless because these values are synthetic. You really can't determine if something came from an Apple product because we can map the above MAC address to a virtual machine by defining it as a parameter. It is best practice not to use the same MAC address is a physical network because all of the computers with that address will pick up the packet off the wire and decode it.

Layer 3 is the communication protocol that is used to create and define packets. With Apple for example, they defined a protocol called Appletalk so that you could talk between Apple computers and devices. This protocol did not really take off. Digital Computers did something similar with VAX/VMS and DecNET. This allowed their computers to talk to each other very efficiently and consume a network without regard for other computers on the network. Over the years the IP protocol has dominated. The protocol is currently in transition from IPv4 to IPv6 because the number of devices attached to the internet have exceeded the available addresses with the protocol. The IPv4 protocol consists of a dotted-quad or dotted-decimal notation with four fields that denote networks. For example, 129.152.168.100 is a valid ip address. All of the four fields can range from 0 to 255 with some of the values reserved. For example, 0.0.0.0 is not considered to be a valid address and neither is 255.255.255.255 because they are reserved for special functions. IPv6 uses a similar notation but addresses are denoted as eight blocks of 16 bit values. An example of this would be 5f05:2000:80ad:5800:58:800:2023:1d71. Note that this give us 128 bits rather than 32 bits to represent an address. IPv4 has 4,294,967,296 possible addresses in its address space, and IPv6 has 340,282,366,920,938,463,463,374,607,431,768,211,456.

With IPv4 addressing there is something called classes of networks. A class A network consists of a leading zero followed by seven bits to define a network and 24 bits to define a specific host. This is typically not used when talking about cloud services. A class B network consists of a leading 1 and 0 followed by 14 bits to define a network and 16 bits to define a host. Data centers typically use something like this because they could have thousands of servers in a data center. A class C network consists of a leading 110 followed by 21 bits to define the network and 8 bits to define a host. This allows 256 computers to be on one network which could be a department or office building. A class D network starts with 1110 and is considered to be a multicast broadcast. If something is written with this sequence, the packets are written to all hosts on the network. All hosts should but are not mandated to pick up this packet and look at the data element. A class E network starts with 1111 and is considered to be reserved and not to be used. The image from Chapter 2 of TCP/IP Illustrated Volume I shows the above visually.

This comes into play when someone talks about netmasks. If you are talking about a 0.0.0.0/16 it means that you are ignoring the leading 16 bits and looking at the remaining 16 bits to use for routing. You might also see 0.0.0.0/24 which means that you use the last 24 bits to route the data. If you set your netmask to be 255.255.255.0 it means that you are using a class B network with the first 16 bits defining the corporate network, the next 8 bits defining the subnet in the company, and the last 8 bits to define the specific host. This means that you can have 255 subnets in the company and 255 computers on each network. A netmask of 255.255.255.0 suggests that you are not going to route outside of your subnet if the first three octets are the same. What this means is that a router either passes the packets through or does not pass the data through based on the netmask and ip address of the destination.

You might hear the term CIDR (Classless inter-domain routing). This term refers to how to get to and from a host if there are multiple ways of traversing the network. We will not get into this but netmasks are good ways of limiting routing tables and spanning trees across networks. This is typically a phrase that you need to know about if you are looking at limiting communication and flow of addresses across a data center.

Earlier we talked about reserved networks and subnets. Some of the network definitions for IPv4 are defined as private and non-routable networks. A list of these addresses include

  • 0.0.0.0/8 Hosts on the local network. May be used only as a source IP address.
  • 10.0.0.0/8 Address for private networks (intranets). Such addresses never appear on the public Internet.
  • 127.0.0.0/8 Internet host loopback addresses (same computer). Typically only 127.0.0.1 is used.
  • 169.254.0.0/16 “Link-local” addresses—used only on a single link and generally assigned automatically.
  • 172.16.0.0/12 Address for private networks (intranets). Such addresses never appear on the public Internet.
  • 192.168.0.0/16 Address for private networks (intranets). Such addresses never appear on the public Internet.
  • 224.0.0.0/4 IPv4 multicast addresses (formerly class D); used only as destination addresses.
  • 240.0.0.0/4 Reserved space (formerly class E), except 255.255.255.255.
  • 255.255.255.255/32 Local network (limited) broadcast address.

Multicast addressing is supported by IPv4 and IPv6. An IP multicast address (also called group or group address) identifies a group of host interfaces, rather than a single one. Most cloud vendors don't allow for multicast and restrict use of communications to unicast from one server to another.

Some of the additional terms that come up are network address translation (NAT), border gateway router (BGP), and firewalls come up around networking discussions. We will defer these conversations to higher layer protocols because they involve more than just the ip address. BGP can be a simple definition that just drops ip addresses and does not pass them outside the corporate network independent of the netmask that the source host uses. If, for example, we want to stop someone from connecting to an ip address outside of our network and force it to go through a firewall or packet filter device a BGP can redirect all traffic through these devices or drop the packets.

In summary, we skimmed over routing. This is a complex subject. We mainly talked about layers 2 and 3 to introduce the terms MAC address, IP address, IPv4, and IPv6. We touched on CIDR and routing tables as well as reserved addresses and BGP and NAT. This is not a complete discussion on these subjects but an introduction of terms. Most cloud vendors do not support multicast or anycast broadcasts inside or outside of their cloud services. Most cloud vendors support IPv4 and IPv6 as well as subnet masking and multiple networks for servers and services. It is important to understand what a router is, how to configure a routing table, and the dangers of creating routing loops. We did not touch on hop count and hop cost because for most cloud implementations the topology is simple and servers inside a cloud implementation is rarely a hop or two away unless you are trying to create a highly available service in another data center, zone, or region. Up next, the data layer and the IP datagram.

TCP/IP Illustrated Vol 1

Mon, 2016-08-08 02:07
Back in 1998 I was working for Sun Microsystems and took an introductory class on networking. One of the big benefits of working for Sun is that it had a very strong affiliation with Stanford University and employees could take classes at no cost. An early rumor was that Sun really stood for Stanford University Networking since two of the founders of the company were living in the Stanford dorms during the early years of Sun. Stanford for years has offered CS 144 - Introduction to Computer Networking. The class is based on Kevin Fall and Richard Steven's book TCP/IP Illustrated Volume 1. I was in an internal training class about cloud services last week and terms and phrases that I remotely remembered kept coming up. As I talked to more and more people, they also knew most of the terms but not all of them. In the next few days we will go through TCP/IP Illustrated and provide a quick tutorial on networking for those of us that have been out of college more than ten years (much more for some of us) and don't work with this on a daily basis.

TCP/IP Illustrated starts out by talking about the history of computer connectivity and the evolution of the 7 layer OSI stack. The seven layers consist of physical (1), link (2), network (3), transport (4), session (5), presentation (6), and application (7). Each of these layers have different protocols, methodologies, and incantations that make them unique and worthy of selection for different problems.

The physical layer is the actual connection between two computers. This might be a copper cable, fiber optic cable, or wireless network. The physical connection media is the definition for this layer. Most of us are familiar with a cable that comes out of the wall, switch, or router and plugs into our server or wifi hub. We are also familiar with a wifi or bluetooth connection that allows us to connect without a physical wire connecting us to other computers. We are not going to focus on this layer but assume that we are wirelessly or ethernet connected to the internet and the cloud servers that we are connecting to are wired to an internet connection. We then use the nebulous internet to route our requests to access our cloud server and responses back to us. This will require higher layers of the stack to make this happen but the default is that we are connected to a network in some manner as well as the server that we want to connect to.

The link or data link layer include protocols for connecting to a link layer and exchanging data. Links can be multi-access layers with more than just two computers talking to each other. WiFi and Ethernet networks are examples of a multi-access layer. We can have more than two computers on these networks and all of them can operate at the same time on the network. Not all of the computers can talk at once but they can time slice the network and share the common physical layer together.

The network or internetwork layer (layer 3) is the protocol layer where we frame packets of information and define communication protocols. Protocols like TCP/IP is defined at this layer. We can put a data analyzer on the physical cable and look at bits streaming by on the wire (or wifi) and decode these packets into data and control blocks. The IP or internet protocol layer is defined here as well as other protocols for creating data packets.

The transport layer (layer 4) is the layer where we describe how data is exchanged and deal with collisions, addresses, and different types of services. TCP, for example, exists at this layer and has protocols for dealing with collisions on the network. If two computers are talking at the same time, bits get overwritten and listeners can not properly read the packets. The TCP layer defines how to request retransmission of data as well as how to avoid collisions in the future for short term. Other protocols like UDP and multicast are defined at this layer that allows us to do things like broadcast messages to all hosts on a network and not wait for a response or acknowledgement. We might want to do this for a video broadcast from a single source where we know that we have one transmitter and multiple receivers on a network.

The session layer (layer 5) are handshaking mechanisms to maintain state between data packets. An example of this would be a cookie in a web browser to maintain a relationship between a client and web server. Server affinity and route preferences are also defined at this layer. If we have a pool of web servers and want to send a client back to the web server that it went to previously, this layer helps create this affinity.

The presentation layer (layer 6) is responsible for format conversions and is typically not manipulated or used for internet protocols or communications.

The application layer (layer 7) is where most of the work is done. A web server, for example, uses http as the communication protocol and defines how screens are painted inside a browser and what files are retrieved from a web server. There are hundreds of layers defined here and we will go into a few examples in future blogs.

If we take an overview of TCP/IP Illustrated Volume I we see that chapter 1 covers the OSI stack and introduces networking and the history of networking as well as layer 1 options. Chapter 2 covers layer 3 and all networking options and touches on the differences between IPv4 and IPv6. Chapter 3 covers the link layer or layer 2 focusing on ethernet, bridges, switches, wireless networks, point to point protocols, and tunneling options. Chapter 4 dives into the ARP protocol which is an implementation of layer 3 used to deal with addressing and computers on a network. Chapter 5 covers the IP definition and discusses packet headers and formats. Chapter 6 goes into addressing more and talks about dynamic host configuration protocol (DHCP) for assigning addresses dynamically. Chapter 7 discusses firewalls and routers as well as network address translations (NAT) concepts. This is the layer that typically gets confusing for cloud vendors and leads to different configurations and options when it comes to protecting servers in the cloud. Chapters 8 and 9 deal with internet control message protocol, broadcasting, and multicasting. Most cloud vendors don't deal with this layer and just prohibit the use of this layer. Chapter 10 focuses on UDP and IP fragmentation. Chapter 11 centers on Domain Naming Services. Each cloud vendor addresses this differently with local and global naming services. We will look at the major cloud vendors and see how they address local naming and name resolution. Chapters 12 through 17 deal with the TCP structure, management, and operation. The Stanford class spent most of the semester on this and ways of optimizing errors and issues. Most cloud vendors do this for you and don't really let you manipulate or modify anything presented in these chapters. The book finishes with Chapter 18 by talking about security in all of its flavors and incantations. We will spend a bit of time talking about this layer since it is of major concern for most users.

In review, we are going to go back and look at networking terms, concepts, and buzzwords so that when someone asks us does this cloud service provide xyz you have a strong context of what they are asking. We are not trying to make everyone a networking expert, just trying to level set the language so that we can compare and contrast services between different cloud vendors.

Pages