Pas Apicella

After installing PHD add on for Pivotal Cloud Foundry 1.1 I quickly created some development services for PHD using the CLI as shown below.

[Tue Apr 15 22:40:08 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hawq-cf free dev-hawq
Creating service dev-hawq in org pivotal / space development as pas...
OK
[Tue Apr 15 22:42:31 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hbase-cf free dev-hbase
Creating service dev-hbase in org pivotal / space development as pas...
OK
[Tue Apr 15 22:44:10 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hive-cf free dev-hive
Creating service dev-hive in org pivotal / space development as pas...
OK
[Tue Apr 15 22:44:22 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-yarn-cf free dev-yarn
Creating service dev-yarn in org pivotal / space development as pas...
OK

Finally using the web console to brow the services in the "Development" space


http://feeds.feedburner.com/TheBlasFromPas

I recently needed to setup a cron script which loaded CSV files from a directory into Greenplum every 2 minutes. Once loaded the files are moved onto Hadoop for archive purposes. The config below shows how to use GPLOAD data load utility which utilises GPFDIST.

1. Create a load table. In this example the data is then moved to the FACT table once the load is complete

  
drop table rtiadmin.rtitrans_etl4;

CREATE TABLE rtiadmin.rtitrans_etl4 (
    imsi character varying(82),
    subscriber_mccmnc character varying(10),
    msisdn character varying(82),
    imei character varying(50),
    called_digits character varying(50),
    start_datetime integer,
    end_datetime integer,
    first_cell_lac integer,
    first_cell_idsac integer,
    current_cell_lac integer,
    current_cell_idsac integer,
    dr_type integer,
    status character varying(50),
    ingest_time bigint,
    processed_time bigint,
    export_time bigint,
    extra_col text,
    gploaded_time timestamp without time zone
)
WITH (appendonly=true) DISTRIBUTED BY (imsi); 

2. GPLOAD yaml file defined as follows

VERSION: 1.0.0.1
DATABASE: mydb
USER: rtiadmin
HOST: 172.1.1.1
PORT: 5432
GPLOAD:
   INPUT:
    - SOURCE:
         LOCAL_HOSTNAME:
            - loadhost
         PORT: 8100
         FILE:
          - /data/rti/stage/run/*.csv
    - COLUMNS:
          - imsi : text
          - subscriber_mccmnc : text
          - msisdn : text
          - imei : text
          - called_digits : text
          - start_datetime : text
          - end_datetime : text
          - first_cell_lac : integer
          - first_cell_idsac : integer
          - current_cell_lac : integer
          - current_cell_idsac : integer
          - dr_type : integer
          - status : text
          - ingest_time : bigint
          - processed_time : bigint
          - export_time : bigint
          - extra_col : text
    - FORMAT: text
    - HEADER: false
    - DELIMITER: ','
    - NULL_AS : ''
    - ERROR_LIMIT: 999999
    - ERROR_TABLE: rtiadmin.rtitrans_etl4_err
   OUTPUT:
    - TABLE: rtiadmin.rtitrans_etl4
    - MODE: INSERT
    - MAPPING:
           imsi : imsi
           subscriber_mccmnc : subscriber_mccmnc
           msisdn : msisdn
           imei : imei
           called_digits : called_digits
           start_datetime : substr(start_datetime, 1, 10)::int
           end_datetime : substr(end_datetime, 1, 10)::int
           first_cell_lac : first_cell_lac
           first_cell_idsac : first_cell_idsac
           current_cell_lac : current_cell_lac
           current_cell_idsac : current_cell_idsac
           dr_type : dr_type
           status : status
           ingest_time : ingest_time
           processed_time : processed_time
           export_time : export_time
           extra_col : extra_col
           gploaded_time : current_timestamp
   PRELOAD:
    - TRUNCATE : true 
    - REUSE_TABLES : true
   SQL:
    - AFTER : "insert into rtitrans select * from rtitrans_etl4" 

3. Call GPLOAD as follows

source $HOME/.bash_profile
gpload -f rtidata.yml


Note: We use the ENV variable as $PGPASSWORD which is used during the load if a password is required which was in this demo
Few things worth noting here.
REUSE_TABLES : This ensures the external tables created during the load are maintained and re-used on next load.
TRUNCATE: This clears the load table prior to load and we use this as we COPY the data once the load is finished into the main FACT table using the "AFTER"http://feeds.feedburner.com/TheBlasFromPas

PCF AWS marketplace provides app direct services and in this example I am going to use the "newrelic" monitoring service to monitor my spring based java application. It's really this simple.

1. Create a service as shown below.

[Tue Mar 04 17:19:34 papicella@:~/cfapps/spring-travel ] $ cf create-service newrelic standard dev-newrelic

2. Create a manifest.yml for my spring application which uses the new relic service above.

applications:
- name: pas-springtravel 
  memory: 1024M 
  instances: 1
  host: pas-springtravel 
  domain: cfapps.io 
  path: ./travel.war
  services:
  - dev-mysql
  - dev-newrelic

3. Push the application

[Tue Mar 04 17:19:34 papicella@:~/cfapps/spring-travel ] $ cf push -f manifest.yml 
Using manifest file manifest.yml

Creating app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

Using route pas-springtravel.cfapps.io
Binding pas-springtravel.cfapps.io to pas-springtravel...
OK

Uploading pas-springtravel...
Uploading from: travel.war
5.3M, 2748 files
OK
Binding service dev-mysql to pas-springtravel in org papicella-org / space development as papicella@gopivotal.com
OK
Binding service dev-newrelic to pas-springtravel in org papicella-org / space development as papicella@gopivotal.com
OK

Starting app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK
-----> Downloaded app package (22M)
-----> Uploading droplet (67M)

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: pas-springtravel.cfapps.io

     state     since                    cpu    memory         disk           
#0   running   2014-03-04 05:24:43 PM   0.0%   610.7M of 1G   155.9M of 1G 

4. Under the services listed on AWS click on "Manage" and here are some screen shots of what the newrelic monitoring service provides with just a simple BIND when we pushed the application.





http://feeds.feedburner.com/TheBlasFromPas

I previously showed how to create a connection in IntelliJ IDEA to Cloud Foundry v2 in the post below.

http://theblasfrompas.blogspot.com.au/2014/02/intellij-idea-version-13-now-includes.html

With a Cloud Foundry CLOUD connection we can now PUSH our application directly from the IDE as shown below.

1. Create a run configuration for your project as shown below. We also specify the memory and number of instances on this page as part of the push  / deployment process.



2. Select the created run configuration and deploy the application, output as follows


3. View the deployed application on AWS hosted instance of Cloud Foundry



http://feeds.feedburner.com/TheBlasFromPas

Just installed IntelliJ IDEA version 13 and found that it now includes a CloudFoundry connection type. You define it under IDE settings as shown below.


Will test deploying to the publicly hosted AWS Cloud Foundry using this connection at some stage.http://feeds.feedburner.com/TheBlasFromPas

By creating a manifest as follows we can push multiple applications in one go as shown below.

1. manifest.yml

applications:
- name: pas-props
  memory: 256M
  instances: 1
  host: pas-props
  domain: cfapps.io
  path: ./props.war
- name: pas-httpsession
  memory: 256M
  instances: 1
  host: pas-httpsession
  domain: cfapps.io
  path: ./haclusterdemo.war

2. Push as follows

[Thu Feb 20 14:48:45 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/other ] $ cf push -f manifest-twoapps.yml
Using manifest file manifest-twoapps.yml

Creating app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

Using route pas-props.cfapps.io
Binding pas-props.cfapps.io to pas-props...
OK

Uploading pas-props...
Uploading from: props.war
2.7K, 5 files
OK

Starting app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

1 of 1 instances running

App started

Showing health and status for app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: pas-props.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-20 02:51:14 PM   0.0%   193.2M of 256M   110.7M of 1G   
Creating app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

Creating route pas-httpsession.cfapps.io...
OK

Binding pas-httpsession.cfapps.io to pas-httpsession...
OK

Uploading pas-httpsession...
Uploading from: haclusterdemo.war
130.2K, 10 files
OK

Starting app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: pas-httpsession.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-20 02:53:53 PM   2.0%   214.3M of 256M   113.3M of 1G   

More Information

http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.htmlhttp://feeds.feedburner.com/TheBlasFromPas

I previously blogged how to PUSH an application into PCF , then bind services as shown below.
http://theblasfrompas.blogspot.com.au/2014/02/i-decided-to-install-new-cf-v6-cli-for.html
If you want to do this in one step simply create a manifest file as shown below. This example isn't the spring books application but shows you what your manifest.yml would look like with services included so you could then create one similar to that below.
1. Create a manifest.yml file as follows
manifest.yml
applications:- name: pas-springtravel   memory: 512M   instances: 1  host: pas-springtravel   domain: cfapps.io   path: ./travel.war  services:  - dev-mysql
2. Push the application and bind to the services at push time.
[Tue Feb 18 20:13:01 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-travel ] $ cf push -f manifest.yml Using manifest file manifest.yml
Creating app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...OK
Creating route pas-springtravel.cfapps.io...OK
Binding pas-springtravel.cfapps.io to pas-springtravel...OK
Uploading pas-springtravel...Uploading from: travel.war5.3M, 2748 filesOKBinding service dev-mysql to pas-springtravel in org papicella-org / space development as papicella@gopivotal.comOK
Starting app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...OK-----> Downloaded app package (22M)-----> Uploading droplet (62M)
0 of 1 instances running, 1 starting0 of 1 instances running, 1 starting1 of 1 instances running
App started
Showing health and status for app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...OK
requested state: startedinstances: 1/1usage: 512M x 1 instancesurls: pas-springtravel.cfapps.io
     state     since                    cpu    memory           disk           #0   running   2014-02-18 08:24:58 PM   0.5%   458.6M of 512M   149.9M of 1G   
3. The application using the MYSQL service

http://feeds.feedburner.com/TheBlasFromPas

I decided to install the new CF v6 CLI for PCF (Pivotal Cloud Foundry). Given this will be the new CLI for PCF here is how it differs to the old Ruby CLI. In this example I deploy / push a spring based application and bind it to existing services.

To download and install the new CLI v6 use the link below.

http://docs.cloudfoundry.com/docs/using/managing-apps/cf/go-cli.html

In this example we have the following 2 services that already exist and I am using the PUBLIC hosted AWS version of PCF at https://console.run.pivotal.io/

• dev-mysql
• dev-redis

1. First lets deploy our application and when we do we DON'T actually start it at this point.

[Sun Feb 16 22:23:22 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf push pas-books -d cfapps.io -i 1 -m 512M -p springmvc31.war --no-start
Creating app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

Creating route pas-books.cfapps.io...
OK

Binding pas-books.cfapps.io to pas-books...
OK

Uploading pas-books...
Uploading from: /Users/papicella/vmware/pivotal/products/cloud-foundry/apps/spring-books/springmvc31.war
511.2K, 44 files
OK

2. Bind the application to the 2 services above.

[Sun Feb 16 22:25:56 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf bind-service pas-books dev-mysql
Binding service dev-mysql to app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK
TIP: Use 'cf push' to ensure your env variable changes take effect

[Sun Feb 16 22:25:56 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf bind-service pas-books dev-redis
Binding service dev-redis to app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK
TIP: Use 'cf push' to ensure your env variable changes take effect

3. Push the application again and this time start it.

[Sun Feb 16 22:29:26 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf push pas-books -p springmvc31.war
Updating app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

Uploading pas-books...
Uploading from: /Users/papicella/vmware/pivotal/products/cloud-foundry/apps/spring-books/springmvc31.war
511.2K, 44 files
OK

Starting app pas-books in org papicella-org / space development as papicella@gopivotal.com...
-----> Downloaded app package (19M)
OK
-----> Java Buildpack source: system
-----> Downloading Open JDK 1.7.0_51 from http://download.pivotal.io.s3.amazonaws.com/openjdk/lucid/x86_64/openjdk-1.7.0_51.tar.gz 
(1.2s)
       Expanding Open JDK to .java-buildpack/open_jdk 
(1.2s)
-----> Downloading Spring Auto Reconfiguration 0.8.7 from http://download.pivotal.io.s3.amazonaws.com/auto-reconfiguration/auto-reconfiguration-0.8.7.jar 
(1.4s)
       Modifying /WEB-INF/web.xml for Auto Reconfiguration
-----> Downloading Tomcat 7.0.50 from http://download.pivotal.io.s3.amazonaws.com/tomcat/tomcat-7.0.50.tar.gz 
(0.3s)
       Expanding Tomcat to .java-buildpack/tomcat 
(0.1s)
-----> Downloading Buildpack Tomcat Support 1.1.1 from http://download.pivotal.io.s3.amazonaws.com/tomcat-buildpack-support/tomcat-buildpack-support-1.1.1.jar 
(0.0s)
-----> Uploading droplet (60M)

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: pas-books.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-16 10:31:13 PM   0.0%   494.9M of 512M   134.2M of 1G   


If all went well we should see something as follows.


And of course a running application


http://feeds.feedburner.com/TheBlasFromPas

In this example below I show how you would create a user defined service for Oracle 11g R2 which is running outside of the PCF instance and how your applications would bind to that service. The example below is from the spring-music sample application which is a spring based web application.

https://github.com/cloudfoundry-samples/spring-music

1. First lets create our user defined service for Oracle , here we just use a simple URI which the Spring Cloud library knows how to use.

c:\pivotal\spring-music-oracle>cf create-service user-provided
Name?> oracle11gr2

What credential parameters should applications use to connect to this service instance?
(e.g. hostname, port, password)> uri

uri> oracle://scott:tiger@192.168.110.209:1521/XE

Creating service oracle11gr2... OK

c:\pivotal\spring-music-oracle>


The spring cloud project is here.

https://github.com/spring-projects/spring-cloud

2. Now with a previously created WAR file lets push the spring music app to PCF and bind to our oracle service as shown below.

c:\pivotal\spring-music-oracle>cf push --path=spring-music-oracle.war
Name> spring-music-oracle

Instances> 1

1: 128M
2: 256M
3: 512M
4: 1G
Memory Limit> 3

Creating spring-music-oracle... OK

1: spring-music-oracle
2: none
Subdomain> spring-music-oracle

1: system.com
2: none
Domain> system.com

Binding spring-music-oracle.system.com to spring-music-oracle... OK

Create services for application?> n

Bind other services to application?> y

1: mysql-dev
2: rmq
3: user-provided-705bf
4: oracle11gr2
Which service?> 4

Binding oracle11gr2 to spring-music-oracle... OK
Bind another service?> n

Save configuration?> n

Uploading spring-music-oracle... OK
Preparing to start spring-music-oracle... OK
-----> Downloaded app package (22M)
-----> Downloading OpenJDK 1.7.0_51 from http://download.pivotal.io.s3.amazonaws.com/openjdk/lucid/x86_64/openjdk-1.7.0_
51.tar.gz (1m 52s)
       Expanding JRE to .java (2.4s)
-----> Downloading Spring Auto-reconfiguration 0.8.4 from http://download.pivotal.io.s3.amazonaws.com/auto-reconfigurati
on/auto-reconfiguration-0.8.4.jar (4.3s)
       Modifying /WEB-INF/web.xml for Auto Reconfiguration
-----> Downloading Tomcat 7.0.50 from http://download.pivotal.io.s3.amazonaws.com/tomcat/tomcat-7.0.50.tar.gz (27.9s)
       Expanding Tomcat to .tomcat (0.3s)
-----> Downloading Buildpack Tomcat Support 1.1.1 from http://download.pivotal.io.s3.amazonaws.com/tomcat-buildpack-supp
ort/tomcat-buildpack-support-1.1.1.jar (0.2s)
-----> Uploading droplet (63M)
Checking status of app 'spring-music-oracle'...
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  0 of 1 instances running (1 starting)
  1 of 1 instances running (1 running)
Push successful! App 'spring-music-oracle' available at spring-music-oracle.system.com

c:\pivotal\spring-music-oracle>


3. Here are some screenshots showing it deployed , invoking the application, and data within the Oracle 11g r2 schema.






The code to make this happen for Oracle is as follows, we do this to make use of the spring cloud code to bind to the user defined service "oracle11gr2" we created above. The spring cloud Library enables cloud applications to connect to services. The classes below show how we enable our oracle service for the spring music application.

OracleServiceInfo.java

  
package org.cloudfoundry.samples.music.cloud;

import org.springframework.cloud.service.ServiceInfo.ServiceLabel;
import org.springframework.cloud.service.common.RelationalServiceInfo;

@ServiceLabel("oracle")
public class OracleServiceInfo extends RelationalServiceInfo {

    public OracleServiceInfo(String id, String url) 
    {       
     super(id, url, "oracle");
    }

 @Override
 public String getJdbcUrl() 
 {
  return String.format("jdbc:%s:thin:%s/%s@%s:%d/%s",
                              "oracle",
                              super.getUserName(), super.getPassword(),
                              super.getHost(), super.getPort(), super.getPath());
 }
      
} 

OracleDataSourceCreator.java

  
package org.cloudfoundry.samples.music.cloud;

import org.cloudfoundry.samples.music.cloud.OracleServiceInfo;
import org.springframework.cloud.service.relational.DataSourceCreator;

public class OracleDataSourceCreator extends DataSourceCreator<OracleServiceInfo> {
    private static final String[] DRIVERS = new String[]{"oracle.jdbc.OracleDriver"};

    private static final String VALIDATION_QUERY = "SELECT 'Y' from dual";

    public OracleDataSourceCreator() {
     super("spring-cloud.oracle.driver", DRIVERS, VALIDATION_QUERY);
    }
} 

UserProvidedOracleServiceInfoCreator.java

  
package org.cloudfoundry.samples.music.cloud;

import org.springframework.cloud.cloudfoundry.CloudFoundryServiceInfoCreator;

import java.util.Map;

public class UserProvidedOracleServiceInfoCreator extends CloudFoundryServiceInfoCreator<OracleServiceInfo> {
    public UserProvidedOracleServiceInfoCreator() {
        super("user-provided");
    }

    @Override
    @SuppressWarnings("unchecked")
    public boolean accept(Map<String, Object> serviceData) {
        String label = (String) serviceData.get("label");

        if (label.equals(getTag())) {
            Map<String, Object> credentials = (Map<String, Object>) serviceData.get("credentials");
            String uri = (String) credentials.get("uri");

            if (uri != null && uri.startsWith("oracle:")) {
                return true;
            }
        }

        return false;
    }

    @Override
    @SuppressWarnings("unchecked")
    public OracleServiceInfo createServiceInfo(Map<String, Object> serviceData) {
        String id = (String) serviceData.get("name");
        Map<String, Object> credentials = (Map<String, Object>) serviceData.get("credentials");
        String uri = credentials.get("uri").toString();
        return new OracleServiceInfo(id, uri);
    }
}  

OracleRepositoryConfig.java

  
package org.cloudfoundry.samples.music.config.data;

import org.hibernate.dialect.MySQL5Dialect;
import org.hibernate.dialect.Oracle10gDialect;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;

@Configuration
@Profile("oracle")
@EnableJpaRepositories("org.cloudfoundry.samples.music.repositories.jpa")
public class OracleRepositoryConfig extends AbstractJpaRepositoryConfig {

    protected String getHibernateDialect() {
        return Oracle10gDialect.class.getName();
    }

}

RelationalCloudDataSourceConfig.java

  
package org.cloudfoundry.samples.music.config.data;

import org.springframework.cloud.config.java.AbstractCloudConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

import javax.sql.DataSource;

@Configuration
@Profile({"mysql-cloud", "postgres-cloud", "oracle-cloud"})
public class RelationalCloudDataSourceConfig extends AbstractCloudConfig {

    @Bean
    public DataSource dataSource() {
        return connectionFactory().dataSource();
    }

} 

src/main/resources/META-INF/servicesorg.springframework.cloud.cloudfoundry.CloudFoundryServiceInfoCreator

org.cloudfoundry.samples.music.cloud.UserProvidedMysqlServiceInfoCreator
org.cloudfoundry.samples.music.cloud.UserProvidedOracleServiceInfoCreator

src/main/resources/META-INF/services/org.springframework.cloud.service.ServiceConnectorCreator

org.springframework.cloud.service.relational.MysqlDataSourceCreator
org.cloudfoundry.samples.music.cloud.OracleDataSourceCreator
org.springframework.cloud.service.relational.PostgresqlDataSourceCreator
org.springframework.cloud.service.keyval.RedisConnectionFactoryCreator
org.springframework.cloud.service.document.MongoDbFactoryCreator
org.springframework.cloud.service.messaging.RabbitConnectionFactoryCreator
org.springframework.cloud.service.smtp.MailSenderCreator

Finally you need to ensure that ojdbc6.jar is shipped with the WAR file within "WEB-INF/lib" which is the Oracle JDBC driver to enable the application to connect to oracle.

More Information on Pivotal CF

http://www.gopivotal.com/products/cloud-foundry



http://feeds.feedburner.com/TheBlasFromPas

Thymeleaf offers a set of Spring integrations that allow you to use it as a full-featured substitute for JSP in Spring MVC applications.

So here is a simple example just so we can compare how things differ in HTML versus JSP, does remind me of JSF / Viewlets to be honest.

1. Let start with what our simple project looks like in IntelliJ


2. The web.xml is defined as follows

  
<web-app version="2.4"
 xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee 
 http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">

 <display-name>Spring MVC Application</display-name>

    <servlet>
  <servlet-name>mvc-dispatcher</servlet-name>
  <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
 </servlet>

    <servlet-mapping>
        <servlet-name>mvc-dispatcher</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

</web-app> 

3. The spring context XML is defined as follows

WEB-INF/mvc-dispatcher-servlet.xml

  
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <context:component-scan base-package="pivotal.au.pas.springapp.mvc"/>

    <bean id="templateResolver"
          class="org.thymeleaf.templateresolver.ServletContextTemplateResolver">
        <property name="prefix" value="/WEB-INF/resources/templates/" />
        <property name="suffix" value=".html" />
        <property name="templateMode" value="HTML5" />
    </bean>

    <bean id="templateEngine"
          class="org.thymeleaf.spring3.SpringTemplateEngine">
        <property name="templateResolver" ref="templateResolver" />
    </bean>

    <bean class="org.thymeleaf.spring3.view.ThymeleafViewResolver">
        <property name="templateEngine" ref="templateEngine" />
    </bean>

</beans>  

4. Now the PeopleController.java class looks just like it would when rendering a JSP page view and so it should all we have done here is change what the view page will be which in this case is HTML itself. I have created some mock up data , rather then connecting to a RDBMS here.

  
package pivotal.au.pas.springapp.mvc;

import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import pivotal.au.pas.springapp.beans.Person;

import java.util.ArrayList;
import java.util.List;

@Controller
public class PeopleController
{
    private static List<Person> people = null;

    static
    {
        people = new ArrayList<Person>();
        people.add(new Person(1, "pas"));
        people.add(new Person(2, "lucia"));
        people.add(new Person(3, "lucas"));
        people.add(new Person(4, "siena"));
    }

    @RequestMapping("/showallpeople")
    public String showallpeople(Model model)
    {
        model.addAttribute("people", people);
        return "people";
    }
}    

5. The view page is just a HTML page as follows

WEB-INF/resources/templates/people.html

  
<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>Spring MVC with Thymeleaf - All people</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h2> Spring MVC with Thymeleaf - All people </h2>

<table border="1">
    <thead>
    <tr>
        <th>Id#</th>
        <th>Name</th>
    </tr>
    </thead>
    <tbody>
    <tr th:each="var : ${people}">
        <td th:text="${var.id}"></td>
        <td th:text="${var.name}"></td>
    </tr>
    </tbody>
</table>

<br />
<hr />
<br />
<address>
    <a href="mailto:papicella@gopivotal.com">Pas Apicella</a>
</address>
</body>
</html>

6. So here is a simple JSP page as the entry page, could of been a HTML page.

index.jsp

  
<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<html>
<head>
    <title>Spring MVC with Thymeleaf</title>
</head>
<body>
<h2>Spring MVC with Thymeleaf</h2>
<ul>
    <li><a href="/greeting">Greeting</a></li>
    <li><a href="/showallpeople">Show All People</a></li>
</ul>
<hr />
<br />
<address>
    <a href="mailto:papicella@gopivotal.com">Pas Apicella</a>
</address>
</body>
</html>   

7. Run index.jsp

8. Finally when run and I click on "Show All People" link we get display as follows


More Information

http://spring.io/guides/gs/serving-web-content/

http://www.thymeleaf.org/index.html

http://feeds.feedburner.com/TheBlasFromPas

Spring XD comes with several batch import and export modules. You can run them out of the box and show using the "Import CSV Files to JDBC (filejdbc)" as shown below.

This example show's inserting data into GemFireXD, you can use any RDBMS which supports a JDBC driver.

Note: XD_BASE = /Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4

1. Configure connection to RDBMS in "$XD_BASE/xd/config/batch-jdbc-import.properties".

# Setting for the JDBC batch import job module
driverClass=com.vmware.sqlfire.jdbc.ClientDriver
url=jdbc:sqlfire://10.32.240.113:1527
username=APP
password=APP

2. Add jdbc jar file to $XD_BASE/xd/lib directory, in this case GemFireXD which uses "sqlfireclient.jar"

3. Start SpringXD single node using "./$XD_BASE/xd/bin/xd-singlenode"

4. Create a file called people.csv with contents as follows

[Fri Jan 10 14:35:00 papicella@:~/vmware/software/spring/spring-xd-1.0.0.M4/files ] $ cat people.csv
1,pas
2,lucia
3,lucas
4,siena

5. Log into SpringXD shell as shown below using "$XD_BASE/shell/bin/xd-shell"
6. Create table in GemFireXD as shown below.

  
sqlf> create table people (id int, name varchar(20));
0 rows inserted/updated/deleted 

7. Create a JOB as shown below

xd:>job create myjob --definition "filejdbc --resources=file:/Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4/files/*.csv --names=id,name --tableName=people"Successfully created and deployed job 'myjob'
8. Start the JOB

xd:>job launch myjob
Successfully launched the job 'myjob'

9. Verify that the CSV data has been inserted into the table PEOPLE in GemFireXD

  
sqlf> select * from people;
ID         |NAME                
--------------------------------
2          |lucia               
1          |pas                 
4          |siena               
3          |lucas               

4 rows selected 

More Information

For more information on SpringXD or GemFireXD see the links below.

http://projects.spring.io/spring-xd/ - SpringXD

http://gopivotal.com/products/pivotal-hd - GemFireXD Betahttp://feeds.feedburner.com/TheBlasFromPas

The following examples shows what we can do out of the box with SpringXD TAIL input source and JDBC sink as the output source. This can all be done without writing and code.

Assumes you have SpringXD installed and GemFireXD installed. See the getting stared guide which explains how to setup SpringXD. As for a database output for JDBC you can use any RDBMS here so you could use MYSQL , ORACLE etc if you don't have GemFireXD installed.

Note: XD_BASE = /Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4

1. Configure connection to RDBMS in "$XD_BASE/xd/config/jdbc.properties".

driverClass=com.vmware.sqlfire.jdbc.ClientDriver
url=jdbc:sqlfire://10.32.240.113:1527
username=APP
password=APP

2. Add jdbc jar file to $XD_BASE/xd/lib directory, in this case GemFireXD which uses "sqlfireclient.jar"

3. Start SpringXD single node using "./$XD_BASE/xd/bin/xd-singlenode"

./$XD_BASE/xd/bin/xd-singlenode
.....

11:29:00,417  INFO main container.XDContainer:175 - started container: 0
11:29:00,424  INFO main launcher.LocalContainerLauncher:95 - 
 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
1.0.0.M4                         eXtreme Data

Using local mode JMX is disabled for XD components
XD Configuration:
XD_HOME=/Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4/xd
XD_TRANSPORT=local
XD_STORE=memory
XD_ANALYTICS=memory
XD_HADOOP_DISTRO=hadoop12

started container : 85631@localhost
Documentation: https://github.com/SpringSource/spring-xd/wiki

4. Create a empty file that the stream will read from using TAIL as the input

> touch filetest

eg:

[Fri Jan 10 11:29:06 papicella@:/tmp/xd/input ] $ ls -la /tmp/xd/input/filetest 
-rw-r--r--  1 papicella  wheel  0 10 Jan 11:29 /tmp/xd/input/filetest

5. Log into SpringXD shell as shown below using "$XD_BASE/shell/bin/xd-shell"
[Fri Jan 10 10:46:39 papicella@:~/vmware/software/spring/spring-xd-1.0.0.M4/shell/bin ] $ ./xd-shell _____                           __   _______/  ___|          (-)             \ \ / /  _  \\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | | `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/      | |                  __/ |      |_|                 |___/eXtreme Data1.0.0.M4 | Admin Server Target: http://localhost:9393Welcome to the Spring XD shell. For assistance hit TAB or type "help".
6. Create a STREAM as shown below.

xd:> stream create --name filetest --definition "tail --name=/tmp/xd/input/filetest |  jdbc --columns=id,name"Created new stream 'filetest'
7. Create table in database as follows.

  
sqlf> create table filetest (id int, name varchar(20));

8. Create file as follows called "testfile" we will copy this to stream file soon.
[Fri Jan 10 11:38:44 papicella@:~/vmware/software/spring/spring-xd-1.0.0.M4 ] $ cat filetest{"id":"1","name":"pas"}{"id":"2","name":"lucia"}{"id":"3","name":"lucas"}{"id":"4","name":"siena"}
7. Copy file "filetest" above to /tmp/xd/input/filetest
9. Check base table in RDBMS for data inserts.

  
sqlf> select * from filetest;
ID         |NAME                
--------------------------------
4          |siena               
1          |pas                 
3          |lucas               
2          |lucia               

4 rows selected

More Information

For more information on SpringXD or GemFireXD see the links below.

http://projects.spring.io/spring-xd/ - SpringXD

http://gopivotal.com/products/pivotal-hd - GemFireXD Beta



http://feeds.feedburner.com/TheBlasFromPas