Senthil Rajendran

In BIG DATA HADOOP Testing with MapReduce Examples Part 1 and BIG DATA HADOOP Testing with MapReduce Examples Part 2 I have resolved some of the issues in getting the HADOOP running but still I have some issues left over and this time it is "Invalid shuffle port number -1 returned" when the mapreduce jobs are submitted.
hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output 13/03/13 14:59:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 13/03/13 14:59:40 INFO input.FileInputFormat: Total input paths to process : 1 13/03/13 14:59:41 INFO mapreduce.JobSubmitter: number of splits:1 13/03/13 14:59:41 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 13/03/13 14:59:41 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/03/13 14:59:41 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 13/03/13 14:59:41 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 13/03/13 14:59:41 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 13/03/13 14:59:41 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 13/03/13 14:59:41 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/03/13 14:59:41 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/03/13 14:59:41 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/03/13 14:59:41 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/03/13 14:59:41 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/03/13 14:59:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363184126427_0001 13/03/13 14:59:41 INFO client.YarnClientImpl: Submitted application application_1363184126427_0001 to ResourceManager at /0.0.0.0:8032 13/03/13 14:59:42 INFO mapreduce.Job: The url to track the job: http://bigdataserver1:8088/proxy/application_1363184126427_0001/ 13/03/13 14:59:42 INFO mapreduce.Job: Running job: job_1363184126427_0001 13/03/13 14:59:53 INFO mapreduce.Job: Job job_1363184126427_0001 running in uber mode : false 13/03/13 14:59:53 INFO mapreduce.Job:  map 0% reduce 0% 13/03/13 14:59:54 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_0, Status : FAILED Container launch failed for container_1363184126427_0001_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_0         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)         at java.lang.Thread.run(Thread.java:619)
13/03/13 14:59:55 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_1, Status : FAILED Container launch failed for container_1363184126427_0001_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_1         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)         at java.lang.Thread.run(Thread.java:619)
13/03/13 14:59:57 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_2, Status : FAILED Container launch failed for container_1363184126427_0001_01_000004 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_2         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)         at java.lang.Thread.run(Thread.java:619)
13/03/13 14:59:59 INFO mapreduce.Job:  map 100% reduce 0% 13/03/13 14:59:59 INFO mapreduce.Job: Job job_1363184126427_0001 failed with state FAILED due to: Task failed task_1363184126427_0001_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0
13/03/13 15:00:00 INFO mapreduce.Job: Counters: 4         Job Counters                 Other local map tasks=3                 Data-local map tasks=1                 Total time spent by all maps in occupied slots (ms)=0                 Total time spent by all reduces in occupied slots (ms)=0 hadoop@bigdataserver1:~/hadoop>

Solution is to update yarn-site.xml with the values below and then restart the HADOOP cluster.













hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output4 13/03/13 15:23:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/03/13 15:23:15 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/03/13 15:23:15 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 13/03/13 15:23:16 INFO input.FileInputFormat: Total input paths to process : 1 13/03/13 15:23:16 INFO mapreduce.JobSubmitter: number of splits:1 13/03/13 15:23:16 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 13/03/13 15:23:16 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/03/13 15:23:16 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 13/03/13 15:23:16 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 13/03/13 15:23:16 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 13/03/13 15:23:16 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 13/03/13 15:23:16 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/03/13 15:23:16 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/03/13 15:23:16 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/03/13 15:23:16 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/03/13 15:23:16 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/03/13 15:23:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363188167312_0001 13/03/13 15:23:17 INFO client.YarnClientImpl: Submitted application application_1363188167312_0001 to ResourceManager at /0.0.0.0:8032 13/03/13 15:23:17 INFO mapreduce.Job: The url to track the job: http://bigdataserver1:8088/proxy/application_1363188167312_0001/ 13/03/13 15:23:17 INFO mapreduce.Job: Running job: job_1363188167312_0001 13/03/13 15:23:27 INFO mapreduce.Job: Job job_1363188167312_0001 running in uber mode : false 13/03/13 15:23:27 INFO mapreduce.Job:  map 0% reduce 0% 13/03/13 15:23:32 INFO mapreduce.Job:  map 100% reduce 0% 13/03/13 15:23:38 INFO mapreduce.Job:  map 100% reduce 100% 13/03/13 15:23:38 INFO mapreduce.Job: Job job_1363188167312_0001 completed successfully 13/03/13 15:23:38 INFO mapreduce.Job: Counters: 43         File System Counters                 FILE: Number of bytes read=2369                 FILE: Number of bytes written=140677                 FILE: Number of read operations=0                 FILE: Number of large read operations=0                 FILE: Number of write operations=0                 HDFS: Number of bytes read=1474                 HDFS: Number of bytes written=1631                 HDFS: Number of read operations=6                 HDFS: Number of large read operations=0                 HDFS: Number of write operations=2         Job Counters                 Launched map tasks=1                 Launched reduce tasks=1                 Data-local map tasks=1                 Total time spent by all maps in occupied slots (ms)=3851                 Total time spent by all reduces in occupied slots (ms)=4999         Map-Reduce Framework                 Map input records=200                 Map output records=199                 Map output bytes=2165                 Map output materialized bytes=2369                 Input split bytes=104                 Combine input records=199                 Combine output records=183                 Reduce input groups=183                 Reduce shuffle bytes=2369                 Reduce input records=183                 Reduce output records=183                 Spilled Records=366                 Shuffled Maps =1                 Failed Shuffles=0                 Merged Map outputs=2                 GC time elapsed (ms)=57                 CPU time spent (ms)=2730                 Physical memory (bytes) snapshot=358436864                 Virtual memory (bytes) snapshot=926806016                 Total committed heap usage (bytes)=303431680         Shuffle Errors                 BAD_ID=0                 CONNECTION=0                 IO_ERROR=0                 WRONG_LENGTH=0                 WRONG_MAP=0                 WRONG_REDUCE=0         File Input Format Counters                 Bytes Read=1370         File Output Format Counters                 Bytes Written=1631 hadoop@bigdataserver1:~/hadoop>


Finally I have verified my HADOOP cluster.

In BIG DATA HADOOP Testing with MapReduce Examples Part 1 the map reduce example was not complete. I ran the wordcount with the proper lower case to see what happens and again it error-ed.


hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:58:06 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "localhost:9001"
13/03/13 14:58:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
        at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:83)
        at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:76)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1188)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1184)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1183)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
hadoop@bigdataserver1:~/hadoop>

Solution is to update configuration file mapred-site.xml with the correct property values.


















Running again results in another round of errors


hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:59:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/03/13 14:59:40 INFO input.FileInputFormat: Total input paths to process : 1
13/03/13 14:59:41 INFO mapreduce.JobSubmitter: number of splits:1
13/03/13 14:59:41 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/03/13 14:59:41 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/03/13 14:59:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363184126427_0001
13/03/13 14:59:41 INFO client.YarnClientImpl: Submitted application application_1363184126427_0001 to ResourceManager at /0.0.0.0:8032
13/03/13 14:59:42 INFO mapreduce.Job: The url to track the job: http://bigdataserver1.f1:8088/proxy/application_1363184126427_0001/
13/03/13 14:59:42 INFO mapreduce.Job: Running job: job_1363184126427_0001
13/03/13 14:59:53 INFO mapreduce.Job: Job job_1363184126427_0001 running in uber mode : false
13/03/13 14:59:53 INFO mapreduce.Job:  map 0% reduce 0%
13/03/13 14:59:54 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_0, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_0
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:55 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_1, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_1
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:57 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_2, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000004 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_2
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:59 INFO mapreduce.Job:  map 100% reduce 0%
13/03/13 14:59:59 INFO mapreduce.Job: Job job_1363184126427_0001 failed with state FAILED due to: Task failed task_1363184126427_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

13/03/13 15:00:00 INFO mapreduce.Job: Counters: 4
        Job Counters
                Other local map tasks=3
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=0
                Total time spent by all reduces in occupied slots (ms)=0
hadoop@bigdataserver1:~/hadoop>


Hunting for fix

hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar - jar file for testing hadoop

wordcount example reads text files and counts how often words occur and here I am passing the name.txt which was copied to the HDFS


hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:58:06 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "localhost:9001"
13/03/13 14:58:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
        at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:83)
        at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:76)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1188)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1184)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1183)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
hadoop@bigdataserver1:~/hadoop>

Solution to resolve the above error was to source the HADOOP_MAPRED_HOME in the hadoop-env.sh file.


Ran again and it resulted in another error


hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
        at java.lang.Class.getMethod0(Class.java:2670)
        at java.lang.Class.getMethod(Class.java:1603)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.(ProgramDriver.java:60)
        at org.apache.hadoop.util.ProgramDriver.addClass(ProgramDriver.java:103)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.partition.InputSampler$Sampler
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        ... 12 more
hadoop@bigdataserver1:~/hadoop>

Solution is to source the mapreduce classpath in the hadoop-env.sh file.



# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=/home/hadoop/hadoop/share/hadoop/mapreduce/*:$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done


hadoop@bigdataserver1:~/hadoop> hadoop classpath /home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/common/*:/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/* hadoop@bigdataserver1:~/hadoop> ls /home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/* /bin/ls: /home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*: No such file or directory hadoop@bigdataserver1:~/hadoop> pwd /home/hadoop/hadoop hadoop@bigdataserver1:~/hadoop> echo $CLASSPATH
hadoop@bigdataserver1:~/hadoop> vi etc/hadoop/hadoop-env.sh hadoop@bigdataserver1:~/hadoop> echo $HADOOP_HOME
hadoop@bigdataserver1:~/hadoop> export HADOOP_HOME=/home/hadoop/hadoop hadoop@bigdataserver1:~/hadoop> $HADOOP_HOME/contrib/capacity-scheduler/*.jar hadoop@bigdataserver1:~/hadoop> ls $HADOOP_HOME/contrib/capacity-scheduler/*.jar /bin/ls: /home/hadoop/hadoop/contrib/capacity-scheduler/*.jar: No such file or directory hadoop@bigdataserver1:~/hadoop> echo $HADOOP_CLASSPATH
hadoop@bigdataserver1:~/hadoop> ls /home/hadoop/hadoop/share/hadoop/mapreduce hadoop-mapreduce-client-app-2.0.0-cdh4.2.0.jar     hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0.jar        lib hadoop-mapreduce-client-common-2.0.0-cdh4.2.0.jar  hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0-tests.jar  lib-examples hadoop-mapreduce-client-core-2.0.0-cdh4.2.0.jar    hadoop-mapreduce-client-shuffle-2.0.0-cdh4.2.0.jar hadoop-mapreduce-client-hs-2.0.0-cdh4.2.0.jar      hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar hadoop@bigdataserver1:~/hadoop>

hadoop@bigdataserver1:~/hadoop> vi etc/hadoop/hadoop-env.sh update class path
hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar WordCount /bigdata1/name.txt /bigdata1/output Unknown program 'WordCount' chosen. Valid program names are:   aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.   aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.   bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.   dbcount: An example job that count the pageview counts from a database.   distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.   grep: A map/reduce program that counts the matches of a regex in the input.   join: A job that effects a join over sorted, equally partitioned datasets   multifilewc: A job that counts words from several files.   pentomino: A map/reduce tile laying program to find solutions to pentomino problems.   pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.   randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.   randomwriter: A map/reduce program that writes 10GB of random data per node.   secondarysort: An example defining a secondary sort to the reduce.   sort: A map/reduce program that sorts the data written by the random writer.   sudoku: A sudoku solver.   teragen: Generate data for the terasort   terasort: Run the terasort   teravalidate: Checking results of terasort   wordcount: A map/reduce program that counts the words in the input files. hadoop@bigdataserver1:~/hadoop>


Looks positive that the mapreduce is working but with a wrong syntax.

Here is a simple step to copy a file into HDFS Hadoop File System.

First Create a Directory



hadoop@bigdataserver1:> hadoop fs -mkdir /bigdata1
hadoop@bigdataserver1:>

Copy the a sample file

hadoop@bigdataserver1:> hadoop fs -put /home/hadoop/hadoop/bigdata/name.txt /bigdata1
hadoop@bigdataserver1:>

Let us check if we can see the directory

hadoop@bigdataserver1:> hadoop fs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2013-03-13 13:14 /bigdata1
hadoop@bigdataserver1:>

Let us check if we can see the file

hadoop@bigdataserver1:> hadoop fs -ls /bigdata1
Found 1 items
-rw-r--r--   1 hadoop supergroup       1370 2013-03-13 13:14 /bigdata1/name.txt
hadoop@bigdataserver1:>

When using start-all.sh , I got the message "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"

So it is clear that start-all.sh should not be user and the same hold true with stop-all.sh

Using start-dfs.sh it gives me JAVA_HOME is not set


hadoop@bigdataserver1:~/hadoop/sbin> sh start-dfs.sh
which: no start-dfs.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin)
13/03/13 11:44:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Error: JAVA_HOME is not set and could not be found.
13/03/13 11:45:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop@bigdataserver1:~/hadoop/sbin> echo $JAVA_HOME
/home/hadoop/jdk1.6.0_18
hadoop@bigdataserver1:~/hadoop/sbin>


But you can see that the environment is sources properly.

Solution is to update hadoop-env.sh with the JAVA_HOME


hadoop@fravm097023:~/hadoop/etc/hadoop> grep -i JAVA_HOME hadoop-env.sh
# The only required environment variable is JAVA_HOME.  All others are
# set JAVA_HOME in this file, so that it is correctly defined on
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/home/hadoop/jdk1.6.0_18
hadoop@fravm097023:~/hadoop/etc/hadoop>

After that DFS and YARN services will come up
hadoop@bigdataserver1:~/hadoop/sbin> sh start-dfs.sh which: no start-dfs.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin) 13/03/13 11:51:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [localhost] localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-bigdataserver1.out localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-bigdataserver1.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-bigdataserver1.out 13/03/13 11:51:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hadoop@bigdataserver1:~/hadoop/sbin>


hadoop@bigdataserver1:~/hadoop/sbin> sh start-yarn.sh starting yarn daemons which: no start-yarn.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin) starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-bigdataserver1.out localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-bigdataserver1.out hadoop@bigdataserver1:~/hadoop/sbin>

To get started with a single node setup here are some simple steps.

- Get a machine which supports CDH , in my case I had a SUSE Linux
- Create a hadoop user
- Download hadoop tar ball hadoop-2.0.0-cdh4.2.0.tar.gz
- Download and Install JDK 1.6 or 1.7
- Unpack the tar ball hadoop-2.0.0-cdh4.2.0.tar.gz
- Source the JAVA_HOME variable
- Modify the hadoop configuration files
- Set password less ssh to the localhost
- Format hadoop namenode
- Start the services
- Validate the setup

I am jumping directly to the configuration files configuration and then to formatting namenode




















Formatting the hadoop nodename

hadoop@bigdataserver1:~> hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

13/03/13 11:15:08 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = bigdataserver1/10.216.9.25
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.0.0-cdh4.2.0
STARTUP_MSG:   classpath = /home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jline-0.9.94.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kfs-0.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-sources.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-test-sources.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/zookeeper-3.4.5-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-sources.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-test-sources.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.2.4.Final.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.2.4.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.0.0-cdh4.2.0.jar
STARTUP_MSG:   build = file:///var/lib/jenkins/workspace/CDH4.2.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.2.0/source/hadoop-common-project/hadoop-common -r 8bce4bd28a464e0a92950c50ba01a9deb1d85686; compiled by 'jenkins' on Fri Feb 15 10:42:32 PST 2013
STARTUP_MSG:   java = 1.6.0_18
************************************************************/
13/03/13 11:15:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-5d53a7be-005c-4d8b-9f93-088c795cbb35
13/03/13 11:15:10 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list
13/03/13 11:15:10 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
13/03/13 11:15:10 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
13/03/13 11:15:10 INFO blockmanagement.BlockManager: defaultReplication         = 1
13/03/13 11:15:10 INFO blockmanagement.BlockManager: maxReplication             = 512
13/03/13 11:15:10 INFO blockmanagement.BlockManager: minReplication             = 1
13/03/13 11:15:10 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
13/03/13 11:15:10 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
13/03/13 11:15:10 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
13/03/13 11:15:10 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
13/03/13 11:15:10 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
13/03/13 11:15:10 INFO namenode.FSNamesystem: supergroup          = supergroup
13/03/13 11:15:10 INFO namenode.FSNamesystem: isPermissionEnabled = false
13/03/13 11:15:10 INFO namenode.FSNamesystem: HA Enabled: false
13/03/13 11:15:10 INFO namenode.FSNamesystem: Append Enabled: true
13/03/13 11:15:11 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
13/03/13 11:15:11 INFO namenode.NNStorage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
13/03/13 11:15:11 INFO namenode.FSImage: Saving image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
13/03/13 11:15:11 INFO namenode.FSImage: Image file of size 121 saved in 0 seconds.
13/03/13 11:15:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
13/03/13 11:15:11 INFO util.ExitUtil: Exiting with status 0
13/03/13 11:15:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bigdataserver1/10.216.9.25
************************************************************/
hadoop@bigdataserver1:~>



Start the service using start-all.sh

Health Check : you can verify the service using the below url 
http://bigdataserver1.bigdata.com:50070/dfshealth.jsp

Cloudera and Oracle partnered to bring a reliable mainstream enterprise solution of Big Data through the Oracle Big Data Appliance. It is designed and optimized for big work load and built using standard hardware.
Oracle Big Data Appliance is a supported big data infrastructure which is affordable , salable and integrated with the key components of big data platform.
CDH (Cloudera Distribution Including Apache Hadoop) is for enterprise deployment which is included with the Oracle Big Data Appliance. It has a free version and it supports up to 50 node but it can scale up with the paid version.

Hadoop was founded by Douglass Read Cutting and picked the name from one of his son's toy elephant. It is a open source project from Apache that evolved rapidly into a major technology movement. It is capable to handle large data set which are structured and unstructured. It had ability to run on low cost clusters and can scale up rapidly.
Hadoop architecture helps applications run on nodes which has thousands of terabytes of data. It has a distributed file system called HDFS (Hadoop Distributed File System) with fast data transfer rates between the clustered nodes and also support failures. It does not require RAID Storage as it achieves reliability by replication technology on multiple hosts. 
Hadoop is a collection of components like MapReduce , Hive , Pig , NoSQL Database , ZooKeeper , Ambari , HCatalog , Oozie, Hue and more.

MapReduce is one key component. It is a framework for writing applications that processes large amount of data which are structured and unstructured in nature. MapReduce applications can be designed for a single node Hadoop and then deployed on a 100 node Hadoop. 
MapReduce engine had one JobTracker to which jobs are submitted. The JobTracker then pushes work to the TaskTracker nodes in the cluster. JobTracker and TaskTracker help to complete the allocated job. 
I will share my experience in getting a single node hadoop running and then running a mapreduce sample application in details.

BIG Data is getting Bigger and Bigger with Oracle , SAP , IBM , EMC , HP , Software AG and with more other players.



With the cost of storage & CPU going down and with the big vendors pitching in with reliable solutions it is now time for companies to look for crunching all those data's which are large and complex.
So how Oracle is helping Companies to eat the Big Mac ?  Oracle Big Data Appliance with clusters of servers.  Cloudera distribution including Apache Hadoop to acquire and organize data. Oracle NoSQL Database Community Edition to acquire data. Additional supporting system software like Oracle Linux , Oracle JVM VM and R distribution.
So what is that had to do with Administrator like me ? No more terabytes , I have to think big like petabyte , exabyte , zettabyte , yottabyte and more Hadoop , future of the big data No more SQL , I have to know NoSQL (Not only SQL) Big data connectors to RDBMS integrating with Oracle Database and Exadata For analyze and visualizing with Oracle Exalytics. Cloud computing and Big Data
So what are the likely responsibilities for Big Data Administrators ? Managing fast growing big data clusters , NoSQL database and linux systems. Meeting operational performance and availability  Setup monitoring and health check of the clusters and all its associated components. Define and enforce operational procedures , processes and best practices. Provide leadership and take ownership in collaboration with teams/vendors/3rd parties to resolving issues. Effectively influence , partner and deliver results in a team environment.

With all the above said I will dedicate some time to snack the Big Mac.

Relevant Posts

BIG DATA Getting Started with HADOOP
BIG DATA Cloudera and Oracle
BIG DATA CDH Single Node Setup
BIG DATA HADOOP Testing with MapReduce Examples Part 1
BIG DATA HADOOP Testing with MapReduce Examples Part 2
BIG DATA HADOOP Testing with MapReduce Examples Part 3

CeBIT is the world's leading high-tech event showcasing digital IT and telecommunications solutions, from 5 to 9 March, 2013 in Hanover, Germany.


This year I had the opportunity to be there.
It was a long day and lots and lots of walk. 
Here are some of the important information I found in CeBIT

• German Chancellor Angela Merkel  officially opened the world’s biggest annual high-tech trade fair.
• SAP's presence was very good with all of its partners.
• Could see SAP HANA's presence more and had a hand's on Smart Energy demo of a 512GB HANA Appliance. 
• Every body looks like in-memory computing
• Could see Chinese dominance over phones, tablets and other electronics.
• Android is every where
• Saw the 3D printer and it is really interesting (spend 2 hours to see a model getting printed)
• Lots of space for Gaming

Planned to visit CeBIT another day in this week and will share some interesting stuffs.

I am in the process of upgrading to 12c Cloud Control from version 11.1.0.1.0 using the one system upgrade.






All looks good with the prerequisite checks but having the problem when authenticating with the database.
The installer gives me ORA-00942 : table or view does not exist but I have no idea.


I am seeing the below message in the installer log files


java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
        at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:851)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
        at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1477)
        at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:392)
        at oracle.sysman.install.oneclick.queries.DBConnect.executeQuery(DBConnect.java:118)
        at oracle.sysman.install.oneclick.queries.executeSql.performQuery(executeSql.java:56)
        at oracle.sysman.install.oneclick.EMGCInputValidation.validate(EMGCInputValidation.java:2370)
        at oracle.sysman.install.oneclick.EMGCDatabaseConnectionDlgonNext.actionsOnClickofNext(EMGCDatabaseConnectionDlgonNext.java:395)
        at oracle.sysman.install.oneclick.EMGCNoSeedInstallDetailsDlg_new$PageValidationListener.wizardValidatePage(EMGCNoSeedInstallDetailsDlg_new.java:667)
        at oracle.bali.ewt.wizard.WizardPage.processWizardValidateEvent(WizardPage.java:710)
        at oracle.bali.ewt.wizard.WizardPage.validatePage(WizardPage.java:680)
        at oracle.bali.ewt.wizard.BaseWizard.validateSelectedPage(BaseWizard.java:2367)
        at oracle.bali.ewt.wizard.BaseWizard._validatePage(BaseWizard.java:3072)
        at oracle.bali.ewt.wizard.BaseWizard.doNext(BaseWizard.java:2152)
        at oracle.bali.ewt.wizard.dWizard.DWizard.doNext(DWizard.java:405)
        at oracle.bali.ewt.wizard.BaseWizard$Action$1.run(BaseWizard.java:3944)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:642)
        at java.awt.EventQueue.access$000(EventQueue.java:85)
        at java.awt.EventQueue$1.run(EventQueue.java:603)
        at java.awt.EventQueue$1.run(EventQueue.java:601)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:612)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
        at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

So far no clue on the errors. Will keep updated if this can help someone.

InnoDB is ACID complaint storage engine for MySQL that has Oracle like commit , rollback and crash recovery to secure data. To quickly start with the InnoDB configuration and setup Packt Publishing has come up with the Instant InnoDB reference book which helps to quickly get on track with the setup.

When Preconfiguring Oracle Unified Directory (OUD) for OIM and OAM 11gR2 Integration the first step say to add OUD Containers using ldapadd command.

http://docs.oracle.com/cd/E27559_01/install.1112/e27301/preconfigoud.htm#CHDCEJKD

ldapadd -h testserver -p 1389 -D "cn=Directory Manager" -w "password" -c -f OUDContainers.ldif

But ldapadd command is not available in OUD 11gR2 , so the same can be loaded using ldapmodify using a special flag. (--defaultAdd)

ldapmodify -h testserver -p 1389 -D "cn=Directory Manager" -w "password"  -c --defaultAdd -f OUDContainers.ldif

ldapadd command is available in OID not in OUD. So this is a documentation bug.

Well my Oracle Cloud trial versions are active now and I have something to play around for a month.
The activation took a long time for me and there are lot of people out there who already had their hands on.

So what are the tools that are basically used on the Oracle Cloud offering

• Oracle Database
• version provided is 11g release 2 enterprise edition
• managed and secured on exadata
• data access and manupulation with sql and plsql standards
• apex - my favorite 
• data movement using sql developer
• it has a lot of productive applications
• it is mobile enabled
• Oracle Java
• it is all java out there
• enterprise manager for managing
• it runs on exalogic
• its associated database runs on exadata
• it runs on weblogic
• it uses identity managment for user/group management
• it uses access manager for single signon

Good to here from Oracle and I am going to explore it in the next up coming days. 

SAP BO and DS with Oracle Database Part 6 Installing DS Client

I downloaded the "SBOP DATA SERVICES DESIGNER 4.0 WITH SP03 WINDOWS (64B)" and used the generic trial licenses for the installation.


 























Using the installed tool access the designer and it would prompt to enter the administrator password.



now we have the console ready for deploying jobs. Wait the jobserver is still not connected to the repository and it had a red cross over the icon which is at the lower right corner. 
The fix took may be a week for me to figure out in the SAP world. It is unlike Oracle world where most of the bug fixes are open and available on the internet. SAP world is still not open based on my 10 years of experience but this is only my view. But still I solved it and I will blog about it later.

SAP BO and DS with Oracle Database Part 5 Verifying CMC and DS

Now that we are done with the installation , we have to verify CMC and DS information.

Access CMC and make sure you are able to login via the administrative user

URL : http://bodsserver001:8080/BOE/CMC


Access Data Services option to see if the repository is active


Edit the repository and confirm if you can connect using the Test Connection button.

Access the Data Services console to see if you can see the repository and the rest of the components.
URL : http://bodsserver001:8080/DataServices

SAP BO and DS with Oracle Database Part 4 Installing DS on SUSE Linux

Now that we are doing with installing BO , I wanted to create the DS environment so that we have a complete setup before we go into the rest of the experiments.

I downloaded "SBOP DATA SERVICES 4.0 WITH SP03 LINUX (64B)" and used the generic trial licenses for implementation.

After downloading unzip the files in Windows and then ftp into the linux server.

Set environment variables relevant for the installation
$ export LANG=en_US.utf8
$ export LC_ALL=en_US.utf8
$ export ORACLE_HOME=..../client/
$ PATH=$PATH:$ORACLE_HOME/bin

Now create a directory to store the installation files.

$ mkdir -p /oraapp/bo/bo4
$ sh setup.sh InstallDir=/oraapp/bo/bo4

Now Choose the Installation language


Make sure that you select the BO installation folder. Initially I created another directory /bo/ds4 and installed  the complete package and nothing happened. So the recommendation is to install DS into BO folder.

 Check the prerequisite , which is fine


accept the trial license





input the trial license key , this will expire in 30 days


Choose option to install data services with cms server , so that it can be controlled via the cms console.


Update the administrative password

select the product which has to be installed , you can custom install after expanding and selecting components which is required.


if there is already running data services configuration than that can be reused here but I have skipped it because of the new install


I am selecting the option to use the existing installed database. remember I created the user in the BO installation procedure.


select Oracle , pay attention if you have Oracle RAC (soon we will see HANA here )


give the details of the oracle environment


create the repository


now configure a new job server to process your jobs , this can be done later with the client tool.


I choose to configure the job server and now I gave new name and a port for the job server to listen


Configure access server here


ports and other inputs has to mentioned


start the installation


it starts


it started


it runs


it finished soon ( no i did not know how long it took , I started the installation in the evening and the next day I saw it got finished )


Now we have BO and DS running. Lets play a bit more with the other options.

SAP BO and DS with Oracle Database Part 3 Installing BO on SUSE Linux

I downloaded "SBOP BI PLATFORM 4.0 SP04 (incl. FP03) SERVER LNX" and used the generic trial licenses for the implementation.

After downloading unzip the files in Windows and then ftp into the linux server.

Set environment variables relevant for the installation
$ export LANG=en_US.utf8
$ export LC_ALL=en_US.utf8
$ export ORACLE_HOME=..../client/
$ PATH=$PATH:$ORACLE_HOME/bin

Now create a directory to store the installation files.

$ mkdir -p /oraapp/bo/bo4
$ sh setup.sh InstallDir=/oraapp/bo/bo4

Now Choose the Installation language


Select the destination folder


Now the prerequisite check  will be done , seems all is fine.


Accept the license agreement

Input the trial key (in my case)


Choose the application language , pay attention if you are deploying BO in the European world as you may have to select mostly all the languages.


Choose the install option


then the install type (here custom installation can be done with option 2) , I choose Full (:-


Choosing the database , here we already have running Oracle Database so option 2


Select Oracle for the CMS repository


Select Oracle for the Audit repository


I used Tomcat application server and there is a possibility to use a wide range of application server. One such is Weblogic which I will try out in a separate installation.


I opt to use the default version management system.


Configure the Server Intelligence Agent which is very helpful when it comes to managing BO components.


Configure the CMS port for the administrative console


Supply the password for the administrator accounts.


Here we have to give the Oracle TNS Name (this should be the name service name that we have mentioned in the tnsnames.ora of the Oracle Client Installation , to create the file source Oracle Home and run netca )

Login into the repository Oracle database and create the below two users.

SQL > grant connect,resource to boecms identified by boecms; (CMS Database User Name)
SQL > grant connect,resource to boeaud identified by boeaud; (Audit Database User Name)




Supply the port number for Tomcat application server


Configure the HTTP listening port


Configure the version management port and administrative password


I do not want to configure SDM agent at this moment so I skip it


I do not want to configure IEM at this moment so I skip it.


Give the confirmation for the installation to start


here we go


go it went to 8% and I can have coffee


now I am going to have lunch will be back and check the status


lunch break is over but still ??? my server is less powerful *may be*


common ....


tea break


good


very good


 Now SAP BO 4.0 SP4 is installed and there is something for me to play this evening.

Next

SAP BO and DS with Oracle Database Part 4 Installing DS on SUSE Linux

SAP BO and DS with Oracle Database Part 2 Installing Oracle Client
BO - Business Objects DS - Data Services
Before starting to install BODS we have install the Oracle Client Software on the machine where BO and DS are going to be installed. Since my implementation was on SUSE Linux I downloaded Oracle Client for Linux and installed it.
The below screenshots will give a overview of the installation process.







With this we are done with the Oracle Client Installation that is required for BO DS Installation.

Next

SAP BO and DS with Oracle Database Part 3 Installing BO on SUSE Linux

SAP BO and DS with Oracle Database Part 1 Installing Oracle Database

BO - Business Objects
DS - Data Services

Every body should be aware of the need to store data on to a database. But when it comes to choosing the database it is always Oracle who wins at Corporate Companies. I have made a BO and DS implementation with Oracle Database running at the back end.

To start with the BO and DS implementation first a database has to be installed. I choose HP Itanium server for the database and SUSE 11 for the BO Server. There are very less implementation of SAP BO and DS on linux environment so I decided to work with linux platforms instead of Windows.

Installing Oracle Database

This is the fun part and I am not going into detailed level as there are many references where the installation are covered. After installing the Oracle home invoke DBCA and install a health database for the repository. Additional databases has to be installed for all the necessary supporting objects , example a data warehouse database.

Next
SAP BO and DS with Oracle Database Part 2 Installing Oracle Client