Surachart Opun
This page contains my experiences and my thoughts about Oracle and etc... about IT. Perhaps With another way in my life.
Updated: 12 hours 59 min ago
Managing Startups: Best Blog Posts By Thomas Eisenmann
I wanted to read a new book for new idea. Maybe, it's about something startup. Yes! For something new, that's a good idea. I chose to read "Managing Startups: Best Blog Posts" book By Thomas Eisenmann. This book, Harvard Business School Professor Tom Eisenmann annually compiles the best posts from many blogs on technology startup management. You’ll find 72 posts from successful entrepreneurs and venture capitalists, such as Fred Wilson, Steve Blank, Ash Maurya, Joel Spolsky, and Ben Yoskovitz. Wow! that's very very cool.Topics in a book, including:
- Management tasks: Engineering, product management, marketing, sales, and business development
- Organizational issues: Cofounder tensions, recruiting, and career planning
- Funding: The latest developments in capital markets that affect startups
Anyway, You don't need to read from chapter 1 to last chapter, you are able to choose which chapter or area you want to read. It has 13 areas for you.
- Lean Startup
- Business Models
- Customer Discovery and Validation
- Marketing: Demand Generation and Optimization
- Sales, Marketing, and PR Management
- Product Management/Product Design
- Business Development and Scaling
- Funding Strategy
- Company Culture, Organizational Structure, Recruiting, and Other HR Issues
- Startup Failure
- Exiting by Selling Your Company
- The Startup Mindset and Coping with Startup Pressures
- Management and Career Advice
I believe a book useful for entrepreneurs and who are interested in the management of technology startups. A book will give you more idea and vision. Contents in a book likes a clue for you.Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
play with vncserver... when it showed nothing
I often use "ssvnc" to connect my servers to install or do something about X. Anyway, I must to start VNC before by using "vncserver".
[surachart@oralearning ~]$ vncserver
New 'oralearning:1 (surachart)' desktop is oralearning:1
Starting applications specified in /home/surachart/.vnc/xstartup
Log file is /home/surachart/.vnc/oralearning:1.log
[surachart@oralearning ~]$I ended up find nothing when I connected VNC as below picture.
So, I checked "/home/surachart/.vnc/oralearning:1.log" file to investigate the issue.
Fri May 17 15:46:12 2013
vncext: VNC extension running!
vncext: Listening for VNC connections on all interface(s), port 5901
vncext: created VNC server for screen 0
/home/surachart/.vnc/xstartup: line 27: xsetroot: command not found
/home/surachart/.vnc/xstartup: line 29: twm: command not found
/home/surachart/.vnc/xstartup: line 28: xterm: command not foundAs messages in log file. I installed some packages, killed VNC and started VNC again.
[root@oralearning ~]# yum install xorg-x11-twm xorg-x11-server-utils xterm
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package xorg-x11-server-utils.x86_64 0:7.5-13.el6 will be installed
--> Processing Dependency: mcpp for package: xorg-x11-server-utils-7.5-13.el6.x86_64
---> Package xorg-x11-twm.x86_64 1:1.0.3-5.1.el6 will be installed
---> Package xterm.x86_64 0:253-1.el6 will be installed
--> Running transaction check
---> Package mcpp.x86_64 0:2.7.2-4.1.el6 will be installed
--> Processing Dependency: libmcpp.so.0()(64bit) for package: mcpp-2.7.2-4.1.el6.x86_64
--> Running transaction check
---> Package libmcpp.x86_64 0:2.7.2-4.1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Installing:
xorg-x11-server-utils x86_64 7.5-13.el6 ol6_latest 158 k
xorg-x11-twm x86_64 1:1.0.3-5.1.el6 ol6_latest 100 k
xterm x86_64 253-1.el6 ol6_latest 357 k
Installing for dependencies:
libmcpp x86_64 2.7.2-4.1.el6 ol6_latest 68 k
mcpp x86_64 2.7.2-4.1.el6 ol6_latest 23 k
Transaction Summary
================================================================================================================================================
Install 5 Package(s)
Total download size: 706 k
Installed size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): libmcpp-2.7.2-4.1.el6.x86_64.rpm | 68 kB 00:01
(2/5): mcpp-2.7.2-4.1.el6.x86_64.rpm | 23 kB 00:00
(3/5): xorg-x11-server-utils-7.5-13.el6.x86_64.rpm | 158 kB 00:00
(4/5): xorg-x11-twm-1.0.3-5.1.el6.x86_64.rpm | 100 kB 00:00
(5/5): xterm-253-1.el6.x86_64.rpm | 357 kB 00:01
------------------------------------------------------------------------------------------------------------------------------------------------
Total 73 kB/s | 706 kB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libmcpp-2.7.2-4.1.el6.x86_64 1/5
Installing : mcpp-2.7.2-4.1.el6.x86_64 2/5
Installing : xorg-x11-server-utils-7.5-13.el6.x86_64 3/5
Installing : 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64 4/5
Installing : xterm-253-1.el6.x86_64 5/5
Verifying : libmcpp-2.7.2-4.1.el6.x86_64 1/5
Verifying : xorg-x11-server-utils-7.5-13.el6.x86_64 2/5
Verifying : mcpp-2.7.2-4.1.el6.x86_64 3/5
Verifying : xterm-253-1.el6.x86_64 4/5
Verifying : 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64 5/5
Installed:
xorg-x11-server-utils.x86_64 0:7.5-13.el6 xorg-x11-twm.x86_64 1:1.0.3-5.1.el6 xterm.x86_64 0:253-1.el6
Dependency Installed:
libmcpp.x86_64 0:2.7.2-4.1.el6 mcpp.x86_64 0:2.7.2-4.1.el6
Complete!
[root@oralearning ~]# exit
logout
[surachart@oralearning ~]$ vncserver -kill :1
Killing Xvnc process ID 1647
[surachart@oralearning ~]$
[surachart@oralearning ~]$ vnc
vncconfig vncpasswd vncserver
[surachart@oralearning ~]$ vncserver
New 'oralearning:1 (surachart)' desktop is oralearning:1
Starting applications specified in /home/surachart/.vnc/xstartup
Log file is /home/surachart/.vnc/oralearning:1.log
[surachart@oralearning ~]$ less /home/surachart/.vnc/oralearning\:1.log
Connected to server again. It was all right for me.
Written By: Surachart Opun http://surachartopun.com
[surachart@oralearning ~]$ vncserver
New 'oralearning:1 (surachart)' desktop is oralearning:1
Starting applications specified in /home/surachart/.vnc/xstartup
Log file is /home/surachart/.vnc/oralearning:1.log
[surachart@oralearning ~]$I ended up find nothing when I connected VNC as below picture.
So, I checked "/home/surachart/.vnc/oralearning:1.log" file to investigate the issue.Fri May 17 15:46:12 2013
vncext: VNC extension running!
vncext: Listening for VNC connections on all interface(s), port 5901
vncext: created VNC server for screen 0
/home/surachart/.vnc/xstartup: line 27: xsetroot: command not found
/home/surachart/.vnc/xstartup: line 29: twm: command not found
/home/surachart/.vnc/xstartup: line 28: xterm: command not foundAs messages in log file. I installed some packages, killed VNC and started VNC again.
[root@oralearning ~]# yum install xorg-x11-twm xorg-x11-server-utils xterm
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package xorg-x11-server-utils.x86_64 0:7.5-13.el6 will be installed
--> Processing Dependency: mcpp for package: xorg-x11-server-utils-7.5-13.el6.x86_64
---> Package xorg-x11-twm.x86_64 1:1.0.3-5.1.el6 will be installed
---> Package xterm.x86_64 0:253-1.el6 will be installed
--> Running transaction check
---> Package mcpp.x86_64 0:2.7.2-4.1.el6 will be installed
--> Processing Dependency: libmcpp.so.0()(64bit) for package: mcpp-2.7.2-4.1.el6.x86_64
--> Running transaction check
---> Package libmcpp.x86_64 0:2.7.2-4.1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Installing:
xorg-x11-server-utils x86_64 7.5-13.el6 ol6_latest 158 k
xorg-x11-twm x86_64 1:1.0.3-5.1.el6 ol6_latest 100 k
xterm x86_64 253-1.el6 ol6_latest 357 k
Installing for dependencies:
libmcpp x86_64 2.7.2-4.1.el6 ol6_latest 68 k
mcpp x86_64 2.7.2-4.1.el6 ol6_latest 23 k
Transaction Summary
================================================================================================================================================
Install 5 Package(s)
Total download size: 706 k
Installed size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): libmcpp-2.7.2-4.1.el6.x86_64.rpm | 68 kB 00:01
(2/5): mcpp-2.7.2-4.1.el6.x86_64.rpm | 23 kB 00:00
(3/5): xorg-x11-server-utils-7.5-13.el6.x86_64.rpm | 158 kB 00:00
(4/5): xorg-x11-twm-1.0.3-5.1.el6.x86_64.rpm | 100 kB 00:00
(5/5): xterm-253-1.el6.x86_64.rpm | 357 kB 00:01
------------------------------------------------------------------------------------------------------------------------------------------------
Total 73 kB/s | 706 kB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libmcpp-2.7.2-4.1.el6.x86_64 1/5
Installing : mcpp-2.7.2-4.1.el6.x86_64 2/5
Installing : xorg-x11-server-utils-7.5-13.el6.x86_64 3/5
Installing : 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64 4/5
Installing : xterm-253-1.el6.x86_64 5/5
Verifying : libmcpp-2.7.2-4.1.el6.x86_64 1/5
Verifying : xorg-x11-server-utils-7.5-13.el6.x86_64 2/5
Verifying : mcpp-2.7.2-4.1.el6.x86_64 3/5
Verifying : xterm-253-1.el6.x86_64 4/5
Verifying : 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64 5/5
Installed:
xorg-x11-server-utils.x86_64 0:7.5-13.el6 xorg-x11-twm.x86_64 1:1.0.3-5.1.el6 xterm.x86_64 0:253-1.el6
Dependency Installed:
libmcpp.x86_64 0:2.7.2-4.1.el6 mcpp.x86_64 0:2.7.2-4.1.el6
Complete!
[root@oralearning ~]# exit
logout
[surachart@oralearning ~]$ vncserver -kill :1
Killing Xvnc process ID 1647
[surachart@oralearning ~]$
[surachart@oralearning ~]$ vnc
vncconfig vncpasswd vncserver
[surachart@oralearning ~]$ vncserver
New 'oralearning:1 (surachart)' desktop is oralearning:1
Starting applications specified in /home/surachart/.vnc/xstartup
Log file is /home/surachart/.vnc/oralearning:1.log
[surachart@oralearning ~]$ less /home/surachart/.vnc/oralearning\:1.log
Connected to server again. It was all right for me.
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
The account is locked on APEX
I have used APEX 4.2.2. I was unable to logon my APEX with INTERNAL workspace as admin. It showed "The account is locked".
What was I able to do? ... run "apxchpwd.sql" script. I didn't think so. After checking in "apxchpwd.sql" script.
So, I checked in "wwv_flow_fnd_user_api" package, I found wwv_flow_fnd_user_api.UNLOCK_ACCOUNT procedure.
PROCEDURE UNLOCK_ACCOUNT
Argument Name Type In/Out Default?
------------------------------ ----------------------- ------ --------
P_USER_NAME VARCHAR2 INI thought I should test it. First of all, I must to make sure "security_group_id"(It should be number 10).
SQL> SELECT workspace_id FROM apex_workspaces WHERE workspace = 'INTERNAL';
WORKSPACE_ID
------------
10Tested to unlock "admin" user on "INTERNAL" workspace.
SQL> alter session set current_schema = APEX_040200;
Session altered.
SQL> begin
wwv_flow_security.g_security_group_id := 10;
wwv_flow_fnd_user_api.UNLOCK_ACCOUNT('ADMIN');
commit;end;
/
PL/SQL procedure successfully completed.
then I tested login again... It works for me. ^______________^
Written By: Surachart Opun http://surachartopun.com
What was I able to do? ... run "apxchpwd.sql" script. I didn't think so. After checking in "apxchpwd.sql" script.So, I checked in "wwv_flow_fnd_user_api" package, I found wwv_flow_fnd_user_api.UNLOCK_ACCOUNT procedure.
PROCEDURE UNLOCK_ACCOUNT
Argument Name Type In/Out Default?
------------------------------ ----------------------- ------ --------
P_USER_NAME VARCHAR2 INI thought I should test it. First of all, I must to make sure "security_group_id"(It should be number 10).
SQL> SELECT workspace_id FROM apex_workspaces WHERE workspace = 'INTERNAL';
WORKSPACE_ID
------------
10Tested to unlock "admin" user on "INTERNAL" workspace.
SQL> alter session set current_schema = APEX_040200;
Session altered.
SQL> begin
wwv_flow_security.g_security_group_id := 10;
wwv_flow_fnd_user_api.UNLOCK_ACCOUNT('ADMIN');
commit;end;
/
PL/SQL procedure successfully completed.
then I tested login again... It works for me. ^______________^
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Learned a little bit about importing data from MySQL into HDFS using Sqoop
I have a chance to read a book - Hadoop Real-World Solutions Cookbook(Thank you ^______^). It pops up in my head, why I have never tested about Sqoop. As you know sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. So, I wanna test a little bit about it. I choose to read data from mysql into HDFS.
I assume my data in mysql
[surachart@centos ~]$ mysql -u surachart mydb -p < a.sql
Enter password:
[surachart@centos ~]$ mysql -u surachart mydb -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.1.69 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select count(*) from mytable;
+----------+
| count(*) |
+----------+
| 1295462 |
+----------+
1 row in set (0.00 sec)
mysql>After I have my data, I download Sqoop (binary) and test it.
[surachart@centos ~]$ ls sqoop-1.4.3.bin__hadoop-0.20.tar.gz
sqoop-1.4.3.bin__hadoop-0.20.tar.gz
[surachart@centos ~]$ tar zxf sqoop-1.4.3.bin__hadoop-0.20.tar.gz
[surachart@centos ~]$ cd sqoop-1.4.3.bin__hadoop-0.20
[surachart@centos sqoop-1.4.3.bin__hadoop-0.20]$ cd bin/
[surachart@centos bin]$ pwd
/home/surachart/sqoop-1.4.3.bin__hadoop-0.20/bin
[surachart@centos bin]$ ./sqoop
Error: /usr/lib/hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.
[surachart@centos bin]$ export HADOOP_COMMON_HOME=/usr
[surachart@centos bin]$ ./sqoop
Error: /usr/lib/hadoop-mapreduce does not exist!
Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.
[surachart@centos bin]$ export HADOOP_MAPRED_HOME=/usr
[surachart@centos bin]$ ./sqoop
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Try 'sqoop help' for usage.
[surachart@centos bin]$ export HBASE_HOME=/home/surachart/hbase
[surachart@centos bin]$ ./sqoop
Try 'sqoop help' for usage.
[surachart@centos bin]$ ./sqoop help
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
[surachart@centos bin]$ hadoop fs -mkdir /user/surachart/import
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import
[surachart@centos bin]$
[surachart@centos bin]$ ./sqoop import -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import
13/05/08 12:11:42 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:11:42 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:11:42 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:11:43 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:716)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:605)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:628)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:235)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:219)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:283)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1255)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1072)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)Note: On Sqoop no mysql lib, So Download "mysql-connector-java-5.1.25" on http://dev.mysql.com/downloads/connector/j/ and copy it to sqoop lib path.
[surachart@centos ~]$ ls mysql-connector-java-5.1.25.zip
mysql-connector-java-5.1.25.zip
[surachart@centos ~]$ unzip mysql-connector-java-5.1.25.zip
[surachart@centos ~]$ cp mysql-connector-java-5.1.25/mysql-connector-java-5.1.25-bin.jar sqoop-1.4.3.bin__hadoop-0.20/lib/After everything should be fine. Try again.
[surachart@centos ~]$ cd sqoop-1.4.3.bin__hadoop-0.20/bin/
[surachart@centos bin]$ ./sqoop import --direct -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import
13/05/08 12:27:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:27:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:27:54 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:27:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:27:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:27:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr
Note: /tmp/sqoop-surachart/compile/7d652c85de6562a56d07c0b4017e3cd4/mytable.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/05/08 12:28:04 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-surachart/compile/7d652c85de6562a56d07c0b4017e3cd4/mytable.jar
13/05/08 12:28:04 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/05/08 12:28:04 INFO mapreduce.ImportJobBase: Beginning import of mytable
13/05/08 12:28:10 INFO mapred.JobClient: Cleaning up the staging area hdfs://centos:8020/user/surachart/.staging/job_201305081226_0002
13/05/08 12:28:10 ERROR security.UserGroupInformation: PriviledgedActionException as:surachart cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/surachart/import already exists
13/05/08 12:28:10 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/surachart/import already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
at org.apache.sqoop.manager.DirectMySQLManager.importTable(DirectMySQLManager.java:92)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
[surachart@centos bin]$ ./sqoop import --direct -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import/mytable
13/05/08 12:28:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:28:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:28:24 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:28:26 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:28:26 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:28:26 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr
Note: /tmp/sqoop-surachart/compile/98909e84f7e2215902ebdbcca40cfa9f/mytable.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/05/08 12:28:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-surachart/compile/98909e84f7e2215902ebdbcca40cfa9f/mytable.jar
13/05/08 12:28:34 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/05/08 12:28:34 INFO mapreduce.ImportJobBase: Beginning import of mytable
13/05/08 12:28:42 INFO mapred.JobClient: Running job: job_201305081226_0003
13/05/08 12:28:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/08 12:29:34 INFO mapred.JobClient: map 100% reduce 0%
13/05/08 12:29:56 INFO mapred.JobClient: Job complete: job_201305081226_0003
13/05/08 12:29:56 INFO mapred.JobClient: Counters: 18
13/05/08 12:29:56 INFO mapred.JobClient: Job Counters
13/05/08 12:29:56 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=54352
13/05/08 12:29:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/05/08 12:29:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/05/08 12:29:56 INFO mapred.JobClient: Launched map tasks=1
13/05/08 12:29:56 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/05/08 12:29:56 INFO mapred.JobClient: File Output Format Counters
13/05/08 12:29:56 INFO mapred.JobClient: Bytes Written=23687032
13/05/08 12:29:56 INFO mapred.JobClient: FileSystemCounters
13/05/08 12:29:56 INFO mapred.JobClient: HDFS_BYTES_READ=87
13/05/08 12:29:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=36476
13/05/08 12:29:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=23687032
13/05/08 12:29:56 INFO mapred.JobClient: File Input Format Counters
13/05/08 12:29:56 INFO mapred.JobClient: Bytes Read=0
13/05/08 12:29:56 INFO mapred.JobClient: Map-Reduce Framework
13/05/08 12:29:56 INFO mapred.JobClient: Map input records=1
13/05/08 12:29:56 INFO mapred.JobClient: Physical memory (bytes) snapshot=73830400
13/05/08 12:29:56 INFO mapred.JobClient: Spilled Records=0
13/05/08 12:29:56 INFO mapred.JobClient: CPU time spent (ms)=16210
13/05/08 12:29:56 INFO mapred.JobClient: Total committed heap usage (bytes)=29818880
13/05/08 12:29:56 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1179168768
13/05/08 12:29:56 INFO mapred.JobClient: Map output records=1295462
13/05/08 12:29:56 INFO mapred.JobClient: SPLIT_RAW_BYTES=87
13/05/08 12:29:56 INFO mapreduce.ImportJobBase: Transferred 22.5897 MB in 81.613 seconds (283.4336 KB/sec)
13/05/08 12:29:56 INFO mapreduce.ImportJobBase: Retrieved 1295462 records.
Check data in HDFS.
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import
Found 1 items
drwx------ - surachart surachart 0 2013-05-08 12:29 /user/surachart/import/mytable
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import/mytable
Found 2 items
-rw------- 3 surachart surachart 0 2013-05-08 12:29 /user/surachart/import/mytable/_SUCCESS
-rw------- 3 surachart surachart 23687032 2013-05-08 12:29 /user/surachart/import/mytable/part-m-00000Note: data in /user/surachart/import/mytable/part-m-00000 file, that is CSV type.
Written By: Surachart Opun http://surachartopun.com
I assume my data in mysql
[surachart@centos ~]$ mysql -u surachart mydb -p < a.sql
Enter password:
[surachart@centos ~]$ mysql -u surachart mydb -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.1.69 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select count(*) from mytable;
+----------+
| count(*) |
+----------+
| 1295462 |
+----------+
1 row in set (0.00 sec)
mysql>After I have my data, I download Sqoop (binary) and test it.
[surachart@centos ~]$ ls sqoop-1.4.3.bin__hadoop-0.20.tar.gz
sqoop-1.4.3.bin__hadoop-0.20.tar.gz
[surachart@centos ~]$ tar zxf sqoop-1.4.3.bin__hadoop-0.20.tar.gz
[surachart@centos ~]$ cd sqoop-1.4.3.bin__hadoop-0.20
[surachart@centos sqoop-1.4.3.bin__hadoop-0.20]$ cd bin/
[surachart@centos bin]$ pwd
/home/surachart/sqoop-1.4.3.bin__hadoop-0.20/bin
[surachart@centos bin]$ ./sqoop
Error: /usr/lib/hadoop does not exist!
Please set $HADOOP_COMMON_HOME to the root of your Hadoop installation.
[surachart@centos bin]$ export HADOOP_COMMON_HOME=/usr
[surachart@centos bin]$ ./sqoop
Error: /usr/lib/hadoop-mapreduce does not exist!
Please set $HADOOP_MAPRED_HOME to the root of your Hadoop MapReduce installation.
[surachart@centos bin]$ export HADOOP_MAPRED_HOME=/usr
[surachart@centos bin]$ ./sqoop
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Try 'sqoop help' for usage.
[surachart@centos bin]$ export HBASE_HOME=/home/surachart/hbase
[surachart@centos bin]$ ./sqoop
Try 'sqoop help' for usage.
[surachart@centos bin]$ ./sqoop help
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
[surachart@centos bin]$ hadoop fs -mkdir /user/surachart/import
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import
[surachart@centos bin]$
[surachart@centos bin]$ ./sqoop import -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import
13/05/08 12:11:42 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:11:42 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:11:42 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:11:43 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:716)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:605)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:628)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:235)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:219)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:283)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1255)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1072)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:390)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)Note: On Sqoop no mysql lib, So Download "mysql-connector-java-5.1.25" on http://dev.mysql.com/downloads/connector/j/ and copy it to sqoop lib path.
[surachart@centos ~]$ ls mysql-connector-java-5.1.25.zip
mysql-connector-java-5.1.25.zip
[surachart@centos ~]$ unzip mysql-connector-java-5.1.25.zip
[surachart@centos ~]$ cp mysql-connector-java-5.1.25/mysql-connector-java-5.1.25-bin.jar sqoop-1.4.3.bin__hadoop-0.20/lib/After everything should be fine. Try again.
[surachart@centos ~]$ cd sqoop-1.4.3.bin__hadoop-0.20/bin/
[surachart@centos bin]$ ./sqoop import --direct -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import
13/05/08 12:27:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:27:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:27:54 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:27:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:27:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:27:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr
Note: /tmp/sqoop-surachart/compile/7d652c85de6562a56d07c0b4017e3cd4/mytable.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/05/08 12:28:04 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-surachart/compile/7d652c85de6562a56d07c0b4017e3cd4/mytable.jar
13/05/08 12:28:04 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/05/08 12:28:04 INFO mapreduce.ImportJobBase: Beginning import of mytable
13/05/08 12:28:10 INFO mapred.JobClient: Cleaning up the staging area hdfs://centos:8020/user/surachart/.staging/job_201305081226_0002
13/05/08 12:28:10 ERROR security.UserGroupInformation: PriviledgedActionException as:surachart cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/surachart/import already exists
13/05/08 12:28:10 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory /user/surachart/import already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
at org.apache.sqoop.manager.DirectMySQLManager.importTable(DirectMySQLManager.java:92)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
[surachart@centos bin]$ ./sqoop import --direct -m 1 --connect jdbc:mysql://localhost:3306/mydb --username surachart --password password --table mytable --target-dir /user/surachart/import/mytable
13/05/08 12:28:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/05/08 12:28:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/05/08 12:28:24 INFO tool.CodeGenTool: Beginning code generation
13/05/08 12:28:26 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:28:26 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `mytable` AS t LIMIT 1
13/05/08 12:28:26 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr
Note: /tmp/sqoop-surachart/compile/98909e84f7e2215902ebdbcca40cfa9f/mytable.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/05/08 12:28:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-surachart/compile/98909e84f7e2215902ebdbcca40cfa9f/mytable.jar
13/05/08 12:28:34 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import
13/05/08 12:28:34 INFO mapreduce.ImportJobBase: Beginning import of mytable
13/05/08 12:28:42 INFO mapred.JobClient: Running job: job_201305081226_0003
13/05/08 12:28:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/08 12:29:34 INFO mapred.JobClient: map 100% reduce 0%
13/05/08 12:29:56 INFO mapred.JobClient: Job complete: job_201305081226_0003
13/05/08 12:29:56 INFO mapred.JobClient: Counters: 18
13/05/08 12:29:56 INFO mapred.JobClient: Job Counters
13/05/08 12:29:56 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=54352
13/05/08 12:29:56 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/05/08 12:29:56 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/05/08 12:29:56 INFO mapred.JobClient: Launched map tasks=1
13/05/08 12:29:56 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/05/08 12:29:56 INFO mapred.JobClient: File Output Format Counters
13/05/08 12:29:56 INFO mapred.JobClient: Bytes Written=23687032
13/05/08 12:29:56 INFO mapred.JobClient: FileSystemCounters
13/05/08 12:29:56 INFO mapred.JobClient: HDFS_BYTES_READ=87
13/05/08 12:29:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=36476
13/05/08 12:29:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=23687032
13/05/08 12:29:56 INFO mapred.JobClient: File Input Format Counters
13/05/08 12:29:56 INFO mapred.JobClient: Bytes Read=0
13/05/08 12:29:56 INFO mapred.JobClient: Map-Reduce Framework
13/05/08 12:29:56 INFO mapred.JobClient: Map input records=1
13/05/08 12:29:56 INFO mapred.JobClient: Physical memory (bytes) snapshot=73830400
13/05/08 12:29:56 INFO mapred.JobClient: Spilled Records=0
13/05/08 12:29:56 INFO mapred.JobClient: CPU time spent (ms)=16210
13/05/08 12:29:56 INFO mapred.JobClient: Total committed heap usage (bytes)=29818880
13/05/08 12:29:56 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1179168768
13/05/08 12:29:56 INFO mapred.JobClient: Map output records=1295462
13/05/08 12:29:56 INFO mapred.JobClient: SPLIT_RAW_BYTES=87
13/05/08 12:29:56 INFO mapreduce.ImportJobBase: Transferred 22.5897 MB in 81.613 seconds (283.4336 KB/sec)
13/05/08 12:29:56 INFO mapreduce.ImportJobBase: Retrieved 1295462 records.
Check data in HDFS.
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import
Found 1 items
drwx------ - surachart surachart 0 2013-05-08 12:29 /user/surachart/import/mytable
[surachart@centos bin]$ hadoop fs -ls /user/surachart/import/mytable
Found 2 items
-rw------- 3 surachart surachart 0 2013-05-08 12:29 /user/surachart/import/mytable/_SUCCESS
-rw------- 3 surachart surachart 23687032 2013-05-08 12:29 /user/surachart/import/mytable/part-m-00000Note: data in /user/surachart/import/mytable/part-m-00000 file, that is CSV type.
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Managing Multimedia and Unstructured Data in the Oracle Database by Marcelle Kratochvil
Today, I have a chance to review a good book about Oracle Database - Managing Multimedia and Unstructured Data in the Oracle Database by Marcelle Kratochvil.
Author, she is CTO and co-founder of Piction, an Oracle PL/SQL based app that allows for the delivery of Images on the internet. Marcelle has been working in the IT industry for 20 years. In this time she has presented over 40 papers in Australia and the United States. Marcelle is both a DBA and Developer and has had experience using Oracle on MVS, VMS, Unix and Windows. She has worked on ultra-small databases to multi-terabyte sized ones. Marcelle has been using Oracle Multimedia since Oracle8, and has been using Oracle as a DBA and Developer since Oracle4. She also specialises in PL/SQL, Java, Gateways, Very Large Databases, Security and Database tuning. Her current focus is on Digital Imaging, Audio, Video, Spatial and all other types of binary object.
As author wrote in a book , the aim of this book is to try and give a basic understanding to a lot of concepts involving unstructured data. Particular focus is given to multimedia (smart media or rich media). So, readers will get idea about Unstructured Data, Digital Objects, and many things. Anyway, if you are interested in increasing your knowledge and understanding more for multimedia, how to manage unstructured data and why you should use Oracle Database with unstructured data. This book fits for you, you are supposed to read it.
This book has 10 chapters. You will learn.
- Discover a whole new world beyond relational databases
- Understand what is involved in selling a digital image
- Learn about the different types of multimedia warehouses
- Uncover the truth behind searching for digital objects
- Understand the complete picture for tuning an Oracle database with multimedia
- Get to grips with all the issues in setting up a digital asset e-commerce system
- Understand what multimedia and unstructured data really is
- Realise how the Oracle database can work with multimedia
- Get to grips with digital image processing and transformation techniques
- Expand your database knowledge to include complex data
- Add credibility to your resume by adopting this new and visually exciting technological direction
Note: Appendix E, Loading and Reading, is not present in the book but is available for download at the following link: http://www.packtpub.com/sites/default/files/downloads/AppendixE_loading_and_reading.pdf
What do I think? Easy to read and clearly about digital data. Give good idea to manage unstructured data, show idea why there uses Oracle Database and learn from good examples. This book will give benefit for some people who are interested in managing multimedia and Unstructured data in the Oracle Database or increasing knowledge about Data Digital. This book covers about database tuning as well. This chapter helps DBAs learn new concepts, skills, and techniques that are required to manage very large multimedia databases. You also get idea to understand the limitations of oracle database with unstructured data.Written By: Surachart Opun http://surachartopun.com
Author, she is CTO and co-founder of Piction, an Oracle PL/SQL based app that allows for the delivery of Images on the internet. Marcelle has been working in the IT industry for 20 years. In this time she has presented over 40 papers in Australia and the United States. Marcelle is both a DBA and Developer and has had experience using Oracle on MVS, VMS, Unix and Windows. She has worked on ultra-small databases to multi-terabyte sized ones. Marcelle has been using Oracle Multimedia since Oracle8, and has been using Oracle as a DBA and Developer since Oracle4. She also specialises in PL/SQL, Java, Gateways, Very Large Databases, Security and Database tuning. Her current focus is on Digital Imaging, Audio, Video, Spatial and all other types of binary object.
As author wrote in a book , the aim of this book is to try and give a basic understanding to a lot of concepts involving unstructured data. Particular focus is given to multimedia (smart media or rich media). So, readers will get idea about Unstructured Data, Digital Objects, and many things. Anyway, if you are interested in increasing your knowledge and understanding more for multimedia, how to manage unstructured data and why you should use Oracle Database with unstructured data. This book fits for you, you are supposed to read it.This book has 10 chapters. You will learn.
- Discover a whole new world beyond relational databases
- Understand what is involved in selling a digital image
- Learn about the different types of multimedia warehouses
- Uncover the truth behind searching for digital objects
- Understand the complete picture for tuning an Oracle database with multimedia
- Get to grips with all the issues in setting up a digital asset e-commerce system
- Understand what multimedia and unstructured data really is
- Realise how the Oracle database can work with multimedia
- Get to grips with digital image processing and transformation techniques
- Expand your database knowledge to include complex data
- Add credibility to your resume by adopting this new and visually exciting technological direction
Note: Appendix E, Loading and Reading, is not present in the book but is available for download at the following link: http://www.packtpub.com/sites/default/files/downloads/AppendixE_loading_and_reading.pdf
What do I think? Easy to read and clearly about digital data. Give good idea to manage unstructured data, show idea why there uses Oracle Database and learn from good examples. This book will give benefit for some people who are interested in managing multimedia and Unstructured data in the Oracle Database or increasing knowledge about Data Digital. This book covers about database tuning as well. This chapter helps DBAs learn new concepts, skills, and techniques that are required to manage very large multimedia databases. You also get idea to understand the limitations of oracle database with unstructured data.Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Play python with twitter streaming
I was interested in how to do twitter streaming. I tried to find out on the internet and got many examples. I tested it with python code(tweepy). I got data in JSON format. I thought it's a good idea, if I was able insert this data into mongodb.
So. I created my application on twitter and tested some a little bit code.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
> db.twittertest.find()Used some code on the internet and changed a bit to insert data into database.
[surachart@centos ~]$ cat ./test-tweeter.py
#!/usr/bin/python
import tweepy
import pymongo
import json
consumer_key = "..."
consumer_secret = "..."
access_token_key = "..."
access_token_secret = "..."
auth1 = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth1.set_access_token(access_token_key, access_token_secret)
class StreamListener(tweepy.StreamListener):
def on_status(self, tweet):
print tweet.text
def on_error(self, status_code):
print 'Error: ' + repr(status_code)
return False
def on_data(self, data):
print 'Ok! Inserting Data.'
from pymongo import MongoClient
client = MongoClient()
client = MongoClient('localhost', 27017)
db = client.test
test_id = db.twittertest.insert(json.loads(data))
l = StreamListener()
streamer = tweepy.Stream(auth=auth1, listener=l)
streamer.filter(track=['oracle'])I filtered by using word 'oracle'.
[surachart@centos ~]$ ./test-tweeter.py
Ok! Inserting Data.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
{ "_id" : ObjectId("51851511cdc8722152c28171"), "contributors" : null, "truncated" : false, "text" : "Job Opportunity for Oracle PL/SQL Developers with HTML/Web development experience for 6+ Months(Contract) for Las...: http://t.co/p6p8cfV6qF", "in_reply_to_status_id" : null, "id" : NumberLong("330684010961448962"), "favorite_count" : 0, "source" : "<a href=\"http://www.linkedin.com/\" rel=\"nofollow\">LinkedIn</a>", "retweeted" : false, "coordinates" : null, "entities" : { "symbols" : [ ], "user_mentions" : [ ], "hashtags" : [ ], "urls" : [ { "url" : "http://t.co/p6p8cfV6qF", "indices" : [ 118, 140 ], "expanded_url" : "http://lnkd.in/G-hhiX", "display_url" : "lnkd.in/G-hhiX" } ] }, "in_reply_to_screen_name" : null, "id_str" : "330684010961448962", "retweet_count" : 0, "in_reply_to_user_id" : null, "favorited" : false, "user" : { "follow_request_sent" : null, "profile_use_background_image" : true, "default_profile_image" : true, "id" : 360409067, "verified" : false, "profile_image_url_https" : "https://si0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "profile_sidebar_fill_color" : "DDEEF6", "profile_text_color" : "333333", "followers_count" : 22, "profile_sidebar_border_color" : "C0DEED", "id_str" : "360409067", "profile_background_color" : "C0DEED", "listed_count" : 1, "profile_background_image_url_https" : "https://si0.twimg.com/images/themes/theme1/bg.png", "utc_offset" : null, "statuses_count" : 1777, "description" : null, "friends_count" : 0, "location" : "", "profile_link_color" : "0084B4", "profile_image_url" : "http://a0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "following" : null, "geo_enabled" : false, "profile_background_image_url" : "http://a0.twimg.com/images/themes/theme1/bg.png", "name" : "Anand P. Sharma", "lang" : "en", "profile_background_tile" : false, "favourites_count" : 0, "screen_name" : "anandph123", "notifications" : null, "url" : null, "created_at" : "Tue Aug 23 04:52:03 +0000 2011", "contributors_enabled" : false, "time_zone" : null, "protected" : false, "default_profile" : true, "is_translator" : false }, "geo" : null, "in_reply_to_user_id_str" : null, "possibly_sensitive" : false, "lang" : "en", "created_at" : "Sat May 04 14:03:06 +0000 2013", "filter_level" : "medium", "in_reply_to_status_id_str" : null, "place" : null }
>I was able to insert data... and I will play with it later.
Written By: Surachart Opun http://surachartopun.com
So. I created my application on twitter and tested some a little bit code.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
> db.twittertest.find()Used some code on the internet and changed a bit to insert data into database.
[surachart@centos ~]$ cat ./test-tweeter.py
#!/usr/bin/python
import tweepy
import pymongo
import json
consumer_key = "..."
consumer_secret = "..."
access_token_key = "..."
access_token_secret = "..."
auth1 = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth1.set_access_token(access_token_key, access_token_secret)
class StreamListener(tweepy.StreamListener):
def on_status(self, tweet):
print tweet.text
def on_error(self, status_code):
print 'Error: ' + repr(status_code)
return False
def on_data(self, data):
print 'Ok! Inserting Data.'
from pymongo import MongoClient
client = MongoClient()
client = MongoClient('localhost', 27017)
db = client.test
test_id = db.twittertest.insert(json.loads(data))
l = StreamListener()
streamer = tweepy.Stream(auth=auth1, listener=l)
streamer.filter(track=['oracle'])I filtered by using word 'oracle'.
[surachart@centos ~]$ ./test-tweeter.py
Ok! Inserting Data.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
{ "_id" : ObjectId("51851511cdc8722152c28171"), "contributors" : null, "truncated" : false, "text" : "Job Opportunity for Oracle PL/SQL Developers with HTML/Web development experience for 6+ Months(Contract) for Las...: http://t.co/p6p8cfV6qF", "in_reply_to_status_id" : null, "id" : NumberLong("330684010961448962"), "favorite_count" : 0, "source" : "<a href=\"http://www.linkedin.com/\" rel=\"nofollow\">LinkedIn</a>", "retweeted" : false, "coordinates" : null, "entities" : { "symbols" : [ ], "user_mentions" : [ ], "hashtags" : [ ], "urls" : [ { "url" : "http://t.co/p6p8cfV6qF", "indices" : [ 118, 140 ], "expanded_url" : "http://lnkd.in/G-hhiX", "display_url" : "lnkd.in/G-hhiX" } ] }, "in_reply_to_screen_name" : null, "id_str" : "330684010961448962", "retweet_count" : 0, "in_reply_to_user_id" : null, "favorited" : false, "user" : { "follow_request_sent" : null, "profile_use_background_image" : true, "default_profile_image" : true, "id" : 360409067, "verified" : false, "profile_image_url_https" : "https://si0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "profile_sidebar_fill_color" : "DDEEF6", "profile_text_color" : "333333", "followers_count" : 22, "profile_sidebar_border_color" : "C0DEED", "id_str" : "360409067", "profile_background_color" : "C0DEED", "listed_count" : 1, "profile_background_image_url_https" : "https://si0.twimg.com/images/themes/theme1/bg.png", "utc_offset" : null, "statuses_count" : 1777, "description" : null, "friends_count" : 0, "location" : "", "profile_link_color" : "0084B4", "profile_image_url" : "http://a0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "following" : null, "geo_enabled" : false, "profile_background_image_url" : "http://a0.twimg.com/images/themes/theme1/bg.png", "name" : "Anand P. Sharma", "lang" : "en", "profile_background_tile" : false, "favourites_count" : 0, "screen_name" : "anandph123", "notifications" : null, "url" : null, "created_at" : "Tue Aug 23 04:52:03 +0000 2011", "contributors_enabled" : false, "time_zone" : null, "protected" : false, "default_profile" : true, "is_translator" : false }, "geo" : null, "in_reply_to_user_id_str" : null, "possibly_sensitive" : false, "lang" : "en", "created_at" : "Sat May 04 14:03:06 +0000 2013", "filter_level" : "medium", "in_reply_to_status_id_str" : null, "place" : null }
>I was able to insert data... and I will play with it later.
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Interactive Data Visualization for the Web An Introduction to Designing with D3 By Scott Murray
Interactive Data Visualization for the Web An Introduction to Designing with D3 By Scott Murray.This book was written about programming data visualization for non-programmer. You're able read this book and use it, even through you have no experience working with data or code.
This book will introduce you to D3, a JavaScript-based tool. You will be able to learn with many examples. Anyway, you might be able to begin with a series of tutorials.
Benefits in this book:
- Learn HTML, CSS, JavaScript, and SVG basics
- Dynamically generate web page elements from your data—and choose visual encoding rules to style them
- Create bar charts, scatter plots, pie charts, stacked bar charts, and force-directed layouts
- Use smooth, animated transitions to show changes in your data
- Introduce interactivity to help users explore data through different views
- Create customized geographic maps with data
- Explore hands-on with downloadable code and over 100 examples
I believe this book will give benefit for someone who is interested in interactive Data Visualization by using D3 and readers will enjoy with this book and examples
BTW, use https://github.com/alignedleft/d3-book/ to download codes.
Check book sampler. Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
MongoDB Applied Design Patterns Practical Use Cases with the Leading NoSQL Database By Rick Copeland
MongoDB is document-oriented database management system and supported by 10gen. Anyway, I was writing about a book, that's MongoDB Applied Design Patterns Practical Use Cases with the Leading NoSQL Database By Rick Copeland.A book will guide you how to do the connection between MongoDB and the business problem it's designed to solve. Learn how to apply MongoDB design patterns with some Cases such as e-commerce, content management ans etc. You will see demonstrates from this book, that's very useful for readers. I believe this book is useful for someone who are interested in learning practical patterns for solving problems and designing applications using MongoDB.
This book has 2 parts: Part I introduces the reader to some generally applicable design patterns in MongoDB and Part II: Use Cases. This is good part. It covers a particular problem domain and the techniques and patterns used to address the problem:
- Operational intelligence: Perform real-time analytics of business data
- Ecommerce: Use MongoDB as a product catalog master or inventory management system
- Content management: Learn methods for storing content nodes, binary assets, and discussions
- Online advertising networks: Apply techniques for frequency capping ad impressions, and keyword targeting and bidding
- Social networking: Learn how to store a complex social graph, modeled after Google+
- Online gaming: Provide concurrent access to character and world data for a multiplayer role-playing game
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Developing with Couchbase Server By MC Brown
I had a chance to install CouchBase (Learned Couchbase - Just installed it). I definitely believe it's easy for installation and you can learn by yourself.
As you know, CouchBase Server is a NoSQL document database for interactive applications that has a flexible data model, is easily scalable, provides consistently high performance, and is “always on”, 24*7*365.
For developers, you can use Java, .NET, PHP, Ruby, C and Python. However, you should know how to connect CouchBase, how to store and retrieve the information
from the database and etc. So, I mention book - Developing with Couchbase Server Building Scalable, Flexible Database-Based Applications By MC Brown.
You will learn how to build web applications with CouchBase Server 2.0, Help for idea to choose a client library for Java, .NET, Ruby, Python, PHP, or C.
[surachart@centos ~]$ cat test.rb
require 'rubygems'
require 'couchbase'
client = Couchbase.connect(:bucket => "default",
:host => "localhost")
v_name = client.get("surachart")
puts "#{v_name['name']}"
client.disconnect
[surachart@centos ~]$ ruby test.rb
Surachart Opun
Although this book have only about 150 pages, It will aid you to understand how to use client libraries and develop them with CouchBase 2.0. You will get idea about View also. I believe a book will help you basic to develop with CouchBase. If you are looking for a book. Good idea to get this one.
Written By: Surachart Opun http://surachartopun.com
As you know, CouchBase Server is a NoSQL document database for interactive applications that has a flexible data model, is easily scalable, provides consistently high performance, and is “always on”, 24*7*365.For developers, you can use Java, .NET, PHP, Ruby, C and Python. However, you should know how to connect CouchBase, how to store and retrieve the information
from the database and etc. So, I mention book - Developing with Couchbase Server Building Scalable, Flexible Database-Based Applications By MC Brown.
You will learn how to build web applications with CouchBase Server 2.0, Help for idea to choose a client library for Java, .NET, Ruby, Python, PHP, or C.
[surachart@centos ~]$ cat test.rb
require 'rubygems'
require 'couchbase'
client = Couchbase.connect(:bucket => "default",
:host => "localhost")
v_name = client.get("surachart")
puts "#{v_name['name']}"
client.disconnect
[surachart@centos ~]$ ruby test.rb
Surachart Opun
- Learn about Couchbase Server’s cluster-based architecture and how it differs from SQL databases
- Choose a client library for Java, .NET, Ruby, Python, PHP, or C, and connect to a cluster
- Structure data in a variety of formats, from serialized objects, a stream of raw bytes, or as JSON documents
- Learn core storage and retrieval methods, including document IDs, expiry times, and concurrent updates
- Create views with map/reduce and learn Couchbase mechanisms for querying and selection
Although this book have only about 150 pages, It will aid you to understand how to use client libraries and develop them with CouchBase 2.0. You will get idea about View also. I believe a book will help you basic to develop with CouchBase. If you are looking for a book. Good idea to get this one.
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Learned Couchbase - Just installed it
As you know, Couchbase is the open source NoSQL database for interactive web and mobile applications. I plan to read a book about Developing with Couchbase Server. First of all, I had to install it. I chose "Community Edition" for my test. If you will implement it you can choose Enterprise Edition and read more.
- Installed in on my host (centos):
[surachart@centos ~]$ wget --continue http://packages.couchbase.com/releases/2.0.0/couchbase-server-community_x86_64_2.0.0.rpm
--2013-02-10 14:46:14-- http://packages.couchbase.com/releases/2.0.0/couchbase-server-community_x86_64_2.0.0.rpm
Resolving packages.couchbase.com... 207.171.187.117
Connecting to packages.couchbase.com|207.171.187.117|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 126528178 (121M) [application/x-redhat-package-manager]
Saving to: ?€?couchbase-server-community_x86_64_2.0.0.rpm?€
100%[======================================================================================================>] 126,528,178 278K/s in 7m 55s
2013-02-10 14:54:10 (260 KB/s) - ?€?couchbase-server-community_x86_64_2.0.0.rpm?€
[surachart@centos ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
Starting couchbase-server[ OK ]
You have successfully installed Couchbase Server.
Please browse to http://centos:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.
Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091 and from 21100 to 21299.
By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.
[surachart@centos ~]$- Opened browser and setup.
Clicked "SETUP"
On Step 5: set password and clicked "Next"
Note: On Step 1, be able to choose "Start a new cluster" or "Join a cluster now". So!
- Installed in on my new host (centostest1):
[surachart@centostest1 ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
[sudo] password for surachart:
error: Failed dependencies:
libcrypto.so.6()(64bit) is needed by couchbase-server-2.0.0-1976.x86_64
libssl.so.6()(64bit) is needed by couchbase-server-2.0.0-1976.x86_64
[surachart@centostest1 ~]$ yum provides */libcrypto.so.6
Loaded plugins: fastestmirror, security
Determining fastest mirrors
* base: mirrors.thzhost.com
* extras: mirrors.thzhost.com
* updates: mirror-fpt-telecom.fpt.net
base | 3.7 kB 00:00
base/primary_db | 4.5 MB 00:03
extras | 3.5 kB 00:00
extras/primary_db | 23 kB 00:00
updates | 3.5 kB 00:00
updates/primary_db | 5.1 MB 00:21
base/filelists_db | 5.9 MB 00:04
extras/filelists_db | 13 kB 00:00
updates/filelists_db | 3.9 MB 00:15
openssl098e-0.9.8e-17.el6.centos.2.i686 : A compatibility version of a general cryptography and TLS library
Repo : base
Matched from:
Filename : /usr/lib/libcrypto.so.6
openssl098e-0.9.8e-17.el6.centos.2.x86_64 : A compatibility version of a general cryptography and TLS library
Repo : base
Matched from:
Filename : /usr/lib64/libcrypto.so.6
[surachart@centostest1 ~]$ sudo yum install openssl098e
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: mirror.neu.edu.cn
* extras: mirror.neu.edu.cn
* updates: mirror-fpt-telecom.fpt.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package openssl098e.x86_64 0:0.9.8e-17.el6.centos.2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Installing:
openssl098e x86_64 0.9.8e-17.el6.centos.2 base 761 k
Transaction Summary
================================================================================================================================================
Install 1 Package(s)
Total download size: 761 k
Installed size: 2.2 M
Is this ok [y/N]: y
Downloading Packages:
openssl098e-0.9.8e-17.el6.centos.2.x86_64.rpm | 761 kB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : openssl098e-0.9.8e-17.el6.centos.2.x86_64 1/1
/sbin/ldconfig: /usr/lib64/libhadoop.so.1 is not a symbolic link
/sbin/ldconfig: /usr/lib64/libhdfs.so.0 is not a symbolic link
Verifying : openssl098e-0.9.8e-17.el6.centos.2.x86_64 1/1
Installed:
openssl098e.x86_64 0:0.9.8e-17.el6.centos.2
Complete!
[surachart@centostest1 ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
Starting couchbase-server[ OK ]
You have successfully installed Couchbase Server.
Please browse to http://centostest1:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.
Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091 and from 21100 to 21299.
By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.
[surachart@centostest1 ~]$ - Opened browser and setup.
On Step 1: Chose "Join a cluster now"
Clicked "Next"
Saw server has been associated with the cluster.
This page showed rebalance operation.
I thought it's ok for installation. I will configuration more later, then tested with some commands.
[surachart@centos bin]$ ./couchbase-cli server-list -c 192.168.111.80:8091 -u Administrator -p password
ns_1@192.168.111.80 192.168.111.80:8091 healthy active
ns_1@192.168.111.81 192.168.111.81:8091 healthy active
[surachart@centos bin]$ ./couchbase-cli server-info -c 192.168.111.80:8091 -u Administrator -p password
{
"availableStorage": {
"hdd": [
{
"path": "/",
"sizeKBytes": 26391624,
"usagePercent": 38
},
{
"path": "/dev/shm",
"sizeKBytes": 961456,
"usagePercent": 0
},
{
"path": "/boot",
"sizeKBytes": 495844,
"usagePercent": 14
}
]
},
"clusterCompatibility": 131072,
"clusterMembership": "active",
"couchApiBase": "http://192.168.111.80:8092/",
"hostname": "192.168.111.80:8091",
"interestingStats": {
"couch_docs_actual_disk_size": 36177370,
"couch_docs_data_size": 36121645,
"couch_views_actual_disk_size": 719005,
"couch_views_data_size": 685418,
"curr_items": 7304,
"curr_items_tot": 7304,
"mem_used": 98621656,
"vb_replica_curr_items": 0
},
"mcdMemoryAllocated": 1502,
"mcdMemoryReserved": 1502,
"memoryFree": 120844288,
"memoryQuota": 1126,
"memoryTotal": 1969065984,
"os": "x86_64-unknown-linux-gnu",
"otpCookie": "kbxxdkfooqhiseqt",
"otpNode": "ns_1@192.168.111.80",
"ports": {
"direct": 11210,
"proxy": 11211
},
"status": "healthy",
"storage": {
"hdd": [
{
"index_path": "/opt/couchbase/var/lib/couchbase/data",
"path": "/opt/couchbase/var/lib/couchbase/data",
"quotaMb": "none",
"state": "ok"
}
],
"ssd": []
},
"storageTotals": {
"hdd": {
"free": 16755514246,
"quotaTotal": 27025022976,
"total": 27025022976,
"used": 10269508730,
"usedByData": 36896375
},
"ram": {
"quotaTotal": 1180696576,
"quotaUsed": 1180696576,
"total": 1969065984,
"used": 1848221696,
"usedByData": 98621656
}
},
"systemStats": {
"cpu_utilization_rate": 100.0,
"swap_total": 4227850240,
"swap_used": 782336
},
"thisNode": true,
"uptime": "2376",
"version": "2.0.0-1976-rel-community"
}Then tested telnet localhost 11211 :)
[surachart@centos ~]$ telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
set test_key 0 0 1
a
STORED
get test_key
VALUE test_key 0 1
a
END
[surachart@centostest1 ~]$ telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
get test_key
VALUE test_key 0 1
a
END It looks good now for learning :)
Written By: Surachart Opun http://surachartopun.com
- Installed in on my host (centos):
[surachart@centos ~]$ wget --continue http://packages.couchbase.com/releases/2.0.0/couchbase-server-community_x86_64_2.0.0.rpm
--2013-02-10 14:46:14-- http://packages.couchbase.com/releases/2.0.0/couchbase-server-community_x86_64_2.0.0.rpm
Resolving packages.couchbase.com... 207.171.187.117
Connecting to packages.couchbase.com|207.171.187.117|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 126528178 (121M) [application/x-redhat-package-manager]
Saving to: ?€?couchbase-server-community_x86_64_2.0.0.rpm?€
100%[======================================================================================================>] 126,528,178 278K/s in 7m 55s
2013-02-10 14:54:10 (260 KB/s) - ?€?couchbase-server-community_x86_64_2.0.0.rpm?€
[surachart@centos ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
Starting couchbase-server[ OK ]
You have successfully installed Couchbase Server.
Please browse to http://centos:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.
Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091 and from 21100 to 21299.
By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.
[surachart@centos ~]$- Opened browser and setup.
Clicked "SETUP"
On Step 5: set password and clicked "Next"
Note: On Step 1, be able to choose "Start a new cluster" or "Join a cluster now". So!- Installed in on my new host (centostest1):
[surachart@centostest1 ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
[sudo] password for surachart:
error: Failed dependencies:
libcrypto.so.6()(64bit) is needed by couchbase-server-2.0.0-1976.x86_64
libssl.so.6()(64bit) is needed by couchbase-server-2.0.0-1976.x86_64
[surachart@centostest1 ~]$ yum provides */libcrypto.so.6
Loaded plugins: fastestmirror, security
Determining fastest mirrors
* base: mirrors.thzhost.com
* extras: mirrors.thzhost.com
* updates: mirror-fpt-telecom.fpt.net
base | 3.7 kB 00:00
base/primary_db | 4.5 MB 00:03
extras | 3.5 kB 00:00
extras/primary_db | 23 kB 00:00
updates | 3.5 kB 00:00
updates/primary_db | 5.1 MB 00:21
base/filelists_db | 5.9 MB 00:04
extras/filelists_db | 13 kB 00:00
updates/filelists_db | 3.9 MB 00:15
openssl098e-0.9.8e-17.el6.centos.2.i686 : A compatibility version of a general cryptography and TLS library
Repo : base
Matched from:
Filename : /usr/lib/libcrypto.so.6
openssl098e-0.9.8e-17.el6.centos.2.x86_64 : A compatibility version of a general cryptography and TLS library
Repo : base
Matched from:
Filename : /usr/lib64/libcrypto.so.6
[surachart@centostest1 ~]$ sudo yum install openssl098e
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: mirror.neu.edu.cn
* extras: mirror.neu.edu.cn
* updates: mirror-fpt-telecom.fpt.net
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package openssl098e.x86_64 0:0.9.8e-17.el6.centos.2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Installing:
openssl098e x86_64 0.9.8e-17.el6.centos.2 base 761 k
Transaction Summary
================================================================================================================================================
Install 1 Package(s)
Total download size: 761 k
Installed size: 2.2 M
Is this ok [y/N]: y
Downloading Packages:
openssl098e-0.9.8e-17.el6.centos.2.x86_64.rpm | 761 kB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : openssl098e-0.9.8e-17.el6.centos.2.x86_64 1/1
/sbin/ldconfig: /usr/lib64/libhadoop.so.1 is not a symbolic link
/sbin/ldconfig: /usr/lib64/libhdfs.so.0 is not a symbolic link
Verifying : openssl098e-0.9.8e-17.el6.centos.2.x86_64 1/1
Installed:
openssl098e.x86_64 0:0.9.8e-17.el6.centos.2
Complete!
[surachart@centostest1 ~]$ sudo rpm --install couchbase-server-community_x86_64_2.0.0.rpm
Starting couchbase-server[ OK ]
You have successfully installed Couchbase Server.
Please browse to http://centostest1:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.
Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091 and from 21100 to 21299.
By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.
[surachart@centostest1 ~]$ - Opened browser and setup.
On Step 1: Chose "Join a cluster now"
Clicked "Next"
Saw server has been associated with the cluster.
This page showed rebalance operation.I thought it's ok for installation. I will configuration more later, then tested with some commands.
[surachart@centos bin]$ ./couchbase-cli server-list -c 192.168.111.80:8091 -u Administrator -p password
ns_1@192.168.111.80 192.168.111.80:8091 healthy active
ns_1@192.168.111.81 192.168.111.81:8091 healthy active
[surachart@centos bin]$ ./couchbase-cli server-info -c 192.168.111.80:8091 -u Administrator -p password
{
"availableStorage": {
"hdd": [
{
"path": "/",
"sizeKBytes": 26391624,
"usagePercent": 38
},
{
"path": "/dev/shm",
"sizeKBytes": 961456,
"usagePercent": 0
},
{
"path": "/boot",
"sizeKBytes": 495844,
"usagePercent": 14
}
]
},
"clusterCompatibility": 131072,
"clusterMembership": "active",
"couchApiBase": "http://192.168.111.80:8092/",
"hostname": "192.168.111.80:8091",
"interestingStats": {
"couch_docs_actual_disk_size": 36177370,
"couch_docs_data_size": 36121645,
"couch_views_actual_disk_size": 719005,
"couch_views_data_size": 685418,
"curr_items": 7304,
"curr_items_tot": 7304,
"mem_used": 98621656,
"vb_replica_curr_items": 0
},
"mcdMemoryAllocated": 1502,
"mcdMemoryReserved": 1502,
"memoryFree": 120844288,
"memoryQuota": 1126,
"memoryTotal": 1969065984,
"os": "x86_64-unknown-linux-gnu",
"otpCookie": "kbxxdkfooqhiseqt",
"otpNode": "ns_1@192.168.111.80",
"ports": {
"direct": 11210,
"proxy": 11211
},
"status": "healthy",
"storage": {
"hdd": [
{
"index_path": "/opt/couchbase/var/lib/couchbase/data",
"path": "/opt/couchbase/var/lib/couchbase/data",
"quotaMb": "none",
"state": "ok"
}
],
"ssd": []
},
"storageTotals": {
"hdd": {
"free": 16755514246,
"quotaTotal": 27025022976,
"total": 27025022976,
"used": 10269508730,
"usedByData": 36896375
},
"ram": {
"quotaTotal": 1180696576,
"quotaUsed": 1180696576,
"total": 1969065984,
"used": 1848221696,
"usedByData": 98621656
}
},
"systemStats": {
"cpu_utilization_rate": 100.0,
"swap_total": 4227850240,
"swap_used": 782336
},
"thisNode": true,
"uptime": "2376",
"version": "2.0.0-1976-rel-community"
}Then tested telnet localhost 11211 :)
[surachart@centos ~]$ telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
set test_key 0 0 1
a
STORED
get test_key
VALUE test_key 0 1
a
END
[surachart@centostest1 ~]$ telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
get test_key
VALUE test_key 0 1
a
END It looks good now for learning :)
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs
Oracle CRM On Demand Administration Essentials
I have a chance to review Oracle CRM On Demand Administration Essentials by Padmanabha Rao and Venkatesan Sundaram.First of all, What is Oracle CRM On Demand? It's Oracle Service, that you access over the Web. Oracle CRM on Demand help you to manage all your company’s sales, customer service, and marketing information. All pages in Oracle CRM On Demand share the same basic, interface framework. You can change the background color, hyperlink color, and so on by selecting a theme in your personal profile. OK! come back to a book that I want to review. This book will be useful for anyone who uses Oracle CRM On Demand. If You don't use Oracle CRM On Demand, you might start your FREE 30-day trial.
A book title is Oracle CRM On Demand Administration Essentials. You should understand before "Administration" in meaning, authors mean the business analyst combines with IT technologist. So, A book covers CRM On Demand Customization, Application Customization, User Access Controls, Business Process Automation and Management, Content Management, Web Services Integration, Reports and Analytics and etc.
Who this book is for? This is a good question. It's will be useful to an architect, a product manager, a business analyst, an interface developer and the professional administrator.
If You want to get this book, you should be able to access Oracle CRM On Demand! It will useful, because you will read and be able to practice from real world and see the picture. I believe this book will help you understand quickly, because you will see illustration on each chapter and authors do provide important informations helpful for readers. If you are using Oracle CRM On Demand, you want to read something over Oracle document. I mention this book. However, this book isn't for anyone. You should work and be able access Oracle CRM On Demand website.
Note: You can download sample chapters in PDF format.
Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs