Skip navigation.

Pythian Group

Syndicate content
Official Pythian Blog - Love Your Data
Updated: 7 hours 49 min ago

Debugging High CPU Usage Using Perf Tool and vmcore Analysis

Fri, 2014-10-17 08:08

There are several tools and technologies available to debug deeper into high CPU utilization in a system; perf, sysrq, oprofile, vmcore, and more. In this post, I will narrate the course of debugging a CPU utilization issue using technologies like perf and vmcore.

Following sar output is from a system which faces high %system usage.

[root@prod-smsgw1 ~]# sar 1 14
Linux 2.6.32-431.20.5.el6.x86_64 (xxxxx) 08/08/2014 _x86_64_ (8 CPU)05:04:57 PM CPU %user %nice %system %iowait %steal %idle
05:04:58 PM all 2.90 0.00 15.01 0.38 0.00 81.72
05:04:59 PM all 2.02 0.00 10.83 0.13 0.00 87.03
05:05:00 PM all 3.27 0.00 13.98 0.76 0.00 81.99
05:05:01 PM all 9.32 0.00 16.62 0.25 0.00 73.80

From ‘man sar’.

%system
Percentage of CPU utilization that occurred while executing at the system level (kernel). Note
that this field includes time spent servicing hardware and software interrupts.

This means that the system is spending considerable time on catering to kernel code. System runs a java application which is showing high CPU usage.

perf – Performance analysis tools for Linux, is a good place to start in these kind of scenarios.

‘perf record’ command would capture system state for all cpus in perf.data file. -g would allow call graph and -p allows profiling a process.

‘perf report’ command would show the report.

Samples: 18K of event ‘cpu-clock’, Event count (approx.): 18445, Thread: java(3284), DSO: [kernel.kallsyms]
58.66% java [k] _spin_lock ?
31.82% java [k] find_inode ?
2.66% java [k] _spin_unlock_irqrestore ?
2.44% java [k] mutex_spin_on_owner

Here we can see that considerable time is spend in spinlock and find_inode code for the java application..

While investigation was going on, system crashed and dumped a vmcore. Vmcore is a memory dump of the system captured by tools like kdump.

I downloaded the debuginfo file and extracted the vmlinux to analyse the vmcore.

# wget wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-431.20.5.el6.x86_64.rpm
# rpm2cpio kernel-debuginfo-2.6.32-431.20.5.el6.x86_64.rpm |cpio -idv ./usr/lib/debug/lib/modules/2.6.32-431.20.5.el6.x86_64/vmlinux

Then ran following command.

# crash ./usr/lib/debug/lib/modules/2.6.32-431.20.5.el6.x86_64/vmlinux /var/crash/127.0.0.1-2014-08-07-17\:56\:19/vmcoreKERNEL: ./usr/lib/debug/lib/modules/2.6.32-431.20.5.el6.x86_64/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2014-08-07-17:56:19/vmcore [PARTIAL DUMP]
CPUS: 8
DATE: Thu Aug 7 17:56:17 2014
UPTIME: 1 days, 13:08:01
LOAD AVERAGE: 91.11, 91.54, 98.02
TASKS: 1417
NODENAME: xxxxx
RELEASE: 2.6.32-431.20.5.el6.x86_64
VERSION: #1 SMP Fri Jul 25 08:34:44 UTC 2014
MACHINE: x86_64 (2000 Mhz)
MEMORY: 12 GB
PANIC: “Oops: 0010 [#1] SMP ” (check log for details)
PID: 11233
COMMAND: “java”
TASK: ffff88019706b540 [THREAD_INFO: ffff880037a90000]
CPU: 6
STATE: TASK_RUNNING (PANIC)

From the vmcore I see that dtracedrv module was loaded and unloaded (possibly for running dtrace), this resulted in several warnings (first warning from ftrace is expected) and then kernel panicked as memory got corrupted. Instruction pointer is corrupted, which points to memory corruption. Looks like Panic was triggered by dtrace module.

/tmp/dtrace/linux-master/build-2.6.32-431.20.5.el6.x86_64/driver/dtrace.c:dtrace_ioctl:16858: assertion failure buf->dtb_xamot != cached
Pid: 8442, comm: dtrace Tainted: P W ————— 2.6.32-431.20.5.el6.x86_64 #1
Pid: 3481, comm: java Tainted: P W ————— 2.6.32-431.20.5.el6.x86_64 #1
Call Trace:
[] ? dump_cpu_stack+0x3d/0×50 [dtracedrv]
[] ? generic_smp_call_function_interrupt+0×90/0x1b0
[] ? smp_call_function_interrupt+0×27/0×40
[] ? call_function_interrupt+0×13/0×20
[] ? _spin_lock+0x1e/0×30
[] ? __mark_inode_dirty+0x6c/0×160
[] ? __set_page_dirty_nobuffers+0xdd/0×160
[] ? nfs_mark_request_dirty+0x1a/0×40 [nfs]
[] ? nfs_updatepage+0x3d2/0×560 [nfs]
[] ? nfs_write_end+0×152/0x2b0 [nfs]
[] ? iov_iter_copy_from_user_atomic+0×92/0×130
[] ? generic_file_buffered_write+0x18a/0x2e0
[] ? nfs_refresh_inode_locked+0x3e1/0xbd0 [nfs]
[] ? __generic_file_aio_write+0×260/0×490
[] ? __put_nfs_open_context+0×58/0×110 [nfs]
[] ? dtrace_vcanload+0×20/0x1a0 [dtracedrv]
[..]
BUG: unable to handle kernel paging request at ffffc90014fb415e
IP: [] 0xffffc90014fb415e
PGD 33c2b5067 PUD 33c2b6067 PMD 3e688067 PTE 0
Oops: 0010 [#1] SMP
last sysfs file: /sys/devices/system/node/node0/meminfo
CPU 6
Modules linked in: cpufreq_stats freq_table nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ipv6 ppdev parport_pc parport microcode vmware_balloon sg vmxnet3 i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dtracedrv]Pid: 11233, comm: java Tainted: P W ————— 2.6.32-431.20.5.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[] [] 0xffffc90014fb415e
RSP: 0018:ffff880037a91f70 EFLAGS: 00010246
RAX: 0000000000000001 RBX: 0000000000000219 RCX: ffff880037a91d40
RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000000
RBP: 00007fba9a67f4c0 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: 00000000000003ff R12: 000000000001d4c0
R13: 0000000000000219 R14: 00007fb96feb06e0 R15: 00007fb96feb06d8
FS: 00007fb96fec1700(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffc90014fb415e CR3: 000000031e49e000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process java (pid: 11233, threadinfo ffff880037a90000, task ffff88019706b540)
Stack:
0000000000000000 0000000000002be1 ffffffff8100b072 0000000000000293
000000000000ebe6 0000000000002be1 0000000000000000 0000000000000007
00000030692df333 000000000001d4c0 0000000000000001 00007fb96feb06d8
Call Trace:
[] ? system_call_fastpath+0×16/0x1b
Code: Bad RIP value.
RIP [] 0xffffc90014fb415e
RSP
CR2: ffffc90014fb415e
crash>

This allowed me to have have a look at the CPU usage issue happening in the system. Other way to capture a vmcore is to manually panic the system using sysrq + c.

None of the runnable and uninterruptable_sleep processes are running for long time..

Looking at the oldest D state process..

crash> bt 4776
PID: 4776 TASK: ffff88027f3daaa0 CPU: 6 COMMAND: “java”
#0 [ffff88027f3dfd88] schedule at ffffffff815287f0
#1 [ffff88027f3dfe50] __mutex_lock_killable_slowpath at ffffffff8152a0ee
#2 [ffff88027f3dfec0] mutex_lock_killable at ffffffff8152a1f8
#3 [ffff88027f3dfee0] vfs_readdir at ffffffff8119f834
#4 [ffff88027f3dff30] sys_getdents at ffffffff8119f9f9
#5 [ffff88027f3dff80] system_call_fastpath at ffffffff8100b072
RIP: 00000030692a90e5 RSP: 00007fa0586c51e0 RFLAGS: 00000206
RAX: 000000000000004e RBX: ffffffff8100b072 RCX: 00007fa0cd2cf000
RDX: 0000000000008000 RSI: 00007fa0bc0de9a8 RDI: 00000000000001f6
RBP: 00007fa0bc004cd0 R8: 00007fa0bc0de9a8 R9: 00007fa0cd2fce58
R10: 00007fa0cd2fcaa8 R11: 0000000000000246 R12: 00007fa0bc004cd0
R13: 00007fa0586c5460 R14: 00007fa0cd2cf1c8 R15: 00007fa0bc0de980
ORIG_RAX: 000000000000004e CS: 0033 SS: 002b

Looking at its stack..

crash> bt -f 4776
PID: 4776 TASK: ffff88027f3daaa0 CPU: 6 COMMAND: “java”
[..]
#2 [ffff88027f3dfec0] mutex_lock_killable at ffffffff8152a1f8
ffff88027f3dfec8: ffff88027f3dfed8 ffff8801401e1600
ffff88027f3dfed8: ffff88027f3dff28 ffffffff8119f834
#3 [ffff88027f3dfee0] vfs_readdir at ffffffff8119f834
ffff88027f3dfee8: ffff88027f3dff08 ffffffff81196826
ffff88027f3dfef8: 00000000000001f6 00007fa0bc0de9a8
ffff88027f3dff08: ffff8801401e1600 0000000000008000
ffff88027f3dff18: 00007fa0bc004cd0 ffffffffffffffa8
ffff88027f3dff28: ffff88027f3dff78 ffffffff8119f9f9
#4 [ffff88027f3dff30] sys_getdents at ffffffff8119f9f9
ffff88027f3dff38: 00007fa0bc0de9a8 0000000000000000
ffff88027f3dff48: 0000000000008000 0000000000000000
ffff88027f3dff58: 00007fa0bc0de980 00007fa0cd2cf1c8
ffff88027f3dff68: 00007fa0586c5460 00007fa0bc004cd0
ffff88027f3dff78: 00007fa0bc004cd0 ffffffff8100b072crash> vfs_readdir
vfs_readdir = $4 =
{int (struct file *, filldir_t, void *)} 0xffffffff8119f7b0
crash>crash> struct file 0xffff8801401e1600
struct file {
f_u = {
fu_list = {
next = 0xffff88033213fce8,
prev = 0xffff88031823d740
},
fu_rcuhead = {
next = 0xffff88033213fce8,
func = 0xffff88031823d740
}
},
f_path = {
mnt = 0xffff880332368080,
dentry = 0xffff8802e2aaae00
},

[..]

crash> mount|grep ffff880332368080
ffff880332368080 ffff88033213fc00 nfs nanas1a.m-qube.com:/vol/test /scratch/test/test.deploy/test/test-internal

The process was waiting while reading from above nfs mount.

Following process seems to the culprit.

crash> bt 9104
PID: 9104 TASK: ffff8803323c8ae0 CPU: 0 COMMAND: “java”
#0 [ffff880028207e90] crash_nmi_callback at ffffffff8102fee6
#1 [ffff880028207ea0] notifier_call_chain at ffffffff8152e435
#2 [ffff880028207ee0] atomic_notifier_call_chain at ffffffff8152e49a
#3 [ffff880028207ef0] notify_die at ffffffff810a11ce
#4 [ffff880028207f20] do_nmi at ffffffff8152c0fb
#5 [ffff880028207f50] nmi at ffffffff8152b9c0
[exception RIP: _spin_lock+30]
RIP: ffffffff8152b22e RSP: ffff88001d209b88 RFLAGS: 00000206
RAX: 0000000000000004 RBX: ffff88005823dd90 RCX: ffff88005823dd78
RDX: 0000000000000000 RSI: ffffffff81fd0820 RDI: ffffffff81fd0820
RBP: ffff88001d209b88 R8: ffff88017b9cfa90 R9: dead000000200200
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88005823dd48
R13: ffff88001d209c68 R14: ffff8803374ba4f8 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
— —
#6 [ffff88001d209b88] _spin_lock at ffffffff8152b22e
#7 [ffff88001d209b90] _atomic_dec_and_lock at ffffffff81283095
#8 [ffff88001d209bc0] iput at ffffffff811a5aa0
#9 [ffff88001d209be0] dentry_iput at ffffffff811a26c0
#10 [ffff88001d209c00] d_kill at ffffffff811a2821
#11 [ffff88001d209c20] __shrink_dcache_sb at ffffffff811a2bb6
#12 [ffff88001d209cc0] shrink_dcache_parent at ffffffff811a2f64
#13 [ffff88001d209d30] proc_flush_task at ffffffff811f9195
#14 [ffff88001d209dd0] release_task at ffffffff81074ec8
#15 [ffff88001d209e10] wait_consider_task at ffffffff81075cc6
#16 [ffff88001d209e80] do_wait at ffffffff810760f6
#17 [ffff88001d209ee0] sys_wait4 at ffffffff810762e3
#18 [ffff88001d209f80] system_call_fastpath at ffffffff8100b072

From upstream kernel source..

/**
* iput – put an inode
* @inode: inode to put
*
* Puts an inode, dropping its usage count. If the inode use count hits
* zero, the inode is then freed and may also be destroyed.
*
* Consequently, iput() can sleep.
*/
void iput(struct inode *inode)
{
if (inode) {
BUG_ON(inode->i_state & I_CLEAR);if (atomic_dec_and_lock(&inode->i_count, &inode->i_lock))
iput_final(inode);
}
}
EXPORT_SYMBOL(iput);#include
/**
* atomic_dec_and_lock – lock on reaching reference count zero
* @atomic: the atomic counter
* @lock: the spinlock in question
*
* Decrements @atomic by 1. If the result is 0, returns true and locks
* @lock. Returns false for all other cases.
*/
extern int _atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock);
#define atomic_dec_and_lock(atomic, lock) \
__cond_lock(lock, _atomic_dec_and_lock(atomic, lock))

#endif /* __LINUX_SPINLOCK_H */

Looks like the process was trying to drop dentry cache and was holding to the spinlock while dropping an inode associated with it. This resulted in other processes waiting on spinlock, resulting in high %system utilization.

When the system again showed high %sys usage I checked and found large slab cache.

[root@xxxxx ~]# cat /proc/meminfo
[..]
Slab: 4505788 kB
SReclaimable: 4313672 kB
SUnreclaim: 192116 kB

Checking slab in a running system using slabtop, I saw that nfs_inode_cache is the top consumer.

ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
[..]
2793624 2519618 90% 0.65K 465604 6 1862416K nfs_inode_cache

I ran ‘sync’ and then ‘echo 2 > /proc/sys/vm/drop_caches’ to drop the dcache, which fixed the high %sys usage in the system.

[root@xxxxx ~]# sar 1 10
Linux 3.10.50-1.el6.elrepo.x86_64 (prod-smsgw4.sav.mqube.us) 08/12/2014 _x86_64_ (8 CPU)11:04:45 AM CPU %user %nice %system %iowait %steal %idle
11:04:46 AM all 1.51 0.00 13.22 0.50 0.00 84.76
11:04:47 AM all 1.25 0.00 12.55 0.13 0.00 86.07
11:04:48 AM all 1.26 0.00 8.83 0.25 0.00 89.66
11:04:49 AM all 1.63 0.00 11.93 0.63 0.00 85.80
^C
[root@xxxxx ~]# sync
[root@xxxxx ~]# sar 1 10
Linux 3.10.50-1.el6.elrepo.x86_64 (prod-smsgw4.sav.mqube.us) 08/12/2014 _x86_64_ (8 CPU)11:05:23 AM CPU %user %nice %system %iowait %steal %idle
11:05:24 AM all 1.50 0.00 13.03 0.75 0.00 84.71
11:05:25 AM all 1.76 0.00 9.69 0.25 0.00 88.30
11:05:26 AM all 1.51 0.00 9.80 0.25 0.00 88.44
11:05:27 AM all 1.13 0.00 10.03 0.25 0.00 88.60
^C
[root@xxxxx ~]# echo 2 > /proc/sys/vm/drop_caches
[root@xxxxx ~]# cat /proc/meminfo
[..]
Slab: 67660 kB

[root@prod-smsgw4 ~]# sar 1 10
Linux 3.10.50-1.el6.elrepo.x86_64 (prod-smsgw4.sav.mqube.us) 08/12/2014 _x86_64_ (8 CPU)

11:05:58 AM CPU %user %nice %system %iowait %steal %idle
11:05:59 AM all 1.64 0.00 1.38 0.13 0.00 96.86
11:06:00 AM all 2.64 0.00 1.38 0.38 0.00 95.60
11:06:01 AM all 2.02 0.00 1.89 0.25 0.00 95.84
11:06:02 AM all 2.03 0.00 1.39 4.68 0.00 91.90
11:06:03 AM all 8.21 0.00 2.27 2.65 0.00 86.87
11:06:04 AM all 1.63 0.00 1.38 0.13 0.00 96.86
11:06:05 AM all 2.64 0.00 1.51 0.25 0.00 95.60

From kernel documentation,

drop_cachesWriting to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

java application was traversing through nfs and was accessing large number of files, resulting in large number of nfs_inode_cache entries, resulting in in a large dcache.

Tuning vm.vfs_cache_pressure would be a persistent solution for this.

From kernel documentation,

vfs_cache_pressure
——————Controls the tendency of the kernel to reclaim the memory which is used for
caching of directory and inode objects.At the default value of vfs_cache_pressure=100 the kernel will attempt to
reclaim dentries and inodes at a “fair” rate with respect to pagecache and
swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer
to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will
never reclaim dentries and inodes due to memory pressure and this can easily
lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.

Categories: DBA Blogs

NZOUG14 Beckons

Fri, 2014-10-17 07:50

New Zealand is famous for Kiwis, pristine landscape, and the New Zealand Oracle User Group (NZOUG) conference.  The location of choice is New Zealand when it comes to making Lord of the Rings and making Oracle Lord of the Databases.

NZOUG 2014 will be held 19–21 November in the Owen G. Glenn Building at the University of Auckland. The main conference will be held on the 20th and 21st, preceded by a day of workshops on the 19th. It’s one of the premier Oracle conferences in Southern hemisphere.

Where there is Oracle, there is Pythian. Pythian will be present in full force in NZOUG 2014.

Following are Pythian sessions at NZOUG14:

12c Multi-Tenancy and Exadata IORM: An Ideal Cloud Based Resource Management
Fahd Mirza Chughtai

Everyone Talks About DR – But Why So Few Implement It
Francisco Munoz Alvarez

DBA 101: Calling All New Database Administrators
Gustavo Rene Antunez

My First 100 Days with an Exadata
Gustavo Rene Antunez

Do You Really Know the Index Structures?
Deiby Gómez

Oracle Exadata: Storage Indexes vs Conventional Indexes
Deiby Gómez

Oracle 12c Test Drive
Francisco Munoz Alvarez

Why Use OVM for Oracle Database
Francisco Munoz Alvarez

Please check the full agenda of NZOUG14 here.

Categories: DBA Blogs

Log Buffer #393, A Carnival of the Vanities for DBAs

Fri, 2014-10-17 07:47

Bloggers get connected to both the databases and their readers through their blogs. Bloggers act like a bridge here. Log Buffer extends this nexus through the Log Buffer Edition.

Oracle:

MS Sharepoint and Oracle APEX integration.

Just a couple of screenshots of sqlplus+rlwrap+cygwin+console.

Say “Big Data” One More Time (I dare you!)

Update OEM Harvester after 12.1.0.4 Upgrade

Insight in the Roadmap for Oracle Cloud Platform Services.

SQL Server:

Troubleshoot SQL P2P replication doesn’t replicate DDL schema change.

Set-based Constraint Violation Reporting in SQL Server.

Where do you start fixing a SQL Server crash when there isn’t a single clue?

A permission gives a principal access to an object to perform certain actions on or with the object.

When you can’t get to your data because another application has it locked, a thorough knowledge of SQL Server concurrency will give you the confidence to decide what to do.

MySQL:

MySQL 5.7.5- More variables in replication performance_schema tables.

Multi-source replication for MySQL has been released as a part of 5.7.5-labs-preview downloadable from labs.mysql.com.

How to install multiple MySQL instances on a single host using MyEnv?

Percona Toolkit for MySQL with MySQL-SSL Connections.

InnoDB: Supporting Page Sizes of 32k and 64k.

Categories: DBA Blogs

Deploying a Private Cloud at Home — Part 4

Thu, 2014-10-16 09:11

Today’s blog post is part four of seven in a series dedicated to Deploying Private Cloud at Home, where I will be demonstrating how to configure Imaging and compute services on controller node. See my previous blog post where we began configuring Keystone Identity Service.

  1. Install the Imaging service
    yum install -y openstack-glance python-glanceclient
  2. Configure Glance (Imaging Service) to use MySQL database
    openstack-config --set /etc/glance/glance-api.conf database connection \
    mysql://glance:Your_Password@controller/glance
    openstack-config --set /etc/glance/glance-registry.conf database connection \
    mysql://glance:Youre_Password@controller/glance
  3. Create Glance database user by running below queries on your MySQL prompt as root
    CREATE DATABASE glance;
    GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' IDENTIFIED BY 'Your_Password';
    GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY 'Your_Password';
  4. Create the database tables for the Image Service
    su -s /bin/sh -c "glance-manage db_sync" glance
  5. Create Glance user to communicate to OpenStack services and Identity services
    keystone user-create --name=glance --pass=Your_Password --email=Your_Email
    keystone user-role-add --user=glance --tenant=service --role=admin
  6. Configuration of Glance config files
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller:5000
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_host controller
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_port 35357
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_protocol http
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_tenant_name service
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_user glance
    openstack-config --set /etc/glance/glance-api.conf keystone_authtoken admin_password Your_Password
    openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller:5000
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_host controller
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_port 35357
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_protocol http
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_tenant_name service
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_user glance
    openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken admin_password Your_Password
    openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone
  7. Register the Image Service with the Identity service
    keystone service-create --name=glance --type=image --description="OpenStack Image Service"
    keystone endpoint-create \
      --service-id=$(keystone service-list | awk '/ image / {print $2}') \
      --publicurl=http://controller:9292 \
      --internalurl=http://controller:9292 \
      --adminurl=http://controller:9292
  8. Start the Glance-api and Glance-registry services and enable them to start at startup
    service openstack-glance-api start
    service openstack-glance-registry start
    chkconfig openstack-glance-api on
    chkconfig openstack-glance-registry on
  9. Download CirrOS cloud image which is created for testing purpose
    wget -q http://cdn.download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img \
    -O /root/cirros-0.3.2-x86_64-disk.img
  10. Upload the image to Glance using admin account
    source /root/admin-openrc.sh
    glance image-create --name "cirros-0.3.2-x86_64" \
    --disk-format qcow2 \
    --container-format bare \
    --is-public True \
    --progress < /root/cirros-0.3.2-x86_64-disk.img
  11. Install Compute controller service on controller node
    yum install -y openstack-nova-api openstack-nova-cert \
    openstack-nova-conductor openstack-nova-console \
    openstack-nova-novncproxy openstack-nova-scheduler \
    python-novaclient
  12. Configure compute service database
    openstack-config --set /etc/nova/nova.conf database connection mysql://nova:Your_Password@controller/nova
  13. Configure compute service configuration file
    openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend qpid
    openstack-config --set /etc/nova/nova.conf DEFAULT qpid_hostname controller
    openstack-config --set /etc/nova/nova.conf DEFAULT my_ip Controller_IP
    openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_listen Controller_IP
    openstack-config --set /etc/nova/nova.conf DEFAULT vncserver_proxyclient_address Controller_IP
  14. Create nova database user by running below queries on your MySQL prompt as root
    CREATE DATABASE nova;
    GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'Your_Password';
    GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'Your_Password';
  15. Create Compute service tables
    su -s /bin/sh -c "nova-manage db sync" nova
  16. Create a nova user that Compute uses to authenticate with the Identity Service
    keystone user-create --name=nova --pass=Your_Passoword --email=Your_Email
    keystone user-role-add --user=nova --tenant=service --role=admin
  17. Configure Compute to use these credentials with the Identity Service running on the controller
    openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
    openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000
    openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_host controller
    openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_protocol http
    openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_port 35357
    openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_user nova
    openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_tenant_name service
    openstack-config --set /etc/nova/nova.conf keystone_authtoken admin_password Your_Password
  18. Register Compute with the Identity Service
    keystone service-create --name=nova --type=compute --description="OpenStack Compute"
    keystone endpoint-create \
      --service-id=$(keystone service-list | awk '/ compute / {print $2}') \
      --publicurl=http://controller:8774/v2/%\(tenant_id\)s \
      --internalurl=http://controller:8774/v2/%\(tenant_id\)s \
      --adminurl=http://controller:8774/v2/%\(tenant_id\\)s
  19. Now start Compute services and configure them to start when the system boots
    service openstack-nova-api start
    service openstack-nova-cert start
    service openstack-nova-consoleauth start
    service openstack-nova-scheduler start
    service openstack-nova-conductor start
    service openstack-nova-novncproxy start
    chkconfig openstack-nova-api on
    chkconfig openstack-nova-cert on
    chkconfig openstack-nova-consoleauth on
    chkconfig openstack-nova-scheduler on
    chkconfig openstack-nova-conductor on
    chkconfig openstack-nova-novncproxy on
  20. You can verify your configuration and list available images
    source /root/admin-openrc.sh
    nova image-list

 

This concludes the initial configuration of controller node before configuration of compute node. Stay tuned for part five where I will demonstrate how to configure compute node.

Categories: DBA Blogs

Deploying a Private Cloud at Home — Part 3

Tue, 2014-10-14 14:59

Today’s blog post is part three of seven in a series dedicated to Deploying Private Cloud at Home, where I will demonstrate how to configure OpenStack Identity service on the controller node. We have already configured the required repo in part two of the series, so let’s get started on configuring Keystone Identity Service.

  1. Install keystone on the controller node.
    yum install -y openstack-keystone python-keystoneclient

    OpenStack uses a message broker to coordinate operations and status information among services. The message broker service typically runs on the controller node. OpenStack supports several message brokers including RabbitMQ, Qpid, and ZeroMQ.I am using Qpid as it is available on most of the distros

  2. Install Qpid Messagebroker server.
    yum install -y qpid-cpp-server

    Now Modify the qpid configuration file to disable authentication by changing below line in /etc/qpidd.conf

    auth=no

    Now start and enable qpid service to start on server startup

    chkconfig qpidd on
    service qpidd start
  3. Now configure keystone to use MySQL database
    openstack-config --set /etc/keystone/keystone.conf \
       database connection mysql://keystone:YOUR_PASSWORD@controller/keystone
  4. Next create keystone database user by running below queries on your mysql prompt as root.
    CREATE DATABASE keystone;
    GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'YOUR_PASSWORD';
    GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'YOUR_PASSWORD';
  5. Now create database tables
    su -s /bin/sh -c "keystone-manage db_sync" keystone

    Currently we don’t have any user accounts that can communicate with OpenStack services and Identity service. So we will setup an authorization token to use as a shared secret between the Identity Service and other OpenStack services and store in configuration file.

    ADMIN_TOKEN=$(openssl rand -hex 10)
    echo $ADMIN_TOKEN
    openstack-config --set /etc/keystone/keystone.conf DEFAULT \
       admin_token $ADMIN_TOKEN
  6. Keystone uses PKI tokens as default. Now create the signing keys and certificates to restrict access to the generated data
    keystone-manage pki_setup --keystone-user keystone --keystone-group keystone
    chown -R keystone:keystone /etc/keystone/ssl
    chmod -R o-rwx /etc/keystone/ssl
  7. Start and enable the keystone identity service to begin at startup
    service openstack-keystone start
    chkconfig openstack-keystone on

    Keystone Identity service stores expired tokens as well in the database. We will create below crontab entry to purge the expired tokens

    (crontab -l -u keystone 2>&1 | grep -q token_flush) || \
    echo '@hourly /usr/bin/keystone-manage token_flush >/var/log/keystone/keystone-tokenflush.log 2>&1' >> /var/spool/cron/keystone
  8. Now we will create admin user for keystone and define roles for admin user
    export OS_SERVICE_TOKEN=$ADMIN_TOKEN
    export OS_SERVICE_ENDPOINT=http://controller:35357/v2.0
    keystone user-create --name=admin --pass=Your_Password --email=Your_Email
    keystone role-create --name=admin
    keystone tenant-create --name=admin --description="Admin Tenant"
    keystone user-role-add --user=admin --tenant=admin --role=admin
    keystone user-role-add --user=admin --role=_member_ --tenant=admin
    keystone user-create --name=pythian --pass= Your_Password --email=Your_Email
    keystone tenant-create --name=pythian --description="Pythian Tenant"
    keystone user-role-add --user=pythian --role=_member_ --tenant=pythian
    keystone tenant-create --name=service --description="Service Tenant"
  9. Now we create a service entry for the identity service
    keystone service-create --name=keystone --type=identity --description="OpenStack Identity"
    keystone endpoint-create --service-id=$(keystone service-list | awk '/ identity / {print $2}') \
    --publicurl=http://controller:5000/v2.0 \
    --internalurl=http://controller:5000/v2.0 \
    --adminurl=http://controller:35357/v2.0
  10. Verify Identity service installation
    unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT
  11. Request an authentication token by using the admin user and the password you chose for that user
    keystone --os-username=admin --os-password=Your_Password \
      --os-auth-url=http://controller:35357/v2.0 token-get
    keystone --os-username=admin --os-password=Your_Password \
      --os-tenant-name=admin --os-auth-url=http://controller:35357/v2.0 \
      token-get
  12. We will save the required parameters in admin-openrc.sh as below
    export OS_USERNAME=admin
    export OS_PASSWORD=Your_Password
    export OS_TENANT_NAME=admin
    export OS_AUTH_URL=http://controller:35357/v2.0
  13. Next Next check if everything is working fine and keystone interacts with OpenStack services. We will source the admin-openrc.sh file to load the keystone parameters
    source /root/admin-openrc.sh
  14. List Keystone tokens using:
    keystone token-get
  15. List Keystone users using
    keystone user-list

If all the above commands give you the output, that means your Keystone Identity Service is all set up, and you can proceed to the next steps—In part four, I will discuss on how to configure and set up Image Service to store images.

Categories: DBA Blogs

Oracle E-Business Suite Updates From OpenWorld 2014

Tue, 2014-10-14 08:29

Oracle OpenWorld has always been my most exciting conference to attend. I always see high energy levels everywhere, and it kind of revs me up to tackle new upcoming technologies. This year I concentrated on attending mostly Oracle E-Business Suite release 12.2 and Oracle 12c Database-related sessions.

On the Oracle E-Business Suite side, I started off with Oracle EBS Customer Advisory Board Meeting with great presentations on new features like the Oracle EBS 12.2.4 new iPad Touch-friendly interface. This can be enabled by setting “Self Service Personal Home Page mode” profile value to “Framework Simplified”. Also discussed some pros and cons of the new downtime mode feature of adop Online patching utility that allows  release update packs ( like 12.2.3 and 12.2.4 patch ) to be applied with out starting up a new online patching session. I will cover more details about that in a separate blog post. In the mean time take a look at the simplified home page look of my 12.2.4 sandbox instance.

Oracle EBS 12.2.4 Simplified Interface

Steven Chan’s presentation on EBS Certification Roadmap announced upcoming support for Android tablets Chrome Browser, IE11 and Oracle Unified Directory etc. Oracle did not extend any support deadlines for Oracle EBS 11i or R12 this time. So to all EBS customers on 11i: It’s time to move to R12.2. I also attended a good session on testing best practices for Oracle E-Business Suite, which had a good slide on some extra testing required during Online Patching Cycle. I am planning to do a separate blog with more details on that, as it is an important piece of information that one might ignore. Also Oracle announced a new product called Flow Builder that is part of Oracle Application Testing Suite, which helps users test functional flows in Oracle EBS.

On the 12c Database side, I attended great sessions by Christian Antognini on Adaptive Query Optimization and Markus Michalewicz sessions on 12c RAC Operational Best Practices and RAC Cache Fusion Internals. Markus Cachefusion presentation has some great recommendations on using _gc_policy_minimum instead of turning off DRM completely using _gc_policy_time=0. Also now there is a way to control DRM of a object using package DBMS_CACHEUTIL.

I also attended attended some new, upcoming technologies that are picking up in the Oracle space like Oracle NoSQL, Oracle Big Data SQL, and Oracle Data Integrator Hadoop connectors. These products seem to have great future ahead and have good chances of becoming mainstream in the data warehousing side of businesses.

Categories: DBA Blogs

Deploying a Private Cloud at Home — Part 2

Fri, 2014-10-10 08:34

Today’s blog post is part two of seven in a series dedicated to Deploying Private Cloud at Home, where I will demonstrate how to do basic configuration setup to get started with OpenStack. In my first blog post, I explained why I decided to use OpenStack.

I am using a two-node setup in my environment, but you can still follow these steps and configure everything on single node. The below configuration reflects my setup. Kindly modify it as per your subnet and settings.

  • My home network has subnet of 192.168.1.0/24
  • My home PC which I am turning into controller node has IP of 192.168.1.140
  • MY KVM Hypervisor which I am turning to compute node has IP of 192.168.1.142
  1. It is advisable to have DNS setup in your intranet but just in case you don’t have it, you need to modify /etc/hosts file on both controller and compute node in order for OpenStack services to communicate to each other like below
    #Controller node
    192.168.1.140 controller
    #Compute node
    192.168.1.142 compute
  2. OpenStack services require a database to store information. You can use any database you are familiar with. I am using MySQL/MariaDB, as I am familiar with it. On the controller node, we will install the MySQL client and server packages, and the Python library.
     yum install -y mysql mysql-server MySQL-python
  3. Enable InnoDB, UTF-8 character set, and UTF-8 collation by default. To do that we need to modify /etc/my.cnf and set the following keys under [mysqld] section.
    default-storage-engine = innodb 
    innodb_file_per_table 
    collation-server = utf8_general_ci 
    init-connect = 'SET NAMES utf8' 
    character-set-server = utf8
  4. Start and enable the MySQL services
    service mysqld start
    chkconfig mysqld on
  5. Finally, set the root password for MySQL database. If you need further details about configuring the MySQL root password, there are many resources available online.
  6. On the compute node we need to install the MySQL Python library
    yum install -y MySQL-python
  7. Set up RDO repository on both controller and compute nodes
    yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-4.noarch.rpm
  8. I am using CentOS 6.2 so I need to have epel repo as well. This step is not required if you are using distro other then RHEL, CentOS, Scientific Linux etc.
    yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
  9. Install OpenStack utilities on both nodes and get started.
    yum install openstack-utils

Stay tuned for the remainder of my series, Deploying a Private Cloud at Home. In part three, we will continue configuring OpenStack services.

Categories: DBA Blogs

SQL Saturday Bulgaria 2014

Fri, 2014-10-10 08:22

 

This Saturday October 11, I will be speaking at SQL Saturday Bulgaria 2014 in Sofia. It’s my first time in the country and I’m really excited to be part of another SQL Saturday :)

I will be speaking about Buffer Pool Extension, a new feature on SQL Server 2014. If you want to learn a little more about the new SQL Server version, don’t hesitate to attend the event. Looking forward to seeing you there!

Categories: DBA Blogs

Log Buffer #392, A Carnival of the Vanities for DBAs

Fri, 2014-10-10 08:19

It seems its all about cloud these days. Even the hardware is being marketed with cloud in perspective. Databases like Oracle, SQL Server and MySQL are ahead in the cloud game and this Log Buffer Edition covers that all.


Oracle:

Oracle Database 12c was launched over a year ago delivering the next-generation of the #1 database, designed to meet modern business needs, providing a new multitenant architecture on top of a fast, scalable, reliable, and secure database platform.

Oracle OpenWorld 2014 Session Presentations Now Available.

Today, Oracle is using big data technology and concepts to significantly improve the effectiveness of its support operations, starting with its hardware support group.

Generating Sales Cloud Proxies using Axis? Getting errors?

How many page views can Apex sustain when running on Oracle XE?

SQL Server:

Send emails using SSIS and SQL Server instead of application-level code.

The public perception is that, when something is deleted, it no longer exists. Often that’s not really the case; the data you serve up to the cloud can be stored out there indefinitely, no matter how hard to try to delete it.

Every day, out in the various online forums devoted to SQL Server, and on Twitter, the same types of questions come up repeatedly: Why is this query running slowly? Why is SQL Server ignoring my index? Why does this query run quickly sometimes and slowly at others?

You need to set up backup and restore strategies to recover data or minimize the risk of data loss in case a failure happens.

Improving the Quality of SQL Server Database Connections in the Cloud

MySQL:

Low-concurrency performance for updates and the Heap engine: MySQL 5.7 vs previous releases.

Database Automation – Private DBaaS for MySQL, MariaDB and MongoDB with ClusterControl.

Removing Scalability Bottlenecks in the Metadata Locking and THR_LOCK Subsystems in MySQL 5.7.

The EXPLAIN command is one of MySQL’s most useful tools for understanding query performance. When you EXPLAIN a query, MySQL will return the plan created by the query optimizer.

Shinguz: Migration between MySQL/Percona Server and MariaDB.

Categories: DBA Blogs

What is Continuous Integration?

Thu, 2014-10-09 10:44

Most companies want to deploy features faster, and fix bugs more quickly—at the same time, a stable product that delivers what the users expected is crucial to winning and keeping the trust of those users.  At face value, stability and Lego Trainspeed appear to be in conflict; developers can either spend their time on features or on stability.  In reality, problems delivering on stability as well as problems implementing new features are both related to a lack of visibility.  Developers can’t answer a very basic question: What will be impacted by my change?

When incompatible changes hit the production servers as a result of bug fixes or new features, they have to be tracked down and resolved.  Fighting these fires is unproductive, costly, and prevents developers from building new features.

The goal of Continuous Integration (CI) is to break out of the mentality of firefighting—it gives developers more time to work on features, by baking stability into the process through testing.

Sample Workflow
  1. Document the intended feature
  2. Write one or more integration tests to validate that the feature functions as desired
  3. Develop the feature
  4. Release the feature

This workflow doesn’t include an integration step—code goes out automatically when all the tests pass. Since all the tests can be run automatically, by a testing system like Jenkins, a failure in any test, even those outside of the developers control, constitutes a break which must be fixed before continuing.  Of course in some cases, users follow paths other than those designed and explicitly tested by developers and bugs happen.  New testing is required to validate that bugs are fixed and these contribute to a library of tests which collectively increase collective confidence in the codebase.  Most importantly, the library of tests limits the scope of any bug which increases the confidence of developers to move faster.

Testing is the Secret Sauce

As the workflow illustrates, the better the tests, the more stable the application.  Instead of trying to determine which parts of the application might be impacted by a change, the tests can prove that things still work, as designed.

 

Continuous Integration is just one of the many ways our DevOps group engages with clients. We also build clouds and solve difficult infrastructure problems. Does that sound interesting to you? Want to come work with us? Get in touch!

Categories: DBA Blogs

Deploying a Private Cloud at Home — Part 1

Wed, 2014-10-08 08:17

Today’s blog post is part one of seven in a series dedicated to Deploying a Private Cloud at Home. In my day-to-day activities, I come across various scenarios where I’m required to do sandbox testing before proceeding further on the production environment—which is great because it allows me to sharpen and develop my skills.

My home network consists of an OpenFiler NAS which also serves DNS, DHCP, iSCSI, NFS and Samba in my network. My home PC is a Fedora 20 Workstation, where I do most of the personal activities.  KVM hypervisor is running on CentOS 6.2 x86_64 to run sandbox VMs for testing.

Recently I decided to move it to the cloud and create a private cloud at home. There are plenty of open source cloud solutions available, but I decided to use OpenStack for two reasons.

  1. I am already running Redhat compatible distros ( CentOS and Fedora ) so I just need to install OpenStack on top of it to get started.
  2. Most of the clients I support have RHEL compatible distros in the environment, so it makes sense having RHEL compatible distros to play around.

Ideally OpenStack cloud consists of minimum three nodes with at least 2 NICs on each node.

  • Controller: As the name suggests, this is the controller node which runs most of the control services.
  • Network: This is the network node which handles virtual networking.
  • Compute : This is the hypervisor node which runs your VMs.

However due to small size of my home network I decided to use legacy networking which only requires controller and compute nodes with single NIC

Stay tuned for the remainder of my series, Deploying a Private Cloud at Home. In part two of seven, I will be demonstrating configuration and setup.

Categories: DBA Blogs

Microsoft Hadoop: Taming the Big Challenge of Big Data – Part Three

Mon, 2014-10-06 11:57

Today’s blog post completes our three-part series with excerpts from our latest white paper, Microsoft Hadoop: Taming the Big Challenge of Big Data. In the first two posts, we discussed the impact of big data on today’s organizations, and its challenges.

Today, we’ll be sharing what organizations can accomplish by using the Microsoft Hadoop solution:

  1. Improve agility. Because companies now have the ability to collect and analyze data essentially in real time, they can more quickly discover which business strategies are working and which are not, and make adjustments as necessary.
  2. Increase innovation. By integrating structured and unstructured data sources, the solution provides decision makers with greater insight into all the factors affecting the business and encouraging new ways of thinking about opportunities and challenges.
  3. Reduce inefficiencies. Data that currently resides in conventional data management systems can be migrated into Parallel Data Warehouse (PDW) for faster information delivery
  4. Better allocate IT resources. The Microsoft Hadoop solution includes a powerful, intuitive interface for installing, configuring, and managing the technology, freeing up IT staff to work on projects that provide higher value to the organization.
  5. Decrease costs. Previously, because of the inability to effectively analyze big data, much of it was dumped into data warehouses on commodity hardware, which is no longer required thanks to Hadoop.

Download our full white paper to learn which companies are currently benefiting from Hadoop, and how you can achieve the maximum ROI from the Microsoft Hadoop solution.

Don’t forget to check out part one and part two of our Microsoft Hadoop blog series.

Categories: DBA Blogs

Log Buffer #391, A Carnival of the Vanities for DBAs

Fri, 2014-10-03 08:04

Oracle Open World is in full bloom. Enthusiasts of Oracle and MySQL are flocking to extract as much knowledge, news, and fun as possible. SQL Server aficionados are not far behind too.

Oracle:

Frank Nimphius have announced REST support for ADF BC feature on OOW today. Probably this functionality will be available in the next JDeveloper 12c update release.

RMAN Enhancements New Privilege A new SYSBACKUP privilege is created in Oracle 12c,  it allows the grantee to perform BACKUP and RECOVERY operations with RMAN SQL in RMAN.

To continue with the objective of separating duties and the least privileges, Oracle 12c introduce new administrative privileges all destined to accomplish specific duties.

Unified Auditing offers a consolidated approach, all the audit data is consolidated in a single place. Unified Auditing consolidate audit records for the following sources.

SOA Suite 12c – WSM-02141 : Unable to connect to the policy access service.

SQL Server:

Data Compression and Snapshot Isolation don’t play well together, you may not see a performance benefit.

Tim Smith answers some questions on SQL Server security like: Is It Better To Mask At the Application Level Or The SQL Server Database Level?

Since SQL Server delivered the entire range of window functions, there has been far less justification for using the non-standard ex-Sybase ‘Quirky Update’ tricks to perform the many permutations of running totals in SQL Server.

Easily synchronize live Salesforce data with SQL Server using the Salesforce SSIS DataFlow Tasks.

Change All Computed Columns to Persisted in SQL Server.

MySQL:

Low-concurrency performance for point lookups: MySQL 5.7.5 vs previous releases.

How to get MySQL 5.6 parallel replication and XtraBackup to play nice together.

The InnoDB labs release includes a snapshot of the InnoDB Native Partitioning feature.

Visualizing the impact of ordered vs. random index insertion in InnoDB.

Single thread performance in MySQL 5.7.5 versus older releases via sql-bench.

Categories: DBA Blogs

Microsoft Hadoop: Taming the Big Challenge of Big Data – Part Two

Thu, 2014-10-02 11:13

Today’s blog post is the second in a three-part series with excerpts from our latest white paper, Microsoft Hadoop: Taming the Big Challenge of Big Data. In our first blog post, we revealed just how much data is being generated globally every minute – and that it has doubled since 2011.

Traditional database management tools were not designed to handle the elements that make big data so much more complex—namely its key differentiators: volume, variety, and velocity.Variety Volume Velocity graphic

Volume is the quantity of data, variety refers to the type of data collected (image, audio, video, etc.), and velocity is its expected growth rate. Many people assume that big data always includes high volume and intuitively understand the
challenges that it presents. In reality, however, data variety and velocity are much more likely to prevent traditional management tools from being able to efficiently capture, store, report, analyze, and archive the data, regardless of volume.

Download our full white paper which explores the technical and business advantages of effectively managing big data, regardless of quantity, scope, or speed.

 

Categories: DBA Blogs

Microsoft Hadoop: Taming the Big Challenge of Big Data – Part One

Tue, 2014-09-30 11:12

Today’s blog post is the first in a three-part series with excerpts from our latest white paper, Microsoft Hadoop: Taming the Big Challenge of Big Data.

As companies increasingly rely on big data to steer decisions, they also find themselves looking for ways to simplify its storage, management, and analysis. The need to quickly access large amounts of data and use them competitively poses a technological challenge to organizations of all sizes.

Every minute, about two terabytes of data are being generated globally. That’s twice the amount from three years ago and half the amount predicted for three years from now.

Volume aside, the sources of data and the shape they take vary broadly. From government records, business transactions and social media, to scientific research and weather tracking, today’s data come in text, graphics, audio, video, and maps.

Download our full white paper which explores the impact of big data on today’s organizations and its challenges.

Categories: DBA Blogs

Log Buffer #390, A Carnival of the Vanities for DBAs

Fri, 2014-09-26 10:50

All eyes are on the Oracle Open World. Thousands of sessions, demos, and labs topped up with the countless networking opportunities, Moscone Center is the place to be.

Oracle:

Oracle OpenWorld 2014 – Bloggers Meetup.

Query to show sql_ids related to SQL Profiles.

Oracle OpenWorld is nearly upon us and the agenda is packed with interesting sessions. Prominent among these are several sessions by customers who are able to share their stories of success with Oracle Multitenant.

The biggest challenge for people coming to such huge events as Oracle OpenWorld is to navigate through all of the events that simultaneously happen at various locations.

Pythian at Oracle OpenWorld 2014.

SQL Server:

Monitoring Longest Running Transaction using SQL Server Agent Alerts .

Free PDF Booklet: 119 SQL Code Smells.

Level 1: What is “SQL Server AlwaysOn“?

10 things I learned about rapidly scaling websites with Azure.

Importing Excel Data into SQL Server Via SSIS: Questions You Were Too Shy to Ask

MySQL:

How to do reverse engineering with MySQL Workbench.

MySQL 5.5.40 Overview and Highlights.

MySQL 5.7.5: GROUP BY respects functional dependencies!

JSON UDF functions version 0.3.2 have been released.

Importing related MySQL tables into an Excel Data Model using MySQL for Excel.

Categories: DBA Blogs

On the Road with Luan

Fri, 2014-09-26 08:59

For the months of September, October, and November, Microsoft SQL Server MVP Luan Moreno will be touring Brazil, Europe, and EUA for various speaking engagements. He’ll be sharing his SQL Server knowledge and insights during SQL Saturdays, conferences, and virtual conferences.

“Pythian is a company that values and truly supports employees in sharing their knowledge at community events,” Luan says. “It’s a pleasure to be a part of this amazing company.”

 

Date Location Event Topic Notes 27-Sep-14 São Paulo, Brazil SQL Saturday #325 In-Memory OLTP a.k.a Hekaton Speaking schedule 28-Sep-14 São Paulo, Brazil SQL Saturday #325 MythBusters – Caçadores de Mitos Speaking schedule 6-Oct-14 Online MVP ShowCast 2014 In-Memory OLTP – Cenários de Aplicação Register here 7-Oct-14 Online 24 Hours of Pass Troubleshooting SQL Server Extended Events Register here 7-Oct-14 Online 24 Hours of Pass In-Memory OLAP a.k.a ColumnStore Index Internals Register here 17-Oct-14 Porto Alegre, Brazil TDC 2014 ORM e Consultas x Performance Speaking schedule 25-Oct-14 Rio de Janerio, Brazil SQL Saturday #329 In-Memory OLAP a.k.a ColumnStore Index Speaking schedule Oct 28-31 Barcelona, Spain TechED 2014 Europe Subject Matter Expert (SME) – SQL Server Event schedule Nov 3-6 Redmond, Seattle MVP Summit 2014 TBD Event schedule Nov 4-7 Redmond, Seattle Pass Summit 2014 TBD Event schedule

 

Will you be attending any of these events? If so, Luan extends the invite to chat SQL Server over a coffee! Reach out to him at moreno@pythian.com or follow him on Twitter at @luansql.

Categories: DBA Blogs

In-Memory OLTP – ORDER BY Behavior Clause Using Range Index

Thu, 2014-09-25 14:43

Some clients are beginning to thinking about the possibilities of migrating some disk tables to In-Memory tables — this process is not so simple. Migrating tables requires a new mindset as some things changed in the SQLOS and SQL Serve architecture.

It is extremely important to know about the differences, the scenarios that can be applied, and the non-supported aspects. First, take a look at the official link for the In-Memory Tables documentation – http://msdn.microsoft.com/en-us/library/dn133186%28v=sql.120%29.aspx

One of the things that we need to know about in In-Memory tables, is that this feature comes with two new types of indexes known as HASH INDEX and RANGE INDEX.

1. Hash Index – The Hash Index is perfect to use for equality purposes. For example, to search for a specific number and character.

hash_index
Figure 1 – Hash Index Structure

 

2. Range Index – Basically the Range Index is perfect for range purposes, So for example search for a range of values and normally is more applied when used with date ranges – DATE, DATETIME and DATETIME2.

range_index
Figure 2 – Range Index Structure

When we realize the range index creation, a very important aspect is the ORDER of the data sort (ASC or DESC). Creating In-Disk Tables (Conventional Tables) the ORDER BY is not normally a big point of concern because the data pages are double linked — this difference becomes significant when you have more than one column in two different directions.

Another interesting aspect of the ORDER BY clause in In-Disk tables is the ORDERING creation and Parallelism usage, and you can check this interesting behavior here sqlmag.com/t-sql/descending-indexes

Now, let’s analyze the ORDER BY clause in In-Memory tables using the RANGE INDEX and check the behavior and the differences when using the ASC and DESC order in queries.

 

USE HktDB

go

sp_help ‘inmem_DadosAtendimentoClientes’

table_info
Figure 3 – Range Index = idxNCL_DataCadastro

 

Creating an In-Memory table with a NONCLUSTERED Range Index in Ascending Order.

 

CREATE TABLE [dbo].[inmem_DadosAtendimentoClientes]

(

INDEX [idxNCL_DataCadastro] NONCLUSTERED

(

[DataCadastro] ASC

)

)

 

Grabbing information’s about RANGE INDEX

 

SELECT SIS.name AS IndexName,

SIS.type_desc AS IndexType,

XIS.scans_started,

XIS.rows_returned,

XIS.rows_touched

FROM sys.dm_db_xtp_index_stats AS XIS

INNER JOIN sys.indexes AS SIS

ON XIS.object_id = SIS.object_id

AND XIS.index_id = SIS.index_id

WHERE XIS.object_id = 50099219

AND  SIS.type_desc = ‘NONCLUSTERED’

index_info
Figure 4 – Range Index Information

 

Execution 1 – Ordering using the ORDER BY ASC

SELECT *

FROM inmem_DadosAtendimentoClientes

WHERE DataCadastro BETWEEN ‘2005-02-15 18:58:48.000? AND ‘2005-02-25 18:58:48.000?

ORDER BY DataCadastro ASC

plan_1

 

Execution 2 – Ordering using the ORDER BY DESC

SELECT *

FROM inmem_DadosAtendimentoClientes

WHERE DataCadastro BETWEEN ‘2005-02-15 18:58:48.000? AND ‘2005-02-25 18:58:48.000?

ORDER BY DataCadastro DESC

plan_2

 

sort_operation

Sort of 80% in this query, Why ?

 

Analyzing the XML of the Execution Plan…

<OrderBy>

  <OrderByColumn Ascending=”false”>

<ColumnReference Database=”[HktDB]” Schema=”[dbo]” Table=”[inmem_DadosAtendimentoClientes]” Column=”DataCadastro” />

</OrderByColumn>

</OrderBy>

 

What happened?

When we execute a query with the ORDER BY clause in Range Index column, we need to verify the order that was created – ASC or DESC. This happened because is this case I created the column ‘DataCadastro‘ with ASC order, this way the data is ordered in the ascending way and not in descending way, and talking about In-Memory tables the order MATTERS a lot, You just can benefit of the ORDER if the order that you searched is the same that the order that you created the tables, this happens because the data is stored in another way in-memory, this is a BY DESIGN consideration.

MSDN – “Nonclustered indexes (not hash indexes) support everything that hash indexes supports plus seek operations on inequality predicates such as greater than or less than, as well as sort order. Rows can be retrieved according to the order specified with index creation. If the sort order of the index matches the sort order required for a particular query, for example if the index key matches the ORDER BY clause, there is no need to sort the rows as part of query execution. Memory-optimized nonclustered indexes are unidirectional; they do not support retrieving rows in a sort order that is the reverse of the sort order of the index. For example, for an index specified as (c1 ASC), it is not possible to scan the index in reverse order, as (c1 DESC).”

 

Recommendation

Always realize the creation of the RANGE Index in the correct ordination that you want, in the most of times the most common ORDER pattern is the DESC, because normally you want to visualize and search the most recent data of your application or search for last transaction that you had in an specific date. You should always be careful about this because if you want to change the ORDER BY is necessary to DROP and CREATE again the table, In-Memory tables don’t enable the ALTER clause option.

You don’t want to see this in your environment, ever!

 

plan_comparison

 

 

Categories: DBA Blogs

Is X a Big Data product?

Wed, 2014-09-24 18:40

Virtually everyone in data space today claims that they are a Big Data vendor and that their products are Big Data products. Of course — if you are not in Big Data then you are legacy. So how do you know whether a product is a Big Data product?

While there might not be fully objective criteria (and mainly because Big Data definition is still in the air and people interpret it as they see fit for their purpose), I think I can provide one good suggestion on how to determine when a certain product is NOT a Big Data product. Of course, it will depend on the definition of Big Data that you believe in.

I believe that Big Data is mostly about being “affordable at scale“, quoting Jeff Needham, my good friend and fellow member of OakTable Network. In practice, that means commodity software, commodity hardware and commodity operations of the solution. I won’t define the thresholds of scale in terabytes or levels of complexity and etc but I can provide some guidelines.

Talking about commodity hardware, it’s generally based on x86 architecture (though, some say ARM is emerging but it’s been emerging way too long for my liking) with some reasonably priced components. That would typically be dual socket systems with up to few hundred GB of RAM and maybe a dozen disks or some SSDs and cost effective networking. If we narrow down to Hadoop-like architectures then a cluster node would typically cost between $4,000 and $10,000. Anything significantly above that is probably overpriced or overspec’ed.

OK. Now that we are good with hardware let’s look at software. Obviously, open-source software without any commercial support qualifies for commodity and being affordable. If you are Facebook-scale (or getting relatively close), your commercial support can be you own large scale, capable engineering team. Otherwise, you will most likely have commercial support. Back to Hadoop world, you should expect to pay for commercially supported Hadoop distribution (whoever it is out of three leading distributions — Cloudera, Hortonworks or MapR) the same order of magnitude as for the hardware itself. Annually, it would be a fraction of hardware cost or over three years it would be about the cost of hardware purchase or slightly above depending on the level of support and platform features. You get an idea. Non-open-source products licensed on similar pricing levels are Big Data products too — you don’t have to be open-source to call your technology Big Data.

Let’s take an example of a supposedly Big Data product. If a product has “Big Data” in the name, it surely must be a Big Data product. Eh?

I love quite a few Oracle products so why don’t I look at their line up… Big Data Appliance is a prebuilt Hadoop system or Hadoop appliance with 18 powerful data nodes per rack and list price tag of $525K per rack. That gets you to almost $30K per data node which is quite high and you would likely not build your own clusters like that. Add to that about $100K per year of support and maintenance for systems and OS (you can check pricing in the public engineered system price list). Big Data Appliance does include commercially supported Cloudera distribution so it might not be that terrible pricing-wise. If you have experience buying Oracle products you also know that customers don’t pay list prices. Thus, I can accept that Big Data Appliance can actually be called a Big Data product… just.

Now let’s looks at another product — Big Data SQL. It has been announced but hasn’t quite been released just yet (or did I miss it?). Awesome product, by the way. Great way to push some of data-intensive SQL processing from Oracle Database down to Hadoop. Now, it’s probably not widely known (since it wasn’t really publicly released and sold yet) that Big Data SQL is licensed per disk spindle and it’s $4,000 per spindle as list-price. Add to that typical 22% of annual software support and maintenance from Oracle. If I were to license Big Data SQL for a 100 nodes Hadoop cluster with 12 disks per node, it would cost me almost $5M based on list-price. Don’t forget to add 22% annually. This is order of magnitude more than I would spend on the hardware building such cluster. But wait, it looks like Big Data SQL is only working with Big Data Appliance. Even in this case, the cost of Big Data SQL per single rack appliance is $864K + 22% annually and that’s just one additional tool for your Big Data platform.

Based on what I know about Big Data SQL (and assuming it works as advertised when released), I love it — push code to data, scalable massive parallel processing, leveraging great features from Exadata Storage software. Great job to the folks who developed this product. Unfortunately, I cannot call it a Big Data product — it’s not affordable at scale.

So when you look at other vendors calling their product Big Data — do this costing assessment and if it doesn’t come as affordable at scale then it’s not a Big Data product. And feel free to share your assessments for the rest of us here. I’m sure not everyone will share my line of thinking here either. Fire way.

Categories: DBA Blogs

Oracle OpenWorld 2014 – Bloggers Meetup

Fri, 2014-09-19 15:35

Oracle OpenWorld Bloggers Meetup Guess what? You all know that it’s coming, when it’s coming and where… That’s right! The Annual Oracle Bloggers Meetup, one of your top favourite events of OpenWorld, is happening at usual place and time.

What: Oracle Bloggers Meetup 2014

When: Wed, 1-Oct-2014, 5:30pm

Where: Main Dining Room, Jillian’s Billiards @ Metreon, 101 Fourth Street, San Francisco, CA 94103 (street view). Please comment with “COUNT ME IN” if coming — we need to know the attendance numbers.


Traditionally, Oracle Technology Network and Pythian sponsor the venue and drinks. We will also have some cool things happening and a few prizes.

In the age of Big Data and Internet of Things, our mingling activity this year will be virtual — using an app we wrote specifically for this event, so bring your iStuff and Androids to participate and win. Hope this will work! :)

As usual, vintage t-shirts, ties, or bandanas from previous meetups will make you look cool — feel free to wear them.

For those of you who don’t know the history: The Bloggers Meetup during Oracle OpenWorld was started by Mark Rittman and continued by Eddie Awad, and then I picked up the flag in 2009 (gosh…  6 years already?) The meetups have been a great success for making new friends and catching up with old, so let’s keep them this way! To give you an idea, here are the photos from the OOW08 Bloggers Meetup (courtesy of Eddie Awad) and OOW09 meetup blog post update from myself, and a super cool video by a good blogging friend, Bjorn Roest from OOW13.

While the initial meetings were mostly targeted to Oracle database folks, guys and gals from many Oracle technologies — Oracle database, MySQL, Apps, Sun technologies, Java and more join in the fun. All bloggers are welcome. We estimate to gather around 150 bloggers.

If you are planning to attend, please comment here with the phrase “COUNT ME IN”. This will help us ensure we have the attendance numbers right. Please provide your blog URL with your comment — it’s a Bloggers Meetup after all! Make sure you comment here if you are attending so that we have enough room, food, and (most importantly) drinks.

Of course, do not forget to blog and tweet about this year’s bloggers meetup. See you there!

Categories: DBA Blogs