DBA Blogs

[oracle@localhost dbs]$ pwd
/home/oracle/app/oracle/product/11.2.0/dbhome_2/dbs
[oracle@localhost dbs]$ date
Mon May 20 08:17:46 PDT 2013
[oracle@localhost dbs]$ ls -ltr|tail -3
-rw-r----- 1 oracle oracle 1536 Oct  2  2010 orapworcl
-rw-rw---- 1 oracle oracle 1544 May 20 08:16 hc_orcl.dat
-rw-r----- 1 oracle oracle 3584 May 20 08:16 spfileorcl.ora

[oracle@localhost dbs]$ rman target /

Recovery Manager: Release 11.2.0.2.0 - Production on Mon May 20 08:17:53 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1229390655)

RMAN> backup current controlfile;

Starting backup at 20-MAY-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=69 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 20-MAY-13
channel ORA_DISK_1: finished piece 1 at 20-MAY-13
piece handle=/home/oracle/app/oracle/flash_recovery_area/ORCL/backupset/2013_05_20/o1_mf_ncnnf_TAG20130520T081800_8snhob80_.bkp tag=TAG20130520T081800 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 20-MAY-13

RMAN> quit


Recovery Manager complete.

[oracle@localhost dbs]$ ls -ltr|tail -3
-rw-rw---- 1 oracle oracle    1544 May 20 08:16 hc_orcl.dat
-rw-r----- 1 oracle oracle    3584 May 20 08:16 spfileorcl.ora
-rw-rw---- 1 oracle oracle 9748480 May 20 08:18 snapcf_orcl.f
[oracle@localhost dbs]$ 

“A beginning is the time for taking the most delicate care that the balances are correct.”

It is spring. Time for planting new seeds. I started on a new job last week, and it seems that few of my friends and former colleagues are on their way to new adventures as well. I’m especially excited because I’m starting not just a new job – I will be working on a new product, far younger than Oracle and even MySQL. I am also making first tiny steps in the open-source community, something I’ve been looking to do for a while.

I’m itching to share lessons I’ve learned in my previous job, three challenging and rewarding years as a consultant. The time will arrive for those, but now is the time to share what I know about starting new jobs. Lessons that I need to recall, and that my friends who are also in the process of starting a new job may want to hear.

Say hello
I’m usually a very friendly person and after years of attending conferences I’m very comfortable talking to people I’ve never met before. But still, Cloudera has around 200 people in the bay area offices, which means that I had to say “Hello, I’m Gwen Shapira the new Solutions Architect, who are you?” around 200 times. This is not the most comfortable feeling in the world. Its important to go through the majority of the introductions in the first week or two, later on it becomes a bit more awkward. So in the first week it will certainly seem like you are doing nothing except meeting people, chatting a bit and franctically memorizing names and faces. This is perfectly OK.

Get comfortable being unproductive
The first week in a new job feels remarkably unproductive. This is normal. I’m getting to know people, processes, culture, about 20 new products and 40 new APIs. I have incredibly high expectations of myself, and naturaly I’m not as fast installing Hadoop cluster as I am installing RAC cluster. It takes me far longer to write Python code than it does to write SQL. My expectations create a lot of pressure, I internally yell at myself for taking an hour or so to load data into Hive when it “should” have taken 5 minutes. But of course, I don’t know how long it “should” take, I did it very few times before. I’m learning and while learning has its own pace, it is an investment and therefore productive.

Have lunch, share drinks
The best way to learn about culture is from people, and the best way to learn about products is from the developers who wrote them and are passionate about how they are used. Conversations at lunch time are better than tackling people in the corridor or interrupting them at their desk. Inviting people for drinks are also a great way to learn about a product. Going to someones cube and asking for an in-depth explanation of Hive architecture can be seen as entitled and bothersome. Sending email to the internal Hive mailing list and saying “I’ll buy beer to anyone who can explain Hive architecture to me” will result in a fun evening.

If its not overwhelming, you may be in the wrong job
I’m overwhelmed right now. So many new things to learn. First there are the Hadoop ecosystem products, I know some but far from all of them, and I feel that I need to learn everything in days. Then there is programming. I can code, but I’m not and never have been a proficient programmer. My colleagues are sending out patches left and right. It also seems like everyone around me is a machine learning expert. When did they learn all this? I feel like I will never catch up.

And that is exactly how I like it.

Make as many mistakes as possible
You can learn faster by doing, and you can do faster if you are not afraid of failing and making mistakes. Mistakes are more understandable and forgivable when you are new. I suggest using this window of opportunity and accelerate your learning by trying to do as much as possible. When you make a mistake smile and say “Sorry about that. I’m still new. Now I know what I should and shouldn’t do”

Take notes
When you are new a lot of things will look stupid. Sometimes just because they are very different from the way you are used to things in a previous job. Don’t give in to the temptation to criticise everything, because you will look like a whiner. No one likes whiner. But take note of them, because you will get used to them soon and never see things with “beginner mind” again. In few month take a look at your list, if things still look stupid, it will be time to take on a project or two to fix them.

Contribute
I may be new at this specific job, but I still have a lot to contribute. I try hard to look for opportunities and I keep finding out that I’m more useful than I thought. I participate in discussions in internal mailing lists, I make suggestions, I help colleagues solve problems. I participate in interviews and file tickets when our products don’t work as expected. I don’t wait to be handed work or to be sent to a customer, I look for places where I can be of use.

I don’t change jobs often. So its quite possible that I don’t know everything there is to know about starting a new job. If you have tips and suggestions to share with me and my readers, please comment!

The red carpet has been laid down at this Log Buffer Edition, and you can witness and cheer the cat-walking blog posts from Oracle, SQL Server and MySQL. Every one of them is chic, elegant, sensual in its own right. Enjoy.
Oracle:

Create colored heat maps in SQL*Plus with Kyle Hailey.

Here’s a quick and dirty script to create a procedure (in the SYS schema – so be careful) to check the Hakan Factor for an object.

Connor has a good post about default null for collection parameter.

This is yet another blogpost on Oracle’s direct path read feature which was introduced for non-parallel query processes in Oracle version 11.

Owen Allen has seen some questions about provisioning Oracle Solaris 11. They boil down to this.

SQL Server:

Shashank Srivastava tells us as how to Change the SQL Server Instance Name after Renaming the Windows Host.

Daniel Calbimonte shares as how to synchronize two SSAS Servers.

Data Architecture underpins just about everything we do in IT.  Without a clear understanding of how data is structured, there is no reliable way to derive meaning from it.

Orlando Colamatteo is login-less in Seattle.

Lets get started testing database with tSQLt with Robert Sheldon.

MySQL:

After a lot of fuzz, Anders Karlsson is now releasing MyQuery version 3.5.1.

Nothing like reestablishing a tradition and Dave Stokes is doing just that for MySQL.

Mare Alff is spreading the word about the performance schema.

Slava Akhmechet talks about secondary indexes, batched inserts performance improvements, soft durability mode.

It is a central part of the MySQL philosophy to try and help you as much as you can. There are many occasions when it could tell you that what you are asking for is utterly stupid or give you a bad execution plan because “you asked for it”.

Sometime back I was involved in a project which was to collect information for the servers we are supporting – creating inventory of servers. Logging into each server and collecting information is bit tedious when you have hundreds of server in your environment.  I have spent sometime and created a script that does this work for me, however, I still need to connect to the server – but using SSMS/Query Window and execute the script. And then the script will provide me the details I needed.

This inventory should have details like below for cluster and stand alone instances:

Server Name

OS Name

OS Edition

OS Patch Level

SQL Server IP

Is Clustered

Node1_Name

Node1_IP

Node2_Name

Node2_IP

SQL Server Edition

SQL Server Patch Level

Server Time Zone

SQL Server Version

SQL Server Platform

Processor Core

Physical Memory

Service Account Name

Domain

Looks good ?  Below is the version 1 of this script.

/*
IP address portion using : http://www.sqlservercentral.com/Forums/Topic150196-8-1.aspx
REMOVE sp_configure parameters if you are executing this script on SQL Server 2000

Created By : Hemantgiri S. Goswami | http://www.sql-server-citation.com
Date : 24th March 2013
Version : 1.0

Tested ON:
Windows Server: 2003, 2008, 2008 R2
SQL Server:2000, 2005, 2008, 2008 R2, 2012

*/
sp_configure 'show advanced options',1
RECONFIGURE WITH OVERRIDE
GO
sp_configure 'xp_cmdshell',1
RECONFIGURE WITH OVERRIDE
GO
DECLARE @TimeZone NVARCHAR(100)
 ,@ProductVersion SYSNAME
 ,@PlatForm SYSNAME
 ,@Windows_Version SYSNAME
 ,@Processors SYSNAME
 ,@PhysicalMemory SYSNAME
 ,@ServiceAccount SYSNAME
 ,@IPAddress SYSNAME
 ,@DOMAIN SYSNAME
 ,@MachineType SYSNAME
 ,@SQLServerIP VARCHAR(255)
 ,@CMD VARCHAR(100)
 ,@Node1 VARCHAR(100)
 ,@Node2 VARCHAR(100)
 ,@Node1IP VARCHAR(100)
 ,@Node2IP VARCHAR(100)
 ,@OSEdition VARCHAR(100)
 ,@OSVersion VARCHAR(100)
 ,@OSName VARCHAR(100)
 ,@OSPatchLevel VARCHAR(100)

CREATE TABLE #TempTable
 (
 [Index] VARCHAR(2000),
 [Name] VARCHAR(2000),
 [Internal_Value] VARCHAR(2000),
 [Character_Value] VARCHAR(2000)
 ) ;

INSERT INTO #TempTable
EXEC xp_msver;

-- Replace @Value_Name to N'TimeZoneKeyName' when running on Windows 2008
EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\TimeZoneInformation',
 @value_name = N'StandardName',
 @value = @TimeZone output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Services\MSSQLServer',
 @value_name = N'ObjectName',
 @value = @ServiceAccount output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\ProductOptions',
 @value_name = N'ProductType',
 @value = @MachineType output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Services\Tcpip\Parameters',
 @value_name = N'Domain',
 @value = @DOMAIN output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'CLUSTER\NODES\1',
 @value_name = N'NodeName',
 @value = @Node1 output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'CLUSTER\NODES\2',
 @value_name = N'NodeName',
 @value = @Node2 output

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SOFTWARE\Microsoft\Windows NT\CurrentVersion',
 @value_name = N'ProductName',
 @value = @OSName output

create table #OSEdition (VALUe varchar(255),OSEdition varchar(255), data varchar(100))
insert into #OSEdition
EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SYSTEM\CurrentControlSet\Control\ProductOptions',
 @value_name = N'ProductSuite'
SET @OSEdition = (SELECT TOP 1 OSedition FROM #OsEdition)

EXEC master.dbo.xp_regread
 @rootkey = N'HKEY_LOCAL_MACHINE',
 @key = N'SOFTWARE\Microsoft\Windows NT\CurrentVersion',
 @value_name = N'CSDVersion',
 @value = @OSPatchLevel output

set @cmd = 'ping ' + @Node1
create table #Node1IP (grabfield varchar(255))
insert into #Node1IP exec master.dbo.xp_cmdshell @cmd
set @cmd = 'ping ' + @Node2
create table #Node2IP (grabfield varchar(255))
insert into #Node2IP exec master.dbo.xp_cmdshell @cmd

set @cmd = 'ping ' + @@servername
create table #SQLServerIP (grabfield varchar(255))
insert into #SQLServerIP exec master.dbo.xp_cmdshell @cmd

SET @SQLServerIP=(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #SQLServerIP where left(grabfield,7) = 'Pinging'
 )
SET @Node1IP =(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #Node1IP where left(grabfield,7) = 'Pinging'
 )

SET @Node2IP =(
 SELECT substring(grabfield, charindex('[',grabfield)+1, charindex(']',grabfield)-charindex('[',grabfield)-1)
 from #Node2IP where left(grabfield,7) = 'Pinging'
 )

SET @ProductVersion = (SELECT Character_Value from #TempTable where [INDEX]=2)
SET @Platform = (SELECT Character_Value from #TempTable where [INDEX]=4)
SET @Windows_Version= (SELECT Character_Value from #TempTable where [INDEX]=15)
SET @Processors = (SELECT Character_Value from #TempTable where [INDEX]=16)
SET @PhysicalMemory = (SELECT Character_Value from #TempTable where [INDEX]=19)
SELECT
 ServerName = @@SERVERNAME
 ,OSName = @OSName
 ,OSEdition = @OSEdition
 ,OSPatchLevel = @OSPatchLevel
 ,SQLServerIP = @SQLServerIP
 ,IsClustered = SERVERPROPERTY('IsClustered')
 ,Node1_Name = @Node1
 ,Node1_IP = @Node1IP
 ,Node2_Name = @Node2
 ,Node2_IP = @Node2IP
 ,SQLServerEdition = SERVERPROPERTY('Edition')
 ,SQLServerLevel = SERVERPROPERTY('ProductLevel')
 ,ServerTimeZone = @TimeZone
 ,SQLServerVersion = @ProductVersion
 ,SQLServerPlatform = @PlatForm
 ,ProcessorCore = @Processors
 ,PhysicalMemory = @PhysicalMemory
 ,ServiceAccountName = @ServiceAccount
 ,WKS_Server = @MachineType
 ,Domain = @DOMAIN

GO
DROP TABLE #Node1IP
DROP TABLE #NODE2IP
DROP TABLE #SQLServerIP
DROP TABLE #TempTable
DROP TABLE #OSEdition
GO
sp_configure 'xp_cmdshell',0
RECONFIGURE WITH OVERRIDE
GO
sp_configure 'show advanced options',0
RECONFIGURE WITH OVERRIDE
GO

Please do post back your feed back for this script, I will try my best to update and post back new version for this script.

- Hemantgiri S. Goswami (Cross posting from http://www.sql-server-citation.com/)

photo credit: Skimaniac via photopin cc

I’ve seen some very good information posted in this feed which combines blog postings from many different Oracle performance experts who are part of what is called the “Oak Table”

http://www.oaktable.net/feed/blog-rss.xml

I’ve been using Internet Explorer to keep track of new posts in its “Feeds” section of the Favorites.  Here is how to add the Oak Table blog feed to Internet Explorer:

Go to the URL listed above and click on “Subscribe to this feed”

Click on Subscribe button

Success!  Now click on Favorites and then Feeds

For any feed in your list if you see the feed name in a darker font it means there is a new post.  So, as I have time, I’ll go to my feeds and see which of the ones I’ve subscribed to have new posts.  If you are looking for performance tuning information I highly recommend the Oak Table feed.

- Bobby

When I started my work in IT, I used to be in a very small shop, and even though we had people in several places in the same state, everything used to be very centralized and from 9 to 5, and because we were basically only 2 people , our action plan used to be a talk over the lunch table and that would be it, we would go ahead and execute it after 5 PM, and I won’t lie sometimes before 5 :) .

Over the years I have understood that even if you are a 2 guy shop or a team of 15 separated by oceans and being miles apart, communication is the most important thing to have on your team, and one of the means of communication is having an action plan in place for any major/medium change you do in your organization. First this will generate discussions amongst your teammates and it will reduce the possibility of errors when you are faced with time and pressure constraints when implementing it.

This might sometimes feel like a mundane and boring task, as it will take an effort to come up with it and it will take time to verify it, but when game day comes along you will see the great benefit of having an action plan.

Another great benefit of having an action plan is that you also have a road map if you need to rollback your change, and that is also critical, because normally any major change or rollback is not done only by one person, take for example a change that takes about 7 or 8 hours to be done, and at the end when the UAT (User Application Testing) is done, 1 or 2 more hours ,the application team decides that a rollback is needed, you are probably not in a good state of mind to do the rollback after 8 hours of continuous work, if you have an action plan, one of your teammates can step in and you can have a rest, even if it is to go to the kitchen and have a sandwich and a coke and forget 10 minutes about that pressure.

As with life and with us being human, having an action plan doesn’t mean that everything will go smoothly or you won’t have an error in there, but believe me, it will reduce in a big way the possibility of an error if you execute it by memory or by doing one yourself without revision.

I do hope that you already have an action plan as part of your major/medium changes, but if you don’t, it is time to get FIT-ACER, here is an example of one (Kudos to Cesar Sanchez as it is his Action Plan Template), use it and modify it to your needs, it is a good start.

Action Plan Template

In my forum discussion about free buffer waits I came across a term that I didn’t understand: “inode lock contention”.  I’m pretty sure I had seen this same term years ago on one of Steve Adams’ pages on IO.  But, I didn’t really understand what the term meant and so it was hard to understand whether this was something I was seeing on our production system that was experiencing “free buffer waits”.

First I had to figure out what an inode was.  I knew that it had something to do with the way Unix filesystems work but reading this article really helped clear up what inodes are at least on HP-UX.  Inodes are small chunks of bytes that are used to define a Unix filesystem.  On HP-UX’s VxFS filesystems a type 1 inode can point to up to 10 extents of one or more contiguous 8K blocks on a large filesystem.  The filesystem I’ve been testing on appears to have 32 meg extents if I’m reading this output from lvdisplay correctly:

LV Size (Mbytes)            1472000
Current LE                  46000

Total size of 1,472,000 meg divided by 46,000 logical extents = 32 meg per extent.

Since the inode can point to 1 to 10 extents it could point to between 32 and 320 meg.

My test case had 15 tables that were more than 1 gigabytes each.  It seems like each table should span multiple inodes so even if there is locking at the inode level it looks like it won’t lock the entire table at once.  Still, it seems unlikely to me that every time a table is updated that reads from all the other parts of the table pointed to by the same inode are really blocked by an inode lock.  Yet that is what this document suggests:

“During a read() system call, VxFS will acquire the inode lock in shared mode, allowing many processes to read a single file concurrently without lock contention. However, when a write() system call is made, VxFS will attempt to acquire the lock in exclusive mode. The exclusive lock allows only one write per file to be in progress at a time, and also blocks other processes reading the file. These locks on the VxFS inode can cause serious performance problems when there are one or more file writers and multiple file readers.”

It uses the term “file” but I assume if you have a large file that has multiple inodes it means it will lock just the pieces associated with the one inode that points to the blocks that are being written.  The article goes on to explain how you can use the “cio” option to enable concurrent IO and eliminate this inode contention preventing writers from blocking readers.  But, I’ve been testing with just the direct IO options and not the cio option and seeing great results.  So, would I see even better improvement with concurrent io?

I didn’t want to mess with our current filesystem mount options since testing had proven them to be so effective but I found that in glance, a performance monitoring tool like top, you have an option to display inode waits.  So, I took a test that was running with direct IO and had 15 merge statements loading data into the same empty table at once and ran glance to see if there were any inode waits.  There were not:

So, I don’t know if I can depend on this statistic in glance or not.  It appears that the direct IO mount options are all we need:

mincache=direct,convosync=direct

filesystemio_options=DIRECTIO

There may be some case within Oracle 11.2.03 on HP-UX 11.31 where you can be hampered by inode lock contention despite having direct IO enabled but my tests have not confirmed it and I’ve banged pretty hard on my test system with a couple of different types of tests.

- Bobby

This week again, the Log Buffer Edition covers the exciting and ever-evolving galaxies of Oracle, MySQL and SQL Server.

Oracle:

When does an Oracle process know it’s on Exadata?

MobaXterm 6.3 has been released. You can find the download and changelog in the usual place.

A new major release (version 3.0) of Randolf’s XPLAN_ASH tool is available for download.

Important !! Clustering Factor Calculation Improvement

Grouping Data Sets by Week Number of the Month

SQL Server:

Jason Crider asks to put some Windows update in his basket.

Data cleansing can be done via SSIS as well as Data Quality Services (DQS) and Master Data Services (MDS).  The lines are a bit blurred when talking about data cleansing using SSIS, DQS and MDS.

How To: Migrate Encrypted Procedures To Windows Azure SQL Database (WASD).

It’s a best practice to keep your index keys as narrow as possible, and SQL Server enforces a maximum key length of 900 bytes on most “normal” clustered and non-clustered indexes.

Webcast Presentation Materials: Introduction to SQL Server Security

MySQL:

This is the start of the heavy travel season for the MySQL Community Team. So if you are attending PHPTek, Drupalcon, Texas LinuxFest, SELF, Redhat Summit, or Lonestar PHP please make sure to say ‘hi’ while you are in Chicago, Portland, Austin, Charlotte, Boston, or Addison.

Your databases in a glance with Glimpsee

Continuent Tungsten 2.0 is a major step forward that includes key feature improvements to help you manage very large datasets on MySQL.

How to create a new (or repair a broken) GTID based slave with Percona XtraBackup

Disk I/O is frequently the performance bottleneck with relational databases. With AWS recently releasing 4,000 PIOPs EBS volumes, Laine Campbell wanted to do some benchmarking with pgbench and PostgreSQL 9.2.

I just read this post about a new patch that will allow you to affect the way the optimizer calculates clustering factor and hence how likely it is to choose a particular index for a query plan.

http://richardfoote.wordpress.com/2013/05/08/important-clustering-factor-calculation-improvement-fix-you/

Pretty cool.  I haven’t tried it, but it looks promising.

- Bobby

This is a good web site with Oracle internals information:

http://www.ixora.com.au/notes/

I’ve benefited from this one for years.

- Bobby

Mothers day is coming up fast. May 12th 2013 is just around the corner and If you have not already done so, make sure you go out and get mom a special gift as soon as possible. If you need a hand picking something great, take a look at this mothers day gift guide by the HuffingtonPost

Many this year like they have in previous years shopped online and bought mom that special gift from an online retailer, a simple, easy and fun process. But is it always? Unfortunately for many, the experience did not live up to expectation due to site performance and both customers and merchants end up losing.

Is this a big deal? We put together this infograph on mothers day shopping. Take a look.