Skip navigation.

Feed aggregator

Solid Conference San Francisco 2014: Complete Video Compilation

Surachart Opun - Tue, 2014-07-29 08:17
Solid Conference focused on the intersection of software and hardware. It's great community with Software and Hardware. Audiences will be able to learn new idea to combine software and hardware. It gathered idea from engineers, researchers, roboticists, artists, founders of startups, and innovators.
Oreilly launched HD videos (Solid Conference San Francisco 2014: Complete Video Compilation Experience the revolution at the intersection of hardware and software—and imagine the future) for this conference. Video files might huge for download. It will spend much time. Please Use some download manager programs for help.
After watched, I excited to learn some things new with it (Run times: 36 hours 8 minutes): machines, devices, components and etc.

Written By: Surachart Opun http://surachartopun.com
Categories: DBA Blogs

The Nature of Digital Disruption

WebCenter Team - Tue, 2014-07-29 08:10
by Dave Gray, Entrepreneur, Author & Consultant

Digital Disruption – The change that occurs when new digital technologies and business models affect the value proposition of existing goods and services or bring to market an entirely new innovation.

Why is the shift to digital so disruptive?

As a global society, we are currently in the process of digitizing everything. We are wrapping our physical world with a digital counterpart, a world of information, which parallels and reflects our own. We want to know everything we can think of about everything we can think of.

This whirl of digital information changes the playing field for businesses, because digital information does not abide by any of the rules that we are used to in business. 

In a digital world, products and services have no physical substance. There are no distribution costs. A single prototype can generate an infinite number of copies at no cost. And since the products and services are so different, the environment around them becomes unstable; as the digital layer interacts with the physical layer, everything in the ecosystem is up for grabs. Suddenly new products become possible and established ones become obsolete overnight.

Science-fiction writer Arthur C. Clarke once said that “Any sufficiently advanced technology is indistinguishable from magic.”

In the business world today, you are competing with sorcerers. You need to learn magic.

Let’s take the music industry as an example of how technology changes the playing field. Music used to be very expensive to record and distribute. Every time a new technology comes along, the music industry has had to adjust.

The graph on the left shows units sold in the music industry, by media, since 1973. See the overlapping curves? Each technology has a lifecycle – early in the lifecycle sales are low, but they rise as more people adopt the technology. When a new technology comes along the older technologies suffer. But not to worry, people still need their music, right? Typically the lifecycle curve for “units sold” closely echoes the revenue curve.

But when the product becomes purely digital – when it enters the realm of magic – the cost of making and distributing the product plummets to nearly zero. This means more people can produce and distribute music, more cheaply and easily. More music becomes available to the public and purchases skyrocket – but the price per unit drops precipitously.

Take a look at the two graphs below. The left chart is units sold and the right one is revenue. Note how digital downloads (units sold) have skyrocketed, while the revenue curve is the smallest in years. 

The core issue is that even though unit sales rise rapidly, the price per unit drops so much faster that the revenue from sales fails to make up the difference. The industrial-age company, which has built its business model on the high costs of producing and distributing physical products, now has a high-cost infrastructure which is suddenly obsolete. What was once an asset is now a critical liability. This opens the entire industry to new players who can offer services to this new world at a dramatically lower cost.

The product is now digital. So the album, which you once charged $15 for, now retails for about $10. Ouch. You just lost a third of your revenue. But it gets worse. In the old days you sold music by the album, because the cost to make and distribute single songs on CD kept the cost of singles relatively high. So people would buy albums which contained a lot of songs, it now appears, that they didn’t really want. The chart below compares the typical mix between album and single sales on CD vs. downloads. The product mix has flipped completely, from most people buying albums for $15, to most people buying songs for $1.

So the revenue per unit drops once again. Even with some people buying albums, the average revenue per unit is about $1.50. That means your entire industry has lost about 90% of your revenue, almost overnight. 

In the world of manufacturing we talk about efficiency and productivity. You look to efficiency to decrease your costs and productivity to increase your revenue. In between you seek to make a profit. But you can’t streamline yourself to profits when the world is changing around you so profoundly. You need different strategies, different tactics.

The digital revolution is the biggest shift in the music industry since the 1920’s, when phonograph records replaced sheet music as the industry’s profit center.

What’s going on here? First, the means of making and distributing the product change. Suddenly the costs are so low that thousands of new competitors enter the market. Every artist can now compete with you from his or her garage, bringing new meaning to the word “garage band.”

But as if that weren’t bad enough, this also changes the things that people buy and the way they buy them. It’s a cascading effect.

So who wins and how do they win? Let’s look at Apple’s iTunes strategy. Apple looked at the entire industry as an ecosystem – people buy music and they play it on a device. If they like the experience they buy more music. In time they might buy another device, and so on, and so on. This is not a business process, it’s a business cycle.

Sony had everything that Apple had – in fact, much more. They had a powerful music-player brand, the Walkman, the established industry leader for portable music players. They had more engineers. They had a music division with 21 record labels. 

Sony’s divisions, which worked in their favor for efficiency and productivity, worked against them when it came to collaboration and innovation. The company was divided into separate operating units which competed with each other internally, making it difficult to collaborate on projects that spanned across multiple units. Sony was a classic industrial-age company, focused on productivity and efficiency.

What did Apple do that Sony didn’t? They focused on the system, not the product.

If you want to record your own music, Apple makes the software for that. If you want to sell your music, you can sell it on iTunes. If you want to play it, Apple makes the device. In case you hadn’t noticed, Apple had to look at the entire ecosystem of the record industry through a new, digital lens, including:

  1. Understand the digital infrastructure and how it changed the playing field.
  2. Relentless focus on user experience – simplicity, “just works” design, delight customers.
  3. Smart partnerships: Apple began by giving away the money: Record companies made 70 cents on every 99 cent purchase, with the rest split between artists and merchandising costs.
  4. Interoperability: Apple chose to support an open format that would work with any player, while Sony chose a proprietary format for their first digital media player.

In short: 

Think creatively. Understand, provide for, and support the entire ecosystem. Fill in the gaps when you can. Eliminate middlemen if you can – partner with them if you must. Partner with value providers (like artists and record companies that own large repositories of music). Be fearless about cannibalizing your own core business – if you’re not doing, it somebody else is.

The core difference is between an industrial, manufacturing-based model which focuses on efficiency and productivity – making more widgets more efficiently, and an information-based model which focuses on creativity and innovation. The industrial model thrives on successful planning and logistics, while the information model thrives on systems thinking, rapid learning and adaptation to a changing environment.

What can you do? As a company, you will need to innovate differently. That’s the subject of my next post, which we will discuss next week.  

In the meantime, you can hear more from Dave on Digital Disruption in our Digital Business Thought Leaders webcast "The Digital Experience: A Connected Company’s Sixth Sense". 

Create Windows Service for Oracle RAC

Pythian Group - Tue, 2014-07-29 08:08

It’s my first time on RAC system for Windows and I’m happy to learn something new to share.

I created a new service for database (restoredb) only to find out the ORACLE_HOME for the service is “c:\\oracle\\product\\10.2.0\\asm_1″

Any ideas as to what was wrong?

C:\\dba_pythian>set oracle_home=C:\\oracle\\product\\10.2.0\\db_1

C:\\dba_pythian>echo %ORACLE_HOME%
C:\\oracle\\product\\10.2.0\\db_1

C:\\dba_pythian>oradim -NEW -SID restoredb -STARTMODE manual
Instance created.

C:\\dba_pythian>env
 1 STOPPED agent11g1Agent                                    c:\\oracle\\app\\11.1.0\\agent11g
 2 STOPPED agent11g1AgentSNMPPeerEncapsulator                c:\\oracle\\app\\11.1.0\\agent11g\\bin\\encsvc.exe
 3 STOPPED agent11g1AgentSNMPPeerMasterAgent                 c:\\oracle\\app\\11.1.0\\agent11g\\bin\\agntsvc.exe
 4 RUNNING +ASM1                                             c:\\oracle\\product\\10.2.0\\asm_1
 5 RUNNING ClusterVolumeService                              C:\\oracle\\product\\10.2.0\\crs
 6 RUNNING CRS                                               C:\\oracle\\product\\10.2.0\\crs
 7 RUNNING CSS                                               C:\\oracle\\product\\10.2.0\\crs
 8 RUNNING EVM                                               C:\\oracle\\product\\10.2.0\\crs
 9 STOPPED JobSchedulerDWH1                                  c:\\oracle\\product\\10.2.0\\db_1
10 STOPPED JobSchedulerRMP1                                  c:\\oracle\\product\\10.2.0\\db_1
11 RUNNING OraASM10g_home1TNSListenerLISTENER_PRD-DB-10G-01  C:\\oracle\\product\\10.2.0\\asm_1
12 STOPPED OraDb10g_home1TNSListener                         c:\\oracle\\product\\10.2.0\\db_1
13 STOPPED ProcessManager                                    "C:\\oracle\\product\\10.2.0\\crs"
14 RUNNING DWH1                                              c:\\oracle\\product\\10.2.0\\db_1
15 RUNNING RMP1                                              c:\\oracle\\product\\10.2.0\\db_1
16 RUNNING agent12c1Agent                                    C:\\agent12c\\core\\12.1.0.4.0
17 RUNNING restoredb                                         c:\\oracle\\product\\10.2.0\\asm_1
18 STOPPED JobSchedulerrestoredb                             c:\\oracle\\product\\10.2.0\\asm_1

Check the PATH variable to find HOME for ASM is listed before DB.

C:\\dba_pythian>set
Path=C:\\oracle\\product\\10.2.0\\asm_1\\bin;C:\\oracle\\product\\10.2.0\\db_1\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS

Create database service specifying the fullpath to oradim from the DB HOME

C:\\dba_pythian>oradim -DELETE -SID restoredb
Instance deleted.

C:\dba_pythian>env
 1 STOPPED agent11g1Agent                                    c:\\oracle\\app\11.1.0\\agent11g
 2 STOPPED agent11g1AgentSNMPPeerEncapsulator                c:\\oracle\\app\11.1.0\\agent11g\\bin\\encsvc.exe
 3 STOPPED agent11g1AgentSNMPPeerMasterAgent                 c:\\oracle\\app\11.1.0\\agent11g\\bin\\agntsvc.exe
 4 RUNNING +ASM1                                             c:\\oracle\\product\\10.2.0\\asm_1
 5 RUNNING ClusterVolumeService                              C:\\oracle\\product\\10.2.0\\crs
 6 RUNNING CRS                                               C:\\oracle\\product\\10.2.0\\crs
 7 RUNNING CSS                                               C:\\oracle\\product\\10.2.0\\crs
 8 RUNNING EVM                                               C:\\oracle\\product\\10.2.0\\crs
 9 STOPPED JobSchedulerDWH1                                  c:\\oracle\\product\\10.2.0\\db_1
10 STOPPED JobSchedulerRMP1                                  c:\\oracle\\product\\10.2.0\\db_1
11 RUNNING OraASM10g_home1TNSListenerLISTENER_PRD-DB-10G-01  C:\\oracle\\product\\10.2.0\\asm_1
12 STOPPED OraDb10g_home1TNSListener                         c:\\oracle\\product\\10.2.0\\db_1
13 STOPPED ProcessManager                                    "C:\\oracle\\product\\10.2.0\\crs"
14 RUNNING DWH1                                              c:\\oracle\\product\\10.2.0\\db_1
15 RUNNING RMP1                                              c:\\oracle\\product\\10.2.0\\db_1
16 RUNNING agent12c1Agent                                    C:\\agent12c\\core\\12.1.0.4.0

C:\\dba_pythian>dir C:\\oracle\\product\\10.2.0\\db_1\\BIN\\orad*
 Volume in drive C has no label.
 Volume Serial Number is D4FE-B3A8

 Directory of C:\\oracle\\product\\10.2.0\\db_1\\BIN

07/08/2010  10:01 AM           121,344 oradbcfg10.dll
07/20/2010  05:20 PM             5,120 oradim.exe
07/20/2010  05:20 PM             3,072 oradmop10.dll
               3 File(s)        129,536 bytes
               0 Dir(s)  41,849,450,496 bytes free

C:\\dba_pythian>C:\\oracle\\product\\10.2.0\\db_1\\BIN\\oradim.exe -NEW -SID restoredb -STARTMODE manual
Instance created.

C:\\dba_pythian>env
 1 STOPPED agent11g1Agent                                    c:\\oracle\\app\\11.1.0\\agent11g
 2 STOPPED agent11g1AgentSNMPPeerEncapsulator                c:\\oracle\\app\\11.1.0\\agent11g\\bin\\encsvc.exe
 3 STOPPED agent11g1AgentSNMPPeerMasterAgent                 c:\\oracle\\app\\11.1.0\\agent11g\\bin\\agntsvc.exe
 4 RUNNING +ASM1                                             c:\\oracle\\product\\10.2.0\\asm_1
 5 RUNNING ClusterVolumeService                              C:\\oracle\\product\\10.2.0\\crs
 6 RUNNING CRS                                               C:\\oracle\\product\\10.2.0\\crs
 7 RUNNING CSS                                               C:\\oracle\\product\\10.2.0\\crs
 8 RUNNING EVM                                               C:\\oracle\\product\\10.2.0\\crs
 9 STOPPED JobSchedulerDWH1                                  c:\\oracle\\product\\10.2.0\\db_1
10 STOPPED JobSchedulerRMP1                                  c:\\oracle\\product\\10.2.0\\db_1
11 RUNNING OraASM10g_home1TNSListenerLISTENER_PRD-DB-10G-01  C:\\oracle\\product\\10.2.0\\asm_1
12 STOPPED OraDb10g_home1TNSListener                         c:\\oracle\\product\\10.2.0\\db_1
13 STOPPED ProcessManager                                    "C:\\oracle\\product\\10.2.0\\crs"
14 RUNNING DWH1                                              c:\\oracle\\product\\10.2.0\\db_1
15 RUNNING RMP1                                              c:\\oracle\\product\\10.2.0\\db_1
16 RUNNING agent12c1Agent                                    C:\\agent12c\\core\\12.1.0.4.0
17 RUNNING restoredb                                         c:\\oracle\\product\\10.2.0\\db_1
18 STOPPED JobSchedulerrestoredb                             c:\\oracle\\product\\10.2.0\\db_1

C:\\dba_pythian>
Categories: DBA Blogs

How SQL Server Browser Service Works

Pythian Group - Tue, 2014-07-29 08:07

Some of you may wonder the role SQL browser service plays in the SQL Server instance. In this blog post, I’ll provide an overview of the how SQL Server browser plays crucial role in connectivity and understand the internals of it by capturing the network monitor output during the connectivity with different scenario.

Here is an executive summary of the connectivity flow:   ExecutiveWorkflow

 

Here is another diagram to explain the SQL Server connectivity status for Named & Default instance under various scenarios:

 DefaultvsNamed

Network Monitor output for connectivity to Named instance when SQL Browser is running:

In the diagram below, we can see that an UDP request over 1434 was sent from a local machine (client) to SQL Server machine (server) and response came from server 1434 port over UDP to client port with list of instances and the port in which it is listening:

image003

 

Network Monitor output for connectivity to Named instance when SQL Browser is stopped/disabled:

 We can see that client sends 5 requests which ended up with no response from UDP 1434 of server. so connectivity will never be established to the named instance.

 image004

 

Network Monitor output for connectivity to Named instance with port number specified in connection string & SQL Browser is stopped/disabled:

 There is no call made to the server’s 1434 port over UDP instead connection is directly made to the TCP port specified in the connection string.

image005  Network Monitor output for connectivity to Default instance when SQL Browser running:

 We can see that no calls were made to server’s 1434 port over UDP in which SQL Server browser is listening.

 image006

 

Network Monitor output for connectivity to Default instance which is configured to listen on different port other than default 1433 when SQL Browser running:

 We can see that connectivity failed after multiple attempts because client assumes that default instance of SQL Server always listens on TCP port 1433.

You can refer the blog below to see some workarounds to handle this situation here:

http://blogs.msdn.com/b/dataaccesstechnologies/archive/2010/03/03/running-sql-server-default-instance-on-a-non-default-or-non-standard-tcp-port-tips-for-making-application-connectivity-work.aspx

image007 References:

SQL Server Browser Service - http://msdn.microsoft.com/en-us/library/ms181087.aspx

Ports used by SQL Server and Browser Service - http://msdn.microsoft.com/en-us/library/ms175483.aspx

SQL Server Resolution Protocol Specification - http://msdn.microsoft.com/en-us/library/cc219703(v=prot.10).aspx

Thanks for reading!

 

Categories: DBA Blogs

Oracle Database 12.1.0.2.0 – Turning OFF the In-Memory Database option

Marco Gralike - Tue, 2014-07-29 07:03
So how to turn it the option off/disabled…As a privileged database user: > Just don’t set the INMEMORY_SIZE parameter to a non zero value…(the default...

Read More

Beta1 of the UnifiedPush Server 1.0.0 released

Matthias Wessendorf - Tue, 2014-07-29 06:47

Today we are announcing the first beta release of our 1.0.0 version. After the big overhaul, including a brand new AdminUI with the last release this release contains several enhancements:

  • iOS8 interactive notification support
  • increased APNs payload (2k)
  • Pagination for analytics
  • improved callback for details on actual push delivery
  • optimisations and improvements

The complete list of included items are avialble on our JIRA instance.

iOS8 interactive notifications

Besides the work on the server, we have updated our Java and Node.js sender libraries to support the new iOS8 interactive notification message format.

If you curious about iOS8 notifications, Corinne Krych has a detailed blog post on it and how to use it with the AeroGear UnifiedPush Server.

Swift support for iOS

On the iOS client side Corinne Krych and Christos Vasilakis were also busy starting some Swift work: our iOS registration SDK supports swift on this branch. To give you an idea how it looks, here is some code:

func application(application: UIApplication!, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: NSData!) {
  // setup registration
  let registration = 
  AGDeviceRegistration(serverURL: NSURL(string: "<# URL of the running AeroGear UnifiedPush Server #>"))

    // attemp to register
    registration.registerWithClientInfo({ (clientInfo: AGClientDeviceInformation!) in
        // setup configuration
        clientInfo.deviceToken = deviceToken
        clientInfo.variantID = "<# Variant Id #>"
        clientInfo.variantSecret = "<# Variant Secret #>"

        // apply the token, to identify THIS device
        let currentDevice = UIDevice()

        // --optional config--
        // set some 'useful' hardware information params
        clientInfo.operatingSystem = currentDevice.systemName
        clientInfo.osVersion = currentDevice.systemVersion
        clientInfo.deviceType = currentDevice.model
        },

        success: {
            println("UnifiedPush Server registration succeeded")
        },
        failure: {(error: NSError!) in
            println("failed to register, error: \(error.description)")
        })
}
Demos

To get easily started using the UnifiedPush Server we have a bunch of demos, supporting various client platforms:

  • Android
  • Apache Cordova (with jQuery and Angular/Ionic)
  • iOS

The simple HelloWorld examples are located here. Some more advanced examples, including a Picketlink secured JAX-RS application, as well as a Fabric8 based Proxy, are available here.

For those of you who that are into Swift, there Swift branches for these demos as well:

Feedback

We hope you enjoy the bits and we do appreciate your feedback! Swing by on our mailing list! We are looking forward to hear from you!


SQL Saturday in Paris on 12 -13 September

Yann Neuhaus - Mon, 2014-07-28 22:35

As you certainly know SQL Saturday events are very popular in SQL Server world community. This is the second time the event takes place in Paris (France), but this time, we have a new format with pre-conferences on Wednesday and classic sessions on Saturday. During pre-conferences, we will talk about a particular subject for a whole day.

This time, I have the opportunity to participate twice by giving two sessions (in French) with the following program:

  • Friday: Inside the SQL Server storage and backups

If you are interested in how the SQL Server storage works and how to deal with corruption as well as backups, this session might be interesting for you.

Be careful: the pre-conferences on Wednesday are fee-paying sessions (but not that expensive). You can still register at this address.

  • Saturday: SQL Server AlwaysOn deep dive

SQL Server AlwaysOn is a new great high-availability and disaster recovery feature provided by Microsoft. You can come take a look at this session if you are concerned by questions like:

  • How to configure my Windows failover cluster and quorum in my situation?
  • What exactly is a read-only secondary replica?
  • What are the built-in tools provided by Microsoft to monitor and troubleshoot this infrastructure?

Good news: the sessions on Saturday are free!

Take a look at the agenda if you want to attend to other interesting sessions. I hope there will be many attendees! Smile

Query with new plan

Bobby Durrett's DBA Blog - Mon, 2014-07-28 18:24

I came up with a simple query that shows a running SQL executing a different plan than what it had in the past.  Here is the query:

-- show currently executing sqls that have history
-- but who have never run with the current plan
-- joins v$session to v$sql to get plan_hash_value of 
-- executing sql.
-- queries dba_hist_sqlstat for previously used values 
-- of plan_hash_value.
-- only reports queries that have an older plan that is 
-- different from the new one.

select
vs.sid,
vs.sql_id,
vs.last_call_et,
sq.plan_hash_value
from
v$session vs,
v$sql sq
where
vs.sql_id=sq.sql_id and
vs.SQL_CHILD_NUMBER=sq.child_number and
sq.plan_hash_value not in 
(select ss.plan_hash_value
from dba_hist_sqlstat ss
where 
ss.sql_id=sq.sql_id) and 
0 < 
(select count(ss.plan_hash_value)
from dba_hist_sqlstat ss
where 
ss.sql_id=sq.sql_id);

Example output:

       SID SQL_ID        LAST_CALL_ET PLAN_HASH_VALUE
---------- ------------- ------------ ---------------
       229 cq8bhsxbbf9k7            0      3467505462

This was a test query.  I ran it a bunch of times with an index and then dropped the index after creating an AWR snapshot.  The query executed with a different plan when I ran it without the index.  The same type of plan change could happen in production if an index were accidentally dropped.

I’m hoping to use this query to show production queries that have run in the past but whose current plan differs from any that they have used before.  Of course, a new plan doesn’t necessarily mean you have a problem but it might be helpful to recognize those plans that are new and that differ from the plans used in the past.

- Bobby

 

Categories: DBA Blogs

Impugn My Character Over Technical Points–But You Should Probably Be Correct When You Do So. Oracle 12c In-Memory Feature Snare? You Be The Judge ‘Cause Here’s The Proof.

Kevin Closson - Mon, 2014-07-28 18:04
Executive Summary

This blog post offers proof that you can trigger In-Memory Column Store feature usage with the default INMEMORY_* parameter settings. These parameters are documented as the approach to ensure In-Memory functionality is not used inadvertently–or at least they are documented as the “enabling” parameters.

During the development of this study, Oracle’s Product Manager in charge of the In-Memory feature has cited Bug #19308780 as it relates to my findings.

Index of Related Posts

This is part 4 in a series: Part I, Part II, Part III, Part IV.

Blog Updates

Please note, blog updates are listed at the end of the article.

What Really Matters?

This is a post about enabling versus using the Oracle Database 12c Release 12.1.0.2 In-Memory Column Store feature which is a part of the separately licensed Database In-Memory Option of 12c. While reading this please be mindful that in this situation all that really matters is what actions on your part effect the internal tables that track feature usage.

Make Software, Not Enemies–And Certainly Not War

There is a huge kerfuffle regarding the separately licensed In-Memory Column Store feature in Oracle Database 12c Release 12.1.0.2–specifically how the feature is enabled and what triggers  usage of the feature.

I pointed out a) the fact that the feature is enabled by default and b) the feature is easily accidentally used. I did that in Part I and Part II in my series on the matter.  In Part III I shared how the issue has lead to industry journalists quoting–and then removing–said quotes. I’ve endured an ungodly amount of shameful backlash even from some friends on the Oaktable Network list as they asserted I was making a mole hill out of something that was a total lark (that was a euphemistic way of saying they all but accused me of misleading my readers).  I even had friends suggesting this is a friendship-ending issue. Emotion and high-technology are watery-oil like in nature.

About the only thing that hasn’t happened is for anyone to apologize for being totally wrong in their blind-faith rooted feelings about this issue. What did he say? Please read on.

From the start I pointed out that the INMEMORY_QUERY feature is enabled by default–and that it is conceivable that someone could use it accidentally. The back lash from that was along the lines of how many parameters and what user actions are needed for that to be a reality.  Maria Colgan–who is Oracle’ s PM for the In-Memory Column Store feature–tweeted that I’m confusing people when announcing her blog post on the fact that In-Memory Column Store usage is controlled not by INMEMORY_QUERY but instead INMEMORY_SIZE. Allow me to add special emphasis to this point. In a blog post on oracle.com, Oracle’s PM for this Oracle database  feature explicitly states that INMEMORY_SIZE must be changed from the default to use the feature.

If I were to show you everyone else was wrong and I was right, would you think less of me? Please, don’t let it make you feel less of them. We’re just people trying to wade through the confusion.

The Truth On The Matter

Here is the truth and I’ll prove it in a screen shot to follow:

  1. INMEMORY_QUERY is enabled by default. If it is set you can trigger feature usage–full stop.
  2. INMEMORY_SIZE is zero by default.  Remember this is the supposedly ueber-powerful setting that precludes usage of the feature and not, in fact, the top-level-sounding INMEMORY_QUERY parameter. As such this should be the parameter that would prevent you for paying for usage of the feature.

In the following screenshot I’ll show that INMEMORY_QUERY is at the default setting of ENABLE  and INMEMORY_SIZE is at the default setting of zero. I prove first there is no prior feature usage. I then issue a CREATE TABLE statement specifying INMEMORY.  Remember, the feature-blocking INMEMORY_SIZE parameter is zero.  If  “they” are right I shouldn’t be able to trigger In-Memory Column Store feature usage, right? Observe–or better yet, try this in your own lab:

proof-mu

So ENABLED Means ENABLED? Really? Imagine That.

So I proved my point which is any instance with the default initialization parameters can trigger feature usage. I also proved that the words in the following three screenshots are factually incorrect:

MariaCallingMeOut

Screenshot of blog post on Oracle.com:

maria-on-inmemory_size_assertion

Screenshot of email to Oracle-L Email list:

 

kerry

 

Summary

I didn’t want to make a mountain out of this mole hill. It’s just a bug. I don’t expect apologies. That would be too human–almost as human as being completely wrong while wrongly clinging to one’s wrongness because others are equally, well, wrong on the matter.

 

BLOG UPDATE 2014.07.31: Click here to view an article on The Register regarding Oracle Database In-Memory feature usage.

BLOG UPDATE 2014.07.30: Oracle’s Maria Colgan has a comment thread on her blog on the In-Memory Column Store feature. In the thread a reader reports precisely the same bug behavior you will see in my proof below. Maria’s comment is that feature usage is tracked in spite of the supposed disabling feature INMEMORY_SIZE set to the default value. While this agrees with what I  already knew about this feature it is in my opinion not sufficient to speak of a bug of such consequence without citing the bug number. Furthermore, such a bug must be visible to users with support contracts. Click here for a  screenshot of the Oracle blog. In case Oracle changes their mind on such an apparently sensitive topic I uploaded the blog to the Wayback Machine here

BLOG UPDATE 2014.07.29: Oracle’s Maria Colgan issued a tweet stating “What u found in you 3rd blog is a bug [...] Bug 19308780.”  Click here for a screenshot of the tweet. Also, click here for a Wayback Machine (web.archive.org) copy of the tweet.

 

 

Sundry References

 Print out of Maria’s post on Oracle.com and link to same: Getting started with Oracle Database In-Memory Part I

Franck Pachot 2014.07.23 findings reported here:  Tweet , screenshot of tweet.

 

 

 


Filed under: oracle

Early Review of Google Classroom

Michael Feldstein - Mon, 2014-07-28 16:36

Meg Tufano is co-Founder of SynaptIQ+ (think tank for social era knowledge) and leader of McDermott MultiMedia Group (an education consulting group focused on Google Apps EDU). We have been checking out Google Classroom – with her as the teacher and me as the student. I include some of Meg’s bio here as it is worth noting her extensive experience designing and teaching online courses for more than a decade.

Meg posted a Google Slides review of her initial experiences using Google Classroom from a teacher’s perspective, which I am sharing below with minimal commentary. The review includes annotated slides showing the various features and Meg’s comments.

I have not done as much work to show the student view, but I will note the following:

  • The student view does not include the link to the Chrome Store that Meg finds to be too confusing.
  • The biggest challenge I’ve had so far is managing my multiple Google accounts (you have to be logged into the Google Apps for Edu as your primary Google account to enter Classroom, which is not that intuitive to students).
  • I wonder if Google will continue to use Google tools so prominently in Classroom (primary GDrive, YouTube, GDocs) or if the full release will make it easier to embed non-Google tools.
  • I have previously written “Why Google Classroom won’t affect institutional LMS market … yet”, and after initial testing, nothing has changed my opinion.
  • I have one other post linking to video-based reviews of Google Classroom here.

The post Early Review of Google Classroom appeared first on e-Literate.

auto-generate SQLAlchemy models

Catherine Devlin - Mon, 2014-07-28 15:30

PyOhio gave my lightning talk on ddlgenerator a warm reception, and Brandon Lorenz got me thinking, and PyOhio sprints filled my with py-drenaline, and now ddlgenerator can inspect your data and spit out SQLAlchemy model definitions for you:


$ cat merovingians.yaml
-
name: Clovis I
reign:
from: 486
to: 511
-
name: Childebert I
reign:
from: 511
to: 558
$ ddlgenerator --inserts sqlalchemy merovingians.yaml

from sqlalchemy import create_engine, Column, Integer, Table, Unicode
engine = create_engine(r'sqlite:///:memory:')
metadata = MetaData(bind=engine)

merovingians = Table('merovingians', metadata,
Column('name', Unicode(length=12), nullable=False),
Column('reign_from', Integer(), nullable=False),
Column('reign_to', Integer(), nullable=False),
schema=None)

metadata.create_all()
conn = engine.connect()
inserter = merovingians.insert()
conn.execute(inserter, **{'name': 'Clovis I', 'reign_from': 486, 'reign_to': 511})
conn.execute(inserter, **{'name': 'Childebert I', 'reign_from': 511, 'reign_to': 558})
conn.connection.commit()

Brandon's working on a pull request to provide similar functionality for Django models!

TechTalk v5.0 – The Age of Big Data with Alex Morrise

Pythian Group - Mon, 2014-07-28 14:06

Who: Hosted by Blackbird, with a speaking session by Alex Morrise, Chief Data Scientist at Pythian.

What: TechTalk presentation, beer, wine, snacks and Q&A

Where: Blackbird HQ – 712 Tehama Street (corner of 8th and Tehama) San Francisco, CA

When: Thursday July 31, 2014 from 6:00-8:00 PM

How: RSVP here!

TechTalk v5.0 welcomes to the stage, Alex Morrise, Chief Data Scientist at Pythian. Alex previously worked with Idle Games, Quid, and most recently Beats Music where he led the development of an adaptive, contextual music recommendation server.  Alex earned a PhD in Theoretical Physics from UC Santa Cruz.

This edition of TechTalk will be based on how the age of big data allows statistical inference on an unprecedented scale. Inference is the process of extracting knowledge from data, many times uncovering latent variables unifying seemingly diverse pieces of information. As data grows in complexity and dimension, visualization becomes increasingly difficult. How do we represent complex data to discover implicit and explicit relationships? We discuss how to Visualize Inference in some interesting data sets that uncover topics as diverse as the growth of technology, social gaming, and music.

You won’t want to miss this event, so be sure to RSVP.

 

Categories: DBA Blogs

Unexpected Shutdown Caused by ASR

Pythian Group - Mon, 2014-07-28 13:45

In past few days I had two incidents and an outage, for just a few minutes. However, outage in a production environment is related to cost relatively and strictly. The server that had outage was because of failing over and then failing back about 4 to 5 times in 15 minutes. I was holding pager, and was then involved in investigating root cause for this fail-over and failed-back. Looking at the events in SQL Server error logs did not give me any clue towards what was happening, or why so I looked at the Windows Event View’s System log. I thought, “Maybe I have something there!”

There were two events that came to my attention:

Event Type:        Error

Event Source:    EventLog

Event Category:                None

Event ID:              6008

Date:                     7/24/2014

Time:                     1:14:12 AM

User:                     N/A

Computer:          SRV1

Description:

The previous system shutdown at 1:00:31 AM on 7/24/2014 was unexpected.

 

Event Type:        Information

Event Source:    Server Agents

Event Category:                Events

Event ID:              1090

Date:                     7/24/2014

Time:                     1:15:16 AM

User:                     N/A

Computer:          SRV1

Description:

System Information Agent: Health: The server is operational again.  The server has previously been shutdown by the Automatic Server Recovery (ASR) feature and has just become operational again.

 

 

The errors are closely related to the feature called Automatic Server Recovery (ASR) which is mainly configured with the server, and comes with the hardware. In our case, HP Blade, ProLiant server. There has been some resources/threads already discussed around similar topic. Most of the hardware vendor has somewhat similar software with similar functionality made available for servers.

In my case, my understanding was that maybe firmware are out of date and requiring updating, or the servers are aged. Further, I have sent my findings to customer with an incident report.  In a couple of hours, I had a reply and the feedback I received was just what I was expecting, the hardware was aged.  This may be the case with you when you see a message in event viewer which reads like “System Information Agent: Health: The server is operational again.  The server has previously been shutdown by the Automatic Server Recovery (ASR) feature and has just become operational again.”  Go check with your system administrator. The root cause of this unexepcted shutdown may not be related or caused by the SQL Server, rather, the system itself.  Please keep in mind that this could be one of the reasons, and certainly not the only.

References:

Automatic System Recovery

 

Categories: DBA Blogs

UTL_FILE_DIR Security Weakness: Why and How To Use Oracle Directories

UTL_FILE_DIR is the database initialization parameter the Oracle Database uses to determine what operating system directories and files PL/SQL packages, functions, and procedures may read from or write to when using the standard UTL_FILE database package.  The directories specified in the UTL_FILE_DIR parameter may be accessed by any database user, which can be a security issue.  In Oracle 9iR2, Oracle released new functionality called “Directories” that provides a more secure and robust capability to access operating system directories and files.  The advantages of using Directories over UTL_FILE_DIR are –

  • Read and/or Write access to a Directory can be granted to individual database accounts or roles
  • A Directory can be added without having to bounce the database
  • Directory name is used in the UTL_FILE.FOPEN statement rather than the directory path, which allows changes to the directory path without modification to the PL/SQL source code
Securing UTL_FILE

The UTL_FILE database package is used to read from and write to operating system directories and files.  By default, PUBLIC is granted execute permission on UTL_FILE. Therefore, any database account may read from and write to files in the directories specified in the UTL_FILE_DIR database initialization parameter.

Oracle usually assumes that PUBLIC has execute permission on UTL_FILE, therefore, many Oracle product installations do not specifically grant execute permission on UTL_FILE to Oracle installed database accounts.  Consequently, revoking execute permission on UTL_FILE from PUBLIC will result in errors in a number of standard Oracle database utilities and The Oracle E-Business Suite.  Also, some Oracle products and third party products will grant execute on UTL_FILE to PUBLIC during the installation of the product.

We do not recommend revoking execute permission on UTL_FILE from PUBLIC in database instances running the Oracle E-Business Suite and other complex applications (i.e., SAP, Peoplesoft, Oracle Clinical, etc.) due to the possibility of encountering errors in the application and third party products.  Only revoke execute permission from PUBLIC in database instances where the application, third party products, and all database management tools can be thoroughly tested.  All Oracle delivered products must be tested since Oracle often assumes UTL_FILE is granted to PUBLIC and does not provide the necessary grants when any products are installed – this includes products like Enterprise Manager Grid Control and Apex.

Security considerations with UTL_FILE can be mitigated by removing all directories from UTL_FILE_DIR and using the Directory functionality instead.

Oracle E-Business Suite and UTL_FILE_DIR

The combination of UTL_FILE being granted to PUBLIC and UTL_FILE_DIR being publicly accessible creates a significant security issue for the Oracle E-Business Suite.  The Oracle E-Business Suite uses UTL_FILE_DIR to read and write concurrent manager request temporary files.  Also, UTL_FILE_DIR is extensively used by most organizations to access interface and conversion data files from PL/SQL interface programs.

In the Oracle E-Business Suite, UTL_FILE_DIR is usually set to include at least the directories specified in $APPLPTMP and $APPLTMP – in the default installation this will include at least “/usr/tmp”.  Frequently, additional custom directories will be included for custom interfaces and other custom programs.

By accessing the APPLSYSPUB database account, an attacker can easy read and write interface data files.  Depending on the exact configuration, implemented modules, and custom interfaces, this could allow access to sensitive information including social security numbers and credit card numbers.

Migrating From UTL_FILE_DIR

For Oracle E-Business Suite customers, migrating from UTL_FILE_DIR to Directories requires only minimal changes and may require no source code changes depending on the design of the interfaces and other custom programs. The steps are as follows -

  1. Identify where UTIL_FILE is used
  2. Create Directories
  3. Change FOPEN calls to use Directories
  4. Edit UTL_FILE_DIR to remove physical Directories

Step One – Identify Where UTIL_FILE Is Used

The most difficult issue is identifying the packages, functions, and procedures using the physical directories in UTL_FILE_DIR.  The UTL_FILE_DIR physical directories are only directly referenced by the UTL_FILE.FOPEN function.  The FOPEN specifies the operating system directory and file name to open.  All subsequent read, write, and close function calls use the file handle returned by FOPEN.

The following SQL may assist in identifying uses of UTL_FILE_DIR in FOPEN statements –

SELECT * FROM dba_source 

WHERE upper(text) like '%FOPEN%' 

AND name like '%<custom prefix>%' 

AND owner = 'APPS' 

 If the calls to FOPEN are not in a common function and are not accessed through some other indirection, it should be fairly straightforward to find and change the necessary FOPEN references in any custom PL/SQL packages, functions, and procedures.  If the physical directory paths are stored in a table or in a concurrent program definition, then no changes to the source code are required.

At this time, converting the standard the Oracle E-Business Suite directories ($APPLPTMP and $APPLTMP) is not recommended as this is not supported by Oracle.  Theoretically it should work without any issues, however, the Oracle E-Business Suite references the directories in multiple places including the $APPLPTMP and $APPLTMP environmental variables, system profile options (e.g., “ECX: XSLT File Path”), and potentially in some configuration files.

Step Two – Create Directories

The following general steps are required to change the references from UTL_FILE_DIR to Directories. Please note that the directory name MUST always be in uppercase in all UTL_FILE.FOPEN statements, otherwise errors may be encountered

For each custom directory in UTL_FILE_DIR, execute the following SQL statements in each development, test, and production database instance –

 
CREATE OR REPLACE DIRECTORY <name> AS '<physical directory path>'; 
GRANT READ, WRITE ON DIRECTORY <name> TO APPS;            

as an example –

CREATE OR REPLACE DIRECTORY TMP_DIR AS '/usr/tmp'; 
GRANT READ, WRITE ON DIRECTORY TMP_DIR TO APPS; 

 

The directories “/usr/tmp” and “../comn/temp” and any other directories specified in $APPLPTMP and $APPLTMP should remain in UTL_FILE_DIR, since these directories are required by Oracle.

Step Three – Change FOPEN Calls to Use Directories

Once directories have been created the next step is to edit your code to use them. The process is straightforward. If a physical directory is specified in the UTL_FILE.FOPEN statement, change the hard-coded path to the Directory name. 

As an example –

FILE_ID := UTL_FILE.FOPEN('/usr/tmp', 'dummy.txt', 'W'); 

 change to –

FILE_ID := UTL_FILE.FOPEN('TMP_DIR', 'dummy.txt', 'W'); 

 Two pointers to keep in mind:

  1. Always be sure to use the directory name in uppercase and it must be enclosed in single quotes  
  2. If the physical directory is specified in a table or as a parameter in a Concurrent Program definition, then just specify the Directory name rather than a physical path – /usr/tmp becomes TMP_DIR

Step Four – Edit UTL_FILE_DIR to Remove Physical Directories

Remove all the custom physical directories from UTL_FILE_DIR.  The standard Oracle directories of ‘/usr/tmp’, ‘../comn/temp’, etc. should not be removed.  The database must be bounced for the change to take effect.

 

If you have questions, please contact us at info@integrigy.com

References Tags: Information DisclosureOracle DatabaseOracle E-Business Suite
Categories: APPS Blogs, Security Blogs

<b>Contributions by Angela Golla,

Oracle Infogram - Mon, 2014-07-28 10:58
Contributions by Angela Golla, Infogram Deputy Editor

Source and Hire the Best Talent
Oracle Taleo Recruiting Cloud Service is the market leading solution that helps enterprises source, assess, and hire the best talent. In addition to automating the process for even the most complex global organization, Oracle Taleo Recruiting Cloud Service delivers insights to continuously improve talent acquisition efficiency and effectiveness.  Learn more at the Oracle Taleo Recruiting homepage. 

Plan Instability

Dominic Brooks - Mon, 2014-07-28 09:04

There seems to me to be a relatively simple choice.

Either you except that the Oracle Optimizer has a wealth of complicated strategies and, in this complex effort to get the best executions it can, will inevitably either get it wrong sometimes (or take some extra executions to realise it’s wrong).

Or you stick your head in the sand and raise a bug for every unexpected poor execution or plan flip.

But let’s say that above else you wanted Plan Stability.

This is an widespread desire.

What would be the best strategy?
And to what lengths would you take it?

SQL Plan Management features are designed to give you some degree of stability.

You could baseline statements which have given you a problem in the past.

What if that is not deemed adequate?

So, on the one hand, you could baseline everything you could find.

Taking a long history of AWR and taking regularly snaps of V$SQL, you could put all in a SQL Plan Baseline.

But, depending on your application, you might regularly get new SQL appear unprotected by a baseline.
In one Hibernate application I know, a change in the application domain model often results in the table aliases changing, meaning that there is lots of brand new, but very familiar, SQL.
So, you then become dependant on having a reliable testing infrastructure which will generate nearly all the SQL you’re likely to get in advance of production.

In addition, you might want multiple plans in your baseline – for ACS for example – and then, once that extra bindset awareness is flushed out of memory, you then need a couple of executions to rediscover it (unless you use a SQL Patch to inject BIND_AWARE into the specific sql statements).

It’s certainly no guarantee of stability.

What is the alternative?

I like to argue that most problematic executions stem from two sources:

  1. Badly written SQL
  2. Using poor combinations of features – e.g. bind variable peeking + histograms

The other day I made a comment to someone that “if you’re heavily reliant on ACS, perhaps you should rethink whether you should be using literals rather than bind variables”.

So, you might then take the position that plan instability stems from increasing complexity in the optimizer.

In which case, maybe a viable strategy might be to turn off much of the complexity:

  • Bind Variable Peeking
  • Cardinality feedback
  • Adaptive Cursor Sharing
  • Dynamic Sampling
  • Adaptive direct path reads
  • 12c adaptive execution plans
  • Anything with the word “adaptive” in it?
  • Default Stats job and/or default method_opt histogram behaviour

This seems quite radical to many. Perhaps justifiably.

Perhaps it would be nice if there were new optimizer strategies – OPTIMIZER_MODE? – perhaps “AGGRESSIVE_PERFORMANCE” (maybe “ADAPTIVE_PERFORMANCE_WITH_THE_ODD_HICCUP” ?) vs “PERFORMANCE_STABILITY” which were an umbrella of many of the features above.

Thoughts?

To what extent are you concerned with performance stability, above all else?

How many of the optimizer features in each release are actually aligned with that widespread concern?


Four Options For Oracle DBA Tuning Training

Four Options For Oracle DBA Tuning Training
Oracle DBAs are constantly solving problems... mysteries. That requires a constant knowledge increase. I received more personal emails from my Oracle DBA Training Options Are Changing posting than ever before. Many of these were from frustrated, angry, and "stuck" DBAs. But in some way, almost all asked the question, "What should I do?"

In response to the "What should I do?" question, I came up with four types of Oracle DBA performance tuning training that are available today. Here they are:

Instructor Led Training (ILT) 
Instructor Led Training (ILT) is the best because you have a personal connection with the teacher. I can't speak for other companies, but I strive to connect with every student and every student knows they can personally email or call me...even years after the training. In fact, I practically beg them to do what we do in class on their production systems and send me the results so I can continue helping them. To me being a great teacher is more than being a great communicator. It's about connection. ILT makes connecting with students easy.

Content Aggregators
Content Aggregators are the folks who pull together free content from various sources, organize and display it. Oh yeah... and they profit from it. Sometimes the content value is high, sometimes not. I tend to think of content aggregators like patent trolls, yet many times they can be a great resource. The problem is you're not dealing with the creator of the content. However, the creator of the content actually knows the subject matter. You can somtimes contact them...as I encourage my students and readers to do.

Content Creators
Content Creators are the folks who create content based on their experiences. We receive that content through their blogs, videos, conference presentations and sometimes through their training. I am a content creator but with an original, almost child-like curiosity, performance research twist. Content creators rarely directly profit from their posted content, but somehow try to transform it into a revenue stream. I can personally attest, it can be a risky financial strategy...but it's personally very rewarding. Since I love do research, it's easy and enjoyable to post my findings so others may benefit.

Online Training (OLT)
Online Training (OLT) is something I have put off for years. The online Oracle training I have seen is mostly complete and total crap. The content is usually technically low and mechanical. The production quality is something a six year old can do on their PC. The teaching quality is ridiculous and the experience puts you to sleep. I do not ever want to be associated with that kind of crowd.

I was determined to do something different. It had to be the highest quality. I have invested thousands of dollars in time, labor, and equipment to make online video training
Craig teaching in an OraPub Online Institute Seminarwork. Based on the encouraging feedback I receive it's working!

This totally caught me by surprise. I have discovered that I can do things through special effects and a highly organized delivery that is impossible to do in a classroom. (Just watch my seminar introductions on YouTube and you'll quickly see what I mean.) This makes the content rich and highly compressed. One hour of OraPub Online Institute training is easily equivalent to two to four hours of classroom training. Easily. I have also strive to keep the price super low, the production at a professional level and ensure the video can be streamed anywhere in the world and on any device. Online training is an option, but you have to search for it.

Summary
So there you have it. Because of economics and the devaluation of DBAs as human beings coupled with new technologies, the Oracle DBA still has at least four main sources of training and knowledge expansion. Don't give up learning!

Some of you reading may be surprised that I'm writing about this topic because it will hurt my traditional instructor led training (public or on-site) classes. I don't think so. If people can attend my classes in person, they will. Otherwise, I hope they will register for an OraPub Online Institute seminar. Or, at least subscribe to my blog (see upper left of page).

All the best in your quest to do great work,

Craig.
You can watch seminar introductions (like above) for free on YouTube! If you enjoy my blog, subscribing will ensure you get a short-concise email about a new posting. Look for the form on this page.

P.S. If you want me to respond to a comment or you have a question, please feel free to email me directly at craig@orapub .com.

Categories: DBA Blogs

Logging for Slackers

Pythian Group - Mon, 2014-07-28 07:41

When I’m not working on Big Data infrastructure for clients, I develop a few internal web applications and side projects. It’s very satisfying to write a Django app in an afternoon and throw it on Heroku, but there comes a time when people actually start to use it. They find bugs, they complain about downtime, and suddenly your little side project needs some logging and monitoring infrastructure. To be clear, the right way to do this would be to subscribe to a SaaS logging platform, or to create some solution with ElasticSearch and Kibana, or just use Splunk. Today I was feeling lazy, and I wondered if there wasn’t an easier way.

Enter Slack

Slack is a chat platform my team already uses to communicate – we have channels for different purposes, and people subscribe to keep up to date about Data Science, our internal Hadoop cluster, or a bunch of other topics. I already get notifications on my desktop and my phone, and the history of messages is visible and searchable for everyone in a channel. This sounds like the ideal lazy log repository.

Slack offers a rich REST API where you can search, work with files, and communicate in channels. They also offer an awesome (for the lazy) Incoming WebHooks feature – this allows you to POST a JSON message with a secret token, which is posted to a pre-configured channel as a user you can configure in the web UI. The hardest part of setting up a new WebHook was choosing which emoji would best represent application errors – I chose a very sad smiley face, but the devil is also available.

The Kludge

Django already offers the AdminEmailHandler, which emails log messages to the admins listed in your project. I could have created a mailing list, added it to the admins list, and let people subscribe. They could then create a filter in their email to label the log messages. That sounds like a lot of work, and there wouldn’t be a history of the messages except in individual recipients’ inboxes.

Instead, I whipped up this log handler for Django which will post the message (and a stack trace, if possible) to your Slack endpoint:

from logging import Handler
import requests, json, traceback
class SlackLogHandler(Handler):
   def __init__(self, logging_url="", stack_trace=False):
      Handler.__init__(self)
      self.logging_url = logging_url
      self.stack_trace = stack_trace
   def emit(self, record):
      message = '%s' % (record.getMessage())
      if self.stack_trace:
         if record.exc_info:
            message += '\n'.join(traceback.format_exception(*record.exc_info))
            requests.post(self.logging_url, data=json.dumps({"text":message} ))

There you go: install the requests library, generate an Incoming WebHook URL at api.slack.com, stick the SlackLogHandler in your Django logging configuration, and your errors will be logged to the Slack channel of your choice. Stack traces are optional – I’ve also been using this to post hourly reports of active users, etc. to the channel under a difference username.

For reference, here’s a log configuration for the Django settings.py. Now go write some code, you slacker.

LOGGING = {
    'version':1,
    'disable_existing_loggers':False,
    'handlers': {
        'console': {
            'level':'DEBUG',
            'class':'logging.StreamHandler',
        },
        'slack-error': {
            'level':'ERROR',
            'class':'SlackLogHandler.SlackLogHandler',
            'logging_url':'<SuperSecretWebHookURL>',
            'stack_trace':True
        }
    }
    'loggers': {
        'django': {
            'level': 'INFO',
            'handlers': ['console', 'slack-error']
        }
    }
}
Categories: DBA Blogs

The Importance of Documentation

Chris Foot - Mon, 2014-07-28 07:05

As a remote data infrastructure services provider, documenting the activities we perform, as well as our customers’ environments, is critical to our success. RDX currently supports thousands (and thousands) of database ecosystems ranging in complexity from simple to “making your hair stand on end.”

My customers hold my organization to extremely high standards. Rightfully so, they have turned over the keys to their most sensitive and mission-critical data stores to RDX. At the end of every email blast I send to our customer base, I end it with, “I personally appreciate you placing your trust in us to administer and safeguard your most valuable data assets. We take that responsibility very seriously here at RDX.” Stating that we take that responsibility seriously is kind of like saying the Titanic sprung a small leak.

Although the importance of a well thought out and detailed documentation library is blatantly obvious, creating documentation is the task most often postponed by an overworked DBA unit.

Documenting processes, procedures and best practices is a task that is often considered to be boring and mundane. Most DBAs would rather perform virtually any other activity than sit in front of a screen using a word processor. As a result, creating documentation is often postponed until the DBA has a little free time to kill. Today’s database administration units are operating with smaller staffs, tighter budgets and ever-increasing workloads. The end result is that the documentation is either never created or created and not kept current.

However, a robust detailed documentation library creates an environment that is less complex, less error-prone, reduces the amount of time DBAs spend learning new database environments and reduces the overall time spent on day-to-day support activities. DBAs are able to spend more time administering the environment rather than finding the objects they are trying to support and the processes and programs used to administer them.

The nature of my business as a remote services provider demands excellent documentation. The majority of environments we administer weren’t designed by my organization. The only way that we can ensure high quality and high-speed administration of these environments is to document them thoroughly. We document everything from initial connectivity and customer contact sheets to detailed information on database and server information, batch job streams and individual program execution (what it does, run-time windows). If we need to be aware of it, we have it documented.

Documentation is also the foundation of many of the other disciplines I will be discussing in future blogs. Let’s continue our discussion with a few helpful hints to get you started.

Understanding the Important Role Good Documentation Plays

We all generally understand the benefits that documentation provides. I think that all readers will see the importance I personally place on documentation in upcoming blogs.

Let me reaffirm my opinion in this one sentence: Good documentation is the foundation that high-quality data infrastructure services are built upon.

Creating an Organizational Environment That Fosters Good Documentation

I’ve been the Vice President of Service Delivery at RDX for 6 years now. It is my responsibility as manager to create an environment that fosters the production of robust and high-quality documentation. Let me describe some of the challenges that I have faced in the past at other organizations and how I have overcome them.

Since I view high quality documentation to be my responsibility as a manager, I ensure that it becomes part of every DBA’s performance appraisal criteria, including my own. If it isn’t on my, and my unit’s, performance appraisal forms, I will ask to have it added or make my own personal addendum and notify both the DBA team and management that I have done so.

I will add time for documentation when I estimate the amount of time it will take me to perform an administrative task during project planning meetings. I don’t settle for “we can do that after the project is complete” as an answer.

If you continuously sell the importance of documentation, sooner or later, you will begin to wear your opponents down. Although I prefer to call it “being relentless,” I’m sure that many of the application development managers (and my own managers) viewed it as “being a ….” (insert your favorite description here).

Every document I have created that provides a list of activities I , or my unit, need to perform during a project has documentation included. It helps to integrate it into the fabric and culture of my organization’s environment.

Making Documentation Second Nature

You also need to ensure that generating documentation becomes a natural part of your daily activities. You must continuously remind yourself that documentation is a primary and integral part of providing high-quality support services to your customers.

You must also remind yourself that it makes your job easier and benefits your fellow DBAs. It is a recipe for disaster when a fellow DBA needs to be out of the office for a time and asks another DBA to “help them out” by performing a complex, application-specific administrative activity and then tries to verbally tell them how to perform the 326 steps it takes to execute it.

Did you ever try to refresh an ERP application test environment from production when that test environment doesn’t have enough space to hold all of production’s data? 4,000 steps later, you begin to second-guess your choice of professions. That was the exact request from one of my fellow DBAs when I first started in this profession, and it quickly taught me the importance of good documentation. Not only did he get me to do the refresh, but I also had to document the process for him along the way. Some call that being a good coworker; I would view that as having a big sucker taped to my forehead.

The moral of this story is this: If you don’t want to be the only one that can perform that 900 step ERP application production to test refresh, document it! If you don’t want to be called by the on-call DBA because he doesn’t know exactly where to add a file in an emergency situation (like someone forgetting to tell you that they were loading 10 million additional rows into that 100 row table), document it! The more you document, the easier your life as a DBA becomes.

I’ve never had a photographic memory. It makes generating documentation easy for me. I also like to write, and that helps, but I will admit that there are times that I would rather perform virtually any other activity than document.

However, it has become easier because I continuously reaffirm to myself the importance of documentation. The more you reinforce that to yourself, the more second nature (and easier) it becomes.

Making Documentation Easy

I’m a huge fan of documentation templates. Here at RDX, we have templates and Standard Operating Procedures for everything we document. If it is repeatable or a complex process, we have an SOP for it. We have templates for documenting connections to our customers’ environments, their backup and recovery environments and their application specific processes, to name a few. If it needs to be documented on a regular basis, we have a template for it. We also have generic templates for documenting environments and activities that don’t fit into other templates.

Word Documents and Templates

Word document templates provide many features that streamline the documentation process and help to improve the quality of the content they store. I try to take advantage of as many features as I can. I use drop-down selection menus, check boxes and radio push buttons to improve the speed and quality of the documentation process. I also take advantage of the help pop-up feature that Microsoft Word provides to create a detailed description of what information is to be entered into that field, check box or radio button.

Wikis

We heavily utilize Wikis to intelligently and securely display information about the environments we are tasked with supporting. A common, menu-driven interface has been designed, tuned and tweaked over our 20 year history. The Wiki’s contents include customer contact and escalation information, detailed database/server information, customer change management procedures, RDX DBAs assigned to the account, on-call team assigned, non-sensitive connection information (VPN type, VPn vendor, etc) and job information. The Wiki page also links to current tickets, current time cards and a history of past problems contained in our problem resolution library.

The Wiki content is controlled by a well-defined change management procedure and relies upon Wiki templates to ensure information is stored and displayed in a common format that adheres to RDX specifications. Once again, templates help improve the quality of content, speed data entry and ensure a uniformity of display pages and menus. We constantly review the Wiki for content and usability as well as leverage new Wiki features as they are released.

Database-Driven Content Managers

There are dozens of software companies that offer content management solutions. Database vendors have also recognized this as a lucrative market. All of the major database vendors now offer advanced content management software, each one trying to outdo the other in the number of bells and whistles that their products offer. Do a quick search on Google for documentation content management software, and you will find out just how many competing products there are.

Content management products offer check-in/check-out features, document versioning, web portal access and advanced workflow capabilities to name just a few of the features designed to improve content management. The competition in the content management market space is fierce to say the least. Content management vendors know that continuously adding new bells and whistles to their products is not just important for increasing market share, but it also is critical for their survival. Product costs can range from thousands to tens of thousands of dollars (or more).

If you have the funds and your management understands the benefits that a full-blown content management package provides, by all means begin a content management product analysis. But if you don’t have the funds, create a shared drive on your network and declare it to be the “DBA Documentation Portal.”

What to Document

By all means, this is not an all-inclusive list of what can be documented. Consider it as a starter kit to help you begin your quest for “documentis nirvanas.” Is some of this overkill for your particular environment? Maybe, but just consider this a general, high-level list. Since most readers will work for a single organization, I’m focusing my recommendations on DBA units that support one corporate environment.

Database Environment Documentation

  • Naming conventions
  • Servers (server names, operating system release, hardware vendor)
  • Databases (vendor, database version, features enabled)

Application-Specific Documentation

  • Application type (i.e. data warehouse, online transaction processing, decision support, third-party application name and functionality it provides).
  • Business unit requirements and related information for supported databases
  • Uptime requirements (i.e. 24 X 7, 8 X 5)
  • Database downtime windows
  • Critical job processes
  • Business unit and application developer contact lists
  • Turnover windows for database changes
  • Problem notification and escalation procedures
  • Security sensitivity- How sensitive is the data?

Process Documentation

  • Repeatable administrative processes (covered in an upcoming blog)
  • Backups – Probably the most critical set of documentation you will ever create- Document how it is backed up, what scripts back it up, where the backup is going to, retention periods and backup message directories. If it is involved with a backup, DOCUMENT IT. Review the document with other units that are involved in the backup and recovery process. It is your responsibility to ensure that you don’t hear an operator say, “What retention period? Nobody told me we were to have a retention on these files” when you are in a recovery situation. Remember that Oracle states that human error, including miscommunications, is responsible for over 90% of failed recoveries. If you want to reduce recovery failures, DOCUMENT THE PROCESS AND REVIEW IT.
  • Anything else you run on a regular basis to support a specific application
  • Change management- I’ll be spending an entire blog, or two, on this
  • A daily monitoring activity checklist to ensure that no activity is missed- We have daily, weekly and monthly activities that are to be performed for each of our customers
  • Complex administrative activities performed regularly
  • Test and reporting database refreshes
  • Data reorganizations
  • Disaster recovery tests- The processes required to perform the recovery AND the criteria that will be used to evaluate whether it was successful or not

Object Documentation

  • DBA-specific stored PL/SQL and TSQL programs
  • Day-to-day support scripts (where they are and what they do)
  • Monitoring scripts (where they are and what they do)
  • Scripts used to perform database administrative changes- I personally utilized specific directories that provide output from critical database changes that I have performed and other directories containing the SQL used to make that change
  • Operating system scripts- Document what the script does in the beginning of each of your scripts. Did you ever try to determine what a 400 line script does that was created by someone who knows much more about UNIX scripting than you do? We have all been in that position at one time or another during our career. Make it easy on your coworkers to understand what the script does by putting comments at the top of the script as well as in the body. Also keep a running history of script changes, what they were and the time they were made

Database Administration Unit Organizational Documentation

  • Contact Information
  • DBA roles and responsibilities- Which applications, databases and tasks they are responsible for supporting
  • DBA unavailability- Allows application developers to plan for a DBA not being available

It is a good practice to distribute this information to all business units supported by the database administration unit.

I hope you enjoyed this blog on documentation and the important role it plays.

The post The Importance of Documentation appeared first on Remote DBA Experts.

Using HAWQ with PHD service in PCF 1.2

Pas Apicella - Mon, 2014-07-28 06:01
The following demo shows how to use the PCF 1.2 PHD service with HAWQ by loading data into the PCF PaaS platform.

1. First lets setup our ENV to use the correct version of HADOOP on our local laptop.

export HADOOP_INSTALL=/Users/papicella/vmware/software/hadoop/hadoop-2.0.5-alpha
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

export PATH=$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

export HADOOP_OPTS="$HADOOP_OPTS  -Djava.awt.headless=true -Djava.security.krb5.realm=-Djava.security.krb5.kdc="

export YARN_OPTS="$YARN_OPTS -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk -Djava.awt.headless=true"

hadoop version

2. Set the HADOOP_USER_NAME to ensure you have write access to load a file.

export HADOOP_USER_NAME=ucc3a04008db2486

3. Create a file called person.txt with some pipe delimited data , example below.

[Mon Jul 28 21:47:37 papicella@:~/vmware/software/hadoop/cloud-foundry/pcf12/demo ] $ head person.txt
1|person1
2|person2
3|person3
4|person4
5|person5

4. Load the file into the PHD instance running in PCF 1.2. You will need to use the name node / path which is correct for your PHD instance.

[Mon Jul 28 21:51:43 papicella@:~/vmware/software/hadoop/cloud-foundry/pcf12/demo ] $ hadoop fs -put person.txt hdfs://x.x.x.x:8020/user/ucc3a04008db2486/

5. Create a HAWQ table to the file person.txt using PXF as shown below.

CREATE EXTERNAL TABLE person (id int, name text)
LOCATION ('pxf://x.x.x.x:50070/user/ucc3a04008db2486/person.txt?Fragmenter=HdfsDataFragmenter&Accessor=TextFileAccessor&Resolver=TextResolver')
FORMAT 'TEXT' (DELIMITER = '|');

6. Query the table as shown below.



For more information on the PHD service see the link below.

http://docs.pivotal.io/pivotalhd-ds/index.html

http://feeds.feedburner.com/TheBlasFromPas
Categories: Fusion Middleware