The Oracle Instructor
Automatic Storage Management (ASM) is becoming the standard for good reasons. Still, the way it mirrors remains a mystery for many customers I encounter, so I decided to cover it briefly here.ASM Basics: What does normal redundancy mean at all?
It means that every stripe is mirrored once. There is a primary on one disk and a mirror on another disk. All stripes are spread across all disks. High redundancy would mean that every primary stripe has two mirrors, each on another disk. Obviously, the mirroring reduces the usable capacity: It’s one half of the raw capacity for normal redundancy and one third for high redundancy. The normal redundancy as on the picture safeguards against the loss of any one disk.ASM Basics: Spare capacity
When disks are lost, ASM tries to re-establish redundancy again. Instead of using spare disks, it uses spare capacity. If enough free space in the diskgroup is left (worth the capacity of one disk) that works as on the picture above.ASM 11g New Feature: DISK_REPAIR_TIME
What if the disk from the picture above is only temporarily offline and comes back online after a short while? These transient failures have been an issue in 10g, because the disk got immediately dropped, followed by a rebalancing to re-establish redundancy. Afterwards an administrator needed to add the disk back to the diskgroup which causes again a rebalancing. To address these transient failures, Fast Mirror Resync was introduced:
No administrator action required if the disk comes back before DISK_REPAIR_TIME (default is 3.6 hours) is over. If you don’t like that, setting DISK_REPAIR_TIME=0 brings back the 10g behavior.ASM 12c New Feature: FAILGROUP_REPAIR_TIME
If you do not specify failure groups explicitly, each ASM disk is its own failgroup. Failgroups are the entities across which mirroring is done. In other words: A mirror must always be in another failgroup. So if you create proper failgroups, ASM can mirror according to your storage layout. Say your storage consists of four disk arrays (each with two disks) like on the picture below:
That is not yet the new thing, failgroups have been possible in 10g already. New is that you can now use the Fast Mirror Resync feature also on the failgroup layer with the 12c diskgroup attribute FAILGROUP_REPAIR_TIME. It defaults to 24 hours.
So if maintenance needs to be done with the disk array from the example, this can take up to 24 hours before the failgroup gets dropped.
I hope you found the explanation helpful, many more details are here :-)
Tagged: ASM, Brief Introduction
I’m in the Leadership Circle again :-) That is a quarterly Corporate Award for the best instructors worldwide according to customer feedback.
Oracle University has generally high quality standards, so it is hard to stand out individually. The more I am proud to be listed together with these great colleagues:
2014 brought a new high water mark with 282,000 hits!
Meanwhile, uhesse.com gets over 1,000 hits per day on average.Who are the referrers?
The vast majority of visitors came to uhesse.com via google searches: 163,000
On a distant second place is Twitter: 2,500 visitors came from there
Facebook and LinkedIn each lead 800 visitors to uhesse.com
400 visitors came from jonathanlewis.wordpress.com – thank you, Jonathan!Where did uhesse.com refer to?
They came from almost everywhere, the full list contains 200 countries in total – isn’t the internet a great thing?
Thank you all for visiting – I hope you come back in 2015 again :-)
These I consider the most important points about Exadata Patching:Where is the most recent information?
MOS Note 888828.1 is your first read whenever you think about Exadata PatchingWhat is to patch with which utility?
Expect quarterly bundle patches for the storage servers and the compute nodes. The other components (Infiniband switches, Cisco Ethernet Switch, PDUs) are less frequently patched and not on the picture therefore.
The storage servers have their software image (which includes Firmware, OS and Exadata Software) exchanged completely with the new one using patchmgr. The compute nodes get OS (and Firmware) updates with dbnodeupdate.sh, a tool that accesses an Exadata yum repository. Bundle patches for the Grid Infrastructure and for the Database Software are being applied with opatch.Rolling or non-rolling?
This the sensitive part! Technically, you can always apply the patches for the storage servers and the patches for compute node OS and Grid Infrastructure rolling, taking down only one server at a time. The RAC databases running on the Database Machine will be available during the patching. Should you do that?
Let’s focus on the storage servers first: Rolling patches are recommended only if you have ASM diskgroups with high redundancy or if you have a standby site to failover to in case. In other words: If you have a quarter rack without a standby site, don’t use rolling patches! That is because the DBFS_DG diskgroup that contains the voting disks cannot have high redundancy in a quarter rack with just three storage servers.
Okay, so you have a half rack or bigger. Expect one storage server patch to take about two hours. That summarizes to 14 hours (for seven storage servers) patching time with the rolling method. Make sure that management is aware about that before they decide about the strategy.
Now to the compute nodes: If the patch is RAC rolling applicable, you can do that regardless of the ASM diskgroup redundancy. If a compute node gets damaged during the rolling upgrade, no data loss will happen. On a quarter rack without a standby site, you put availability at risk because only two compute nodes are there and one could fail while the other is just down.Why you will want to have a Data Guard Standby Site
Apart from the obvious reason for Data Guard – Disaster Recovery – there are several benefits associated to the patching strategy:
You can afford to do rolling patches with ASM diskgroups using normal redundancy and with RAC clusters that have only two nodes.
You can apply the patches on the standby site first and test it there – using the snapshot standby database functionality (and using Database Replay if you licensed Real Application Testing)
A patch set can be applied on the standby first and the downtime for end users can be reduced to the time it takes to do a switchover
A release upgrade can be done with a (Transient) Logical Standby, reducing again the downtime to the time it takes to do a switchover
I suppose this will be my last posting in 2014, so Happy Holidays and a Happy New Year to all of you :-)
The Oracle circus went to Liverpool this year for the annual conference of the UK Oracle User Group and it was a fantastic event there! Top speakers and a very knowledgeable audience too, I was really impressed by the quality we have experienced. Together with my friends and colleagues Iloon and Joel, I was waving the flag for Oracle University again – and it was really fun to do so :-)
One little obstacle was that I actually did many presentations and roundtables. So less time for me to listen to the high quality talks of the other speakers…
Joel and I hosted three roundtables:
About Exadata, where we had amongst others Dan Norris (Member of the Platform Integration MAA Team, Oracle) and Jason Arneil (Solutions Architect, e-DBA) contributing
About Grid Infrastructure & RAC, where Ian Cookson (Product Manager Clusterware, Oracle) took many questions from the audience. We could have had Markus Michalewicz also if I only would have told him the day before during the party night – I’m still embarrassed about that.
About Data Guard, where Larry Carpenter (Master product Manager Data Guard and Maximum Availability Architecture, Oracle) took all the questions as usual. AND he hit me for the article about the Active Data Guard underscore parameter, so I think I will remove it…
Iloon delivered her presentation about Apex for DBA Audience, which was very much appreciated and attracted a big crowd again, same as in Nürnberg before.
Joel had two talks on Sunday already: Managing Sequences in a RAC Environment (This is actually a more complex topic than you may think!) and Oracle Automatic Parallel Execution (Obviously complex stuff)
I did two presentations as well: The Data Guard Broker – Why it is recommended and Data Guard 12c New Features in Action
Both times, the UKOUG was so kind to give me very large rooms, and I can say that they haven’t looked empty although I faced tough competition by other interesting talks. This is from the first presentation:
A big THANK YOU goes out to all the friendly people of UKOUG who made this possible and maintained the great event Tech14 was! And also to the bright bunch of Oracle colleagues and Oracle techies (speakers and attendees included) that gave me good company there: You guys are the best! Looking forward to see you at the next conference :-)
Together with Craig Shallahamer, Lucas Jellema, Mark Rittman, Martin Bach, Pete Finnigan and Tim Fox, I will be presenting in Dubai. My topic is Minimizing Downtime with Rolling Upgrade using Data Guard
Click on the picture for details, please!
Hope to see you there :-)
Just wanted to share some pieces of information from the recent DOAG annual conference that you may find interesting.
From Mike Dietrich’s presentation about Database Upgrade:
From Carsten Czarski’s talk about XML DB:
From Ulrike Schwinn’s talk about the Resource Manager I took away thatThe resource manager becomes more and more popular and important, especially for Multitenant
- something Hans Forbrich reinforced later on.
Particularly I liked way she presented later on about ADO: Very many live demonstrations – that’s how I try to do my own presentations also :-)
Frank Schneede did a great job debunking Exadata myths. For example,You don’t need to have all cores enabled with Exadata X4 in order to save license cost. That’s called Capacity on Demand.
If I should name one presentation that was most useful for me, it’ll be probably Frank’s.
Markus Michalewicz delivered an excellent talk as expected about RAC cache fusion:
Two important messages:RAC scales well (far) beyond three nodes because there are never more than three nodes involved for cache fusion intercommunication. And Multitenant and RAC are a perfect fit.
One Data Guard snippet out of Larry Carpenter’s talk about Global Data Services (GDS):GDS makes it possible to automate the failover of the Real-Time Query service to the primary in case the physical standby has an outage.
Hans Forbrich talked about Multitenant. He showed great presentation skills and although I knew the technical details before, the way he highlighted certain aspects was still very helpful for me.
One key message was thatMultitenant is here to stay. DBAs should learn about it and become familiar with it as soon as possible, because sooner than later it will have to be administered in production!
When you look into V$RECOVERY_AREA_USAGE, you see a strange row at the bottom:
SQL> select * from v$recovery_area_usage; FILE_TYPE PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES CON_ID ----------------------- ------------------ ------------------------- --------------- ---------- CONTROL FILE 0 0 0 0 REDO LOG 0 0 0 0 ARCHIVED LOG 10.18 0 73 0 BACKUP PIECE 0 0 0 0 IMAGE COPY 0 0 0 0 FLASHBACK LOG 0 0 0 0 FOREIGN ARCHIVED LOG 0 0 0 0 AUXILIARY DATAFILE COPY 0 0 0 0
Curious what that could be? You will see values other than zero on a Logical Standby Database:
SQL> connect sys/oracle@logst as sysdba Connected. SQL> select database_role from v$database; DATABASE_ROLE ---------------- LOGICAL STANDBY SQL> select * from v$recovery_area_usage; FILE_TYPE PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES CON_ID ----------------------- ------------------ ------------------------- --------------- ---------- CONTROL FILE 0 0 0 0 REDO LOG 0 0 0 0 ARCHIVED LOG 14.93 0 9 0 BACKUP PIECE 0 0 0 0 IMAGE COPY 0 0 0 0 FLASHBACK LOG 0 0 0 0 FOREIGN ARCHIVED LOG 2.03 0 26 0 AUXILIARY DATAFILE COPY 0 0 0 0
In contrast to a Physical Standby Database, this one writes not only into standby logs but also into online logs while being in standby role. That leads to two different kinds of archive logs:
When DML (like insert and update) is done on the primary 1) that leads to redo entries into online logs 2) that are simultaneously shipped to the standby and written there into standby logs 2) also. The online logs on the primary and the standby logs on the standby will be archived 3) eventually. So far that is the same for both physical and logical standby. But now a difference: Logical standby databases do SQL Apply 4) by logmining the standby or the archive logs that came from the primary. That generates similar DML on the standby which in turn leads LGWR there to write redo into online logs 5) that will eventually get archived 6) as well.
A logical standby could do recovery only with its own archive logs (if there was a backup taken before) but not with the foreign archive logs. Therefore, those foreign archive logs can and do get deleted automatically. V$ARCHIVED_LOG and V$FOREIGN_ARCHIVED_LOG can be queried to monitor the two different kinds of logs.
That was one topic of the course Oracle Database 12c: Data Guard Administration that I’m delivering as an LVC this week, by the way. Hope you find it useful :-)
Tagged: Data Guard, High Availability
As every year, there’s a long list of great speakers with interesting talks to attend at the DOAG (German Oracle User Group) annual conference. Sadly I cannot attend them all, so I’ve got to make a choice:First day
Datenbank-Upgrade nach Oracle 184.108.40.206 – Aufwand, Vorgehen, Kunden by Mike Dietrich, Oracle
Die unheimliche Begegnung der dritten Art: XML DB für den DBA by Carsten Czarski, Oracle
Advanced RAC Programming Features by Martin Bach, Enkitec
Automatische Daten Optimierung, Heatmap und Compression 12c live by Ulrike Schwinn, OracleSecond day
Understanding Oracle RAC Internals The Cache Fusion Edition by Markus Michalewicz, Oracle
Die Recovery Area: Warum ihre Verwendung empfohlen ist – I have to go to that one because I present it myself :-)
Geodistributed Oracle GoldenGate and Oracle Active Data Guard: Global Data Services by Larry Carpenter, Oracle
Oracle Database In-Memory – a game changer for data warehousing? by Hermann Baer & Maria Colgan, Oracle
Oracle Distributed Transactions by Joel Goodman, OracleThird day
High Noon – Bessere Überlebenschancen beim Datenbank Security Shoot Out by Heinz-Wilhelm Fabry, Oracle
Tuning Tools für echte Männer und Sparfüchse – vom Leben ohne EM12c by Björn Rost, portrix Systems
Best Practices in Managing Oracle RAC Performance in Real Time by Mark Scardina, Oracle
Maximum Availability with Oracle Multitenant: Seeing Is Believing by Larry Carpenter, Oracle
Here’s a collection of customer quotes as a follow-up to my last post about the importance of attitude towards Live Virtual Classes (LVCs). They are from courses that I have taught personally this year with an average delivery score of about 96%:
Always important to make a good first impression!“The whole LVC package just worked. From the comfort of my own environment with a great instructor makes for happy learning :)”
And that is exactly what we strive to deliver.“Both, host and producer were very professional and guided the students through the course.”
An LVC producer takes care for all technical aspects apart from the course itself, like access to the learning platform. The instructor appears as “host” on the learning platform.“Instructor professionally answered students’ questions and kept up a positive mood in the community!”
LVCs can be funny too :-)“I appreciate the way how the course was presented. Very well controlled time, organization of presentation, exercises. Interaction with us was great. Always ready to answer a question, give an examples to difficult topic, illustrating topics.”
So much about allegedly missing interaction in LVCs.“I work few years on RAC databases, my knowledge was not so clear regarding some topic on RAC and Grid after completing this training I’m sure that I will handle our RAC and Grid environment differently and for sure will have positive impact in our production environment. Great thank!”
You cannot top that with a classroom course either :-)“LVC is offering great and flexible way to gain knowledge without travel or hotels etc.” “LVCs reduce travel costs and help the students to manage their time on their own, i.e. to join the classes from home and focus on the presented content.”
Trust me, I didn’t make up the last two although they may sound like your manager talking – or mine, for that matter ;-)
Tagged: Digital Learning, LVC
When using the Data Guard Broker, you don’t need to set any LOG_ARCHIVE_* parameter for the databases that are part of your Data Guard configuration. The broker is doing that for you. Forget about what you may have heard about VALID_FOR – you don’t need that with the broker. Actually, setting any of the LOG_ARCHIVE_* parameters with an enabled broker configuration might even confuse the broker and lead to warning or error messages. Let’s look at a typical example about the redo log transport mode. There is a broker configuration enabled with one primary database prima and one physical standby physt. The broker config files are mirrored on each site and spfiles are in use that the broker (the DMON background process, to be precise) can access:
When connecting to the broker, you should always connect to a DMON running on the primary site. The only exception from this rule is when you want to do a failover: That must be done connected to the standby site. I will now change the redo log transport mode to sync for the standby database. It helps when you think of the log transport mode as an attribute (respectively a property) of a certain database in your configuration, because that is how the broker sees it also.
[oracle@uhesse1 ~]$ dgmgrl sys/oracle@prima DGMGRL for Linux: Version 220.127.116.11.0 - 64bit Production Copyright (c) 2000, 2009, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> edit database physt set property logxptmode=sync; Property "logxptmode" updated
In this case, physt is a standby database that is receiving redo from primary database prima, which is why the LOG_ARCHIVE_DEST_2 parameter of that primary was changed accordingly:
[oracle@uhesse1 ~]$ sqlplus sys/oracle@prima as sysdba SQL*Plus: Release 18.104.22.168.0 Production on Tue Sep 30 17:21:41 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 22.214.171.124.0 - 64bit Production With the Partitioning, OLAP and Data Mining options SQL> show parameter log_archive_dest_2 NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ log_archive_dest_2 string service="physt", LGWR SYNC AFF IRM delay=0 optional compressi on=disable max_failure=0 max_c onnections=1 reopen=300 db_uni que_name="physt" net_timeout=3 0, valid_for=(all_logfiles,pri mary_role)
The mirrored broker configuration files on all involved database servers contain that logxptmode property now. There is no new entry in the spfile of physt required. The present configuration allows now to raise the protection mode:
DGMGRL> edit configuration set protection mode as maxavailability; Succeeded.
The next broker command is done to support a switchover later on while keeping the higher protection mode:
DGMGRL> edit database prima set property logxptmode=sync; Property "logxptmode" updated
Notice that this doesn’t lead to any spfile entry; only the broker config files store that new property. In case of a switchover, prima will then receive redo with sync.
DGMGRL> show configuration; Configuration - myconf Protection Mode: MaxAvailability Databases: prima - Primary database physt - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS DGMGRL> switchover to physt; Performing switchover NOW, please wait... New primary database "physt" is opening... Operation requires shutdown of instance "prima" on database "prima" Shutting down instance "prima"... ORACLE instance shut down. Operation requires startup of instance "prima" on database "prima" Starting instance "prima"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "physt"
All I did was the switchover command, and without me specifying any LOG_ARCHIVE* parameter, the broker did it all like this picture shows:
[oracle@uhesse2 ~]$ sqlplus sys/oracle@physt as sysdba SQL*Plus: Release 126.96.36.199.0 Production on Tue Oct 14 15:43:41 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 188.8.131.52.0 - 64bit Production With the Partitioning, OLAP and Data Mining options SQL> show parameter log_archive_dest_2 NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ log_archive_dest_2 string service="prima", LGWR SYNC AFF IRM delay=0 optional compressi on=disable max_failure=0 max_c onnections=1 reopen=300 db_uni que_name="prima" net_timeout=3 0, valid_for=(all_logfiles,pri mary_role)
Not only is it not necessary to specify any of the LOG_ARCHIVE* parameters, it is actually a bad idea to do so. The guideline here is: Let the broker control them! Else it will at least complain about it with warning messages. So as an example what you should not do:
[oracle@uhesse1 ~]$ sqlplus sys/oracle@prima as sysdba SQL*Plus: Release 184.108.40.206.0 Production on Tue Oct 14 15:57:11 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 220.127.116.11.0 - 64bit Production With the Partitioning, OLAP and Data Mining options SQL> alter system set log_archive_trace=4096; System altered.
Although that is the correct syntax, the broker now gets confused, because that parameter setting is not in line with what is in the broker config files. Accordingly that triggers a warning:
DGMGRL> show configuration; Configuration - myconf Protection Mode: MaxAvailability Databases: physt - Primary database prima - Physical standby database Warning: ORA-16792: configurable property value is inconsistent with database setting Fast-Start Failover: DISABLED Configuration Status: WARNING DGMGRL> show database prima statusreport; STATUS REPORT INSTANCE_NAME SEVERITY ERROR_TEXT prima WARNING ORA-16714: the value of property LogArchiveTrace is inconsistent with the database setting
In order to resolve that inconsistency, I will do it also with a broker command – which is what I should have done instead of the alter system command in the first place:
DGMGRL> edit database prima set property LogArchiveTrace=4096; Property "logarchivetrace" updated DGMGRL> show configuration; Configuration - myconf Protection Mode: MaxAvailability Databases: physt - Primary database prima - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS
Thanks to a question from Noons (I really appreciate comments!), let me add the complete list of initialization parameters that the broker is supposed to control. Most but not all is LOG_ARCHIVE*
Tagged: Data Guard, High Availability
The single most important factor for successful digital learning is the attitude both of the instructor as well as of the attendees towards the course format. Delivery of countless Live Virtual Classes (LVCs) for Oracle University made me realize that. There are technical prerequisites of course: A reliable and fast network connection and the usage of a good headset is mandatory, else the participant is doomed from the start. Other prerequisites are the same as for traditional courses: Good course material, working lab environment for hands on practices and last not least knowledgeable instructors. For that part notice that we have the very same courseware, lab environments and instructors like for our classroom courses at Oracle University education centers also for LVCs. The major difference is in your head :-)
Delivering my first couple of LVCs, I felt quite uncomfortable with that new format. Accordingly, my performance was not as good as usual. Meanwhile, I consider the LVC format as totally adequate for my courses and that attitude enables me to deliver them with the same performance as my classroom courses. Actually, they are even better to some degree: I always struggle producing clean sketches with readable handwriting on the whiteboard. Now look at this MS paint sketch from one of my Data Guard LVCs:
Attendees get all my sketches per email if they like afterwards.
In short: Because I’m happy delivering through LVC today, I’m now able to do it with high quality. The attitude defines the outcome.
Did you ever have a teacher in school that you just disliked for some reason? It was hard to learn anything from that teacher, right? Even if that person was competent.
So this is also true on the side of the attendee: The attitude defines the outcome. If you take an LVC thinking “This cannot work!”, chances are that you are right just because of your mindset. When you attend an LVC with an open mind – even after some initial trouble because you need to familiarize yourself with the learning platform and the way things are presented there – it is much more likely that you will benefit from it. You may even like it better than classroom courses because you can attend from home without the time and expenses it takes to travel :-)
Some common objections against LVC I have heard from customers and my usual responses:
An LVC doesn’t deliver the same amount of interaction like a classroom course!
That is not necessarily so: You are in a small group (mostly less than 10) that is constantly in an audio conference. Unmute yourself and say anything you like, just like in a classroom. Additionally, you have a chatbox available. This is sometimes extremely helpful, especially with non-native speakers in the class :-) You can easily exchange email addresses using the chatbox as well and stay in touch even after the LVC.
I have no appropriate working place to attend an LVC!
You have no appropriate working place at all, then, for something that requires a certain amount of concentration. Talk to your manager about it – maybe there is something like a quiet room available during the LVC.
I cannot keep up the attention when starring the whole day on the computer screen!
Of course not, that is why we have breaks and practices in between the lessons.
Finally, I would love to hear about your thoughts and experiences with online courses! What is your attitude towards Digital Learning?
Tagged: Digital Learning, LVC