Yann Neuhaus

Subscribe to Yann Neuhaus feed
dbi services technical blog
Updated: 12 hours 53 min ago

Oracle 12c – Difference between ‘%F’ #default and ‘%F’

Mon, 2016-10-31 07:14

Do you see any differences between these two RMAN SHOW commands?

RMAN> SHOW CONTROLFILE AUTOBACKUP FORMAT;

RMAN configuration parameters for database with db_unique_name OCM121 are:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default

and

RMAN> SHOW CONTROLFILE AUTOBACKUP FORMAT;

RMAN configuration parameters for database with db_unique_name OCM121 are:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';

 

They are the same, except that the first output has the “# default” at the end and the second one doesn’t.

Whenever you see a “# default” at the end of a RMAN show command, it simply means that this is the default value which comes out of the box, and even if you change it to the same value, the “# default” at the end disappears.

Sometimes, Oracle is funny. :-) The strange thing about it, is that ‘%F’ #default and ‘%F’ are not the same. Due to the Oracle documentation, the format specification for ‘%F’ means the following:

‘%F’ combines the DBID, day, month, year, and sequence into a unique and repeatable generated name.
This variable translates into c-IIIIIIIIII-YYYYMMDD-QQ, where:

  • IIIIIIIIII stands for the DBID. The DBID is printed in decimal so that it can be easily associated with the target database.
  • YYYYMMDD is a time stamp in the Gregorian calendar of the day the backup is generated
  • QQ is the sequence in hexadecimal number that starts with 00 and has a maximum of ‘FF’ (256)

HINT: %F is valid only in the CONFIGURE CONTROLFILE AUTOBACKUP FORMAT command.

Ok. Let’s take a look into the ASM directory, where my controlfile autobackups are located.

ASMCMD> pwd
+fra/ocm121/autobackup/2016_10_29
ASMCMD> ls -l
Type        Redund  Striped  Time             Sys  Name
AUTOBACKUP  UNPROT  COARSE   OCT 29 11:00:00  Y    s_926509179.514.926509183
AUTOBACKUP  UNPROT  COARSE   OCT 29 12:00:00  Y    s_926511850.517.926511853

Regarding the format specification ‘%F’, I see no DBID, no time stamp and also no sequence number. Looks like, that the format specification is not valid for the autobackups located in ASM.

So … with the RMAN Default (‘%F'; # default), RMAN will send the autobackup to the flash recovery area, if it is used, and in my case, it sends it to ASM. However, the format specification has nothing to do with the ‘%F’ described in the documentation. I do have something totally different.

Ok. Let’s change the controlfile autobackup format to exactly the same thing like it was before and check what happens. Afterwards, lets quickly force a controlfile autobackup creation.

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';

new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';
new RMAN configuration parameters are successfully stored

SQL> ALTER SYSTEM SET "_controlfile_autobackup_delay"=0;

System altered.

SQL> create tablespace DBI datafile size 16M;

Tablespace created.

A new controlfile should have been created right away, but in the ASM directory we see no new controlfile autobackup.

ASMCMD> ls -l
Type        Redund  Striped  Time             Sys  Name
AUTOBACKUP  UNPROT  COARSE   OCT 29 11:00:00  Y    s_926509179.514.926509183
AUTOBACKUP  UNPROT  COARSE   OCT 29 12:00:00  Y    s_926511850.517.926511853

Where did Oracle put my new controlfile autobackup? I just changed the same value with the same value. After a little bit of looking around, I have noticed that a new controlfile autobackup was created in the good old $ORACLE_HOME/dbs/

oracle@oel001:/u00/app/oracle/product/12.1.0.2/dbs/ [OCM121] ls -rtl c-*
-rw-r----- 1 oracle asmadmin 10387456 Oct 31 10:03 c-3827054096-20161031-00

And furthermore, the format specification looks now exactly, how it is documented, incl. the DBID, the time stamp
and so on.

So, with ‘%F’ non-default, Oracle will put it in $ORACLE_HOME/dbs on UNIX and to %ORACLE_HOME%\Database in case you are on Windows. In the end, the following configurations look the same, but they are totally different.

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F';

Now you could say, I don’t care if Oracle puts it into ASM or into $ORACLE_HOME/dbs. In case you are running RAC you should care, because it makes a major difference. If one host crashes, you can’t access your controlfile autobackups anymore. And what about crosschecks? If you run now a crosscheck from host B, it does not see the controlfile autobackups from host A, and after a while you will end up with a big mess.

Conclusion. Sometimes, the same value is not the same value. ;-) And for RAC and RAC One Node, better stay with the default, which is ‘%F'; #default.
Cheers,
William

 

Cet article Oracle 12c – Difference between ‘%F’ #default and ‘%F’ est apparu en premier sur Blog dbi services.

Oracle 12c – Automatic Control File Backups

Sat, 2016-10-29 05:29

By default, automatic control file backups are disabled (even with 12c), maybe for performance reasons.

RMAN> SHOW CONTROLFILE AUTOBACKUP;

RMAN configuration parameters for database with db_unique_name OCM121 are:
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default

And also good to know, the autobackup after structural changes does not occur for databases in NOARCHIVELOG mode. So, if your database is running in NOARCHIVELOG mode, you will never see any impact, independent of controlfile autobackup is on or off.
However, my database is running in ARCHIVELOG mode and at the moment the controlfile autobackup feature is disabled.
But even when the auto backup feature is disabled, RMAN will still back up the current controlfile and the server parameter file whenever any backup command includes datafile 1 from the data files that belong to the target database. In an Oracle database, data file 1 is always part of the system tablespace, which contains the data dictionary.

But I have heard that it is highly recommended to enable automatic controlfile backups. It will ensure that the critical controlfile is backed up regularly following a backup or structural change to the database. Once you configure automatic controlfile backup, RMAN will automatically back up your target database controlfile, as well as the current server parameter file, when any of the following events occurs:

  • Successful completion of either a backup or the copy command
  • After a create catalog command from the RMAN prompt is successfully completed
  • Any structural changes to the database modify the contents of the control file

Any changes to the physical structure of your database, even if they are made through SQL*Plus, will trigger a controlfile auto backup, e.g.

  • adding a tablespace or data file
  • dropping a data file
  • placing a tablespace offline or online
  • adding an online redo log, and renaming a data file

So, to follow the recommendation, I will enable the automatic backup of the controlfile. At the moment I have no backup of the controlfile, which is not good at all.

RMAN>  list backup of controlfile;

specification does not match any backup in the repository

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;

new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored

But nothing happens. What is going on here. Is it a bug, a feature or something else.
Hold on. Then … all of a sudden, the backup of the controlfile popps up.

RMAN> list backup of controlfile;

List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
131     Full    9.89M      DISK        00:00:04     28-OCT-2016 15:42:23
        BP Key: 314   Status: AVAILABLE  Compressed: NO  Tag: TAG20161028T154219
        Piece Name: +FRA/OCM121/AUTOBACKUP/2016_10_28/s_926437339.526.926437343
  Control File Included: Ckp SCN: 10102765     Ckp time: 28-OCT-2016 15:42:19

 

To understand what is happening here, the answer is like always. It depends. :-) In case you are running an Oracle database with 11gR2 or higher, then it is a feature. Anything below would be a bug.

Beginning with Oracle 11gR2, the controlfile autobackup deferral feature has been implemented. In order to increase performance, the controlfile autobackup creation after structural changes, has been deferred. In previous releases, one controlfile autobackup is created with each DDL command that makes structural changes in the database and we can see in the alert.log a message about controlfile autobackup creation after each DDL command executed.
This can provoke serious performance problems when multiple structural changes are made together. Starting from Oracle Database Release 11g Release 2, RMAN takes only one control file autobackup when multiple structural changes contained in a script have been applied (for example, adding tablespaces, altering the state of a tablespace or datafile,
adding a new online redo log, renaming a file, and so on) during a specified time.

But what does time mean here exactly. It is 1 minute, 1 hour, or 1 day?

The deferral time is controlled by an underscore parameter that defaults to 300 seconds (5 minutes). The parameter is the following:

_controlfile_autobackup_delay=300

The minimum value for that parameter is 0, which simulates the behavior before 11gR2. The maximum value in 12c is (1024*1024*1024*2)-1, which is 2147483647 seconds. However, I don’t see any practical value for setting the value that high.

In 11gR2 or higher, the controlfile autobackups are created by MMON slaves after few minutes (5 minutes per default) of the structural changes, which increases performance. It is also expected that no message about controlfile autobackup creation will appear in the alert.log.
However, there will be one MMON slave trace file with the controlfile creation information, that will be a file named: SID__m000_<OS_PID>.trc
Ok. Let’s try to simulate the old behavior by setting the autobackup delay to 0, and to create another tablespace afterwards.

SQL> alter system set "_controlfile_autobackup_delay"=0;

System altered.

SQL> create tablespace unpatient2 datafile size 16M;

Tablespace created.

And now the controlfile autobackup is created immediately.

RMAN> list backup of controlfile;

using target database control file instead of recovery catalog

List of Backup Sets
===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
131     Full    9.89M      DISK        00:00:04     28-OCT-2016 15:42:23
        BP Key: 314   Status: AVAILABLE  Compressed: NO  Tag: TAG20161028T154219
        Piece Name: +FRA/OCM121/AUTOBACKUP/2016_10_28/s_926437339.526.926437343
  Control File Included: Ckp SCN: 10102765     Ckp time: 28-OCT-2016 15:42:19

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ --------------------
132     Full    9.89M      DISK        00:00:05     28-OCT-2016 17:41:16
        BP Key: 315   Status: AVAILABLE  Compressed: NO  Tag: TAG20161029T113939
        Piece Name: +FRA/OCM121/AUTOBACKUP/2016_10_29/s_926509179.514.926509183
  Control File Included: Ckp SCN: 10271426     Ckp time: 28-OCT-2016 17:41:11

So, do I recommend to set the RMAN controlfile autobackup to ON. Yes absolutely. And do I recommend to set the controlfile autobackup delay to 0. No, probably not. I think, that the 5-minute interval is a quite good compromise. You just need to be aware, that it exists.

Another hint is, that you should not rely too much on the view V$RMAN_BACKUP_JOB_DETAILS. In this view, the autobackups should be populated, whenever a autobackup happend. But in my case the AUTOBACKUP_DONE is always set to NO.

SQL> select start_time,end_time,status,autobackup_done, AUTOBACKUP_COUNT from
2 V$RMAN_BACKUP_JOB_DETAILS where autobackup_done = 'YES';

no rows selected

There is a patch available from Oracle: “Patch 18074513: V$RMAN_BACKUP_JOB_DETAILS VIEWS COLUMN AUTOBACKUP_DONE DOESNOT GET POPULATED”, but it is not available for every platform and every version.

Better use the RMAN “list backup of controlfile;” command. That one is much more reliable.

Cheers,
William

 

 

 

 

 

Cet article Oracle 12c – Automatic Control File Backups est apparu en premier sur Blog dbi services.

Oracle 12c – When the RMAN Dummy Instance does not start up …

Fri, 2016-10-28 06:53

Not too often, but sometimes you might run into a situation when you lose everything, your DB Files, your Controlfiles and even your spfile. In situations like that, you need to restore first your spfile, then your controlfile and then the rest.

For restoring the spfile, RMAN has a chicken/egg issue. To be able to restore the spfile, RMAN needs at least a running instance, but how do we start the instance without having the spfile? There are several methods to do it, one of it, is to let RMAN itself create a dummy instance.

But if you think your situation can’t get worse than the following happens. Oracle raises an ORA-04031 error and even the Dummy Instance does not start.

oracle@oel001:/home/oracle/ [rdbms121] export ORACLE_SID=LAX
oracle@oel001:/home/oracle/ [LAX] rman target /

Recovery Manager: Release 12.1.0.2.0 - Production on Fri Oct 28 11:45:08 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup nomount

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u00/app/oracle/product/12.1.0.2/dbs/initLAX.ora'

starting Oracle instance without parameter file for retrieval of spfile
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 10/28/2016 11:45:15
RMAN-04014: startup failed: ORA-04031: unable to allocate 111264 bytes of shared memory 
("shared pool","unknown object","sga heap(1,0)","KEWS sesstat values")

 

The error message means, that the SGA allocated for the dummy instance is to small. This is where the environment variable ORA_RMAN_SGA_TARGET comes into play. The environment variable ORA_RMAN_SGA_TARGET sets the SGA to a value in Megabytes which can be used by RMAN to start the Dummy Instance. In the following example to 1024 MB.

oracle@oel001:/home/oracle/ [LAX] export ORA_RMAN_SGA_TARGET=1024
oracle@oel001:/home/oracle/ [LAX]

oracle@oel001:/home/oracle/ [LAX] rman target /

Recovery Manager: Release 12.1.0.2.0 - Production on Fri Oct 28 13:12:13 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup nomount

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u00/app/oracle/product/12.1.0.2/dbs/initLAX.ora'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area    1073741824 bytes

Fixed Size                     2932632 bytes
Variable Size                293601384 bytes
Database Buffers             771751936 bytes
Redo Buffers                   5455872 bytes

RMAN>

Behind the scenes, Oracle starts a Dummy Instance with the following parameters, and creates all necessary directories in the DIAG destination.

sga_target               = 1G
compatible               = "12.1.0.2.0"
_dummy_instance          = TRUE
remote_login_passwordfile= "EXCLUSIVE"
_diag_adr_trace_dest='/u00/app/oracle/diag/rdbms/dummy/LAX/trace'
core_dump_dest='/u00/app/oracle/diag/rdbms/dummy/LAX/cdump'
db_name='DUMMY' 
  
oracle@oel001:/u00/app/oracle/diag/rdbms/dummy/LAX/ [LAX] pwd
/u00/app/oracle/diag/rdbms/dummy/LAX
oracle@oel001:/u00/app/oracle/diag/rdbms/dummy/LAX/ [LAX] ls
alert  cdump  hm  incident  incpkg  ir  lck  log  metadata  metadata_dgif  metadata_pv  stage  sweep  trace

 

Now it worked and I can retrieve my spfile.

run {
restore spfile to pfile '/tmp/initLAX.ora' for db_unique_name='LAX' from
'+fra/lax/autobackup/2016_10_28/s_894893708.1292.894893713';
}

From now on, I can use the correct parameter file to continue with the other steps. Another option would have been, to search through your alert.log for “System parameters with non-default values”. Whenever you startup your instance, Oracle dumps out the non-default parameter values into the alert.log. Those values can be used to manually create an init.ora file and then the spfile. The drawback of using the values from alert.log is, that the values might be very old. In case the instance was not bounced for several months or longer (not so unusual), then you miss all the new setting since then.

...
...
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =51
LICENSE_MAX_USERS = 0
SYS auditing is enabled
NOTE: remote asm mode is local (mode 0x1; from cluster type)
NOTE: Using default ASM root directory ASM
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options.
ORACLE_HOME = /u00/app/oracle/product/12.1.0.2
System name:    Linux
Node name:      oel001
Release:        2.6.32-642.6.1.el6.x86_64
Version:        #1 SMP Tue Oct 4 15:19:03 PDT 2016
Machine:        x86_64
System parameters with non-default values:
  processes                = 300
  _disable_highres_ticks   = TRUE
  event                    = "10720 trace name context forever, level 0x10000000"
  event                    = "10795 trace name context forever, level 2"
  sga_max_size             = 1536M
  use_large_pages          = "ONLY"
  shared_pool_size         = 256M
  _high_priority_processes = "LGWR"
  _highest_priority_processes= "LGWR"
  filesystemio_options     = "SETALL"
  sga_target               = 1536M
  control_files            = "+DATA/LAX/CONTROLFILE/current.265.918392661"
  control_files            = "+FRA/LAX/CONTROLFILE/current.256.918392661"
  control_file_record_keep_time= 32
  db_block_size            = 8192
  compatible               = "12.1.0.2.0"
  log_archive_format       = "%t_%s_%r.dbf"
  db_create_file_dest      = "+DATA"
  db_create_online_log_dest_1= "+DATA"
  db_recovery_file_dest    = "+FRA"
  db_recovery_file_dest_size= 32G
  undo_tablespace          = "UNDOTBS1"
  undo_retention           = 3600
  db_securefile            = "PERMITTED"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = ""
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=LAXXDB)"
  local_listener           = "LAX_LISTENER"
  session_cached_cursors   = 512
  parallel_max_servers     = 80
  audit_file_dest          = "/u00/app/oracle/admin/LAX/adump"
  audit_trail              = "DB"
  cell_offload_processing  = FALSE
  db_name                  = "LAX"
  open_cursors             = 300
  pga_aggregate_target     = 512M
  _disable_directory_link_check = TRUE
  diagnostic_dest          = "/u00/app/oracle"
...
...

In the end, you have different possibilities to restore the spfile. Either with the RMAN Dummy Instance, or via the alert.log. In case of the Dummy Instance, you might need to play around with the ORA_RMAN_SGA_TARGET environment variable.

By the way … with Oracle 10.2.0.5 your chance to hit the ORA-04031 error during the Dummy Instance startup was much higher, because the default was only 152MB.

oracle@oel001:/home/oracle/ [LAX] rman target /

Recovery Manager: Release 10.2.0.5.0 - Production on Fri Oct 28 11:26:09 2016

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database (not started)

RMAN> startup nomount

startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u00/app/oracle/product/10.2.0.5/dbs/initLAX.ora'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area     159383552 bytes

Fixed Size                     2094736 bytes
Variable Size                 67111280 bytes
Database Buffers              83886080 bytes
Redo Buffers                   6291456 bytes

RMAN>

 

Cheers,
William

 

 

Cet article Oracle 12c – When the RMAN Dummy Instance does not start up … est apparu en premier sur Blog dbi services.

Documentum story – SSL Password for the JKS of the DSearch & IndexAgent (xPlore)

Fri, 2016-10-28 02:00

In a previous blog (click here), I described how to setup the DSearch and IndexAgent(s) in HTTPS using the Groovy script provided by EMC in newer versions of Documentum. This script is quite cool because it will allow you to automatically do some stuff like updating the xml configuration files, put the java keystore in the right location and configure JBoss to use it, aso… It also allows you to quickly apply/revert changes to do some tests for example. Several months ago when I first used this script, I faced some issues. I will present in this blog some of the errors I’ve seen and what was the reason behind that.

 

Actually I have a “funny” story related to these errors: several months after seeing these errors for the first time (and telling EMC about it with complete description of the issue and according solution), I worked on another new issue related to the Full Text Server with the help of EMC. Our EMC contact was trying to replicate our new issue and for that purpose, he had to setup the IndexAgent in SSL and he faced an error with the SSL setup… So he opened a ticket with the EMC Engineering Team to ask them to check why this was happening. In the meantime, I also asked him to share the error he was facing with me because I was curious… Of course, it turned out to be one of the errors I saw and shared with EMC several months before that. Therefore I provided him the solution to correct this bug and he was able to install the IndexAgent in SSL. I found that funny how sometimes you can actually be the one helping EMC with their own products ^^.

 

So let’s start with some of the errors I saw:

1. While configuring a DSearch in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IS \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -isname PrimaryDsearch

Component: IS
====== Configure Index Server: PrimaryDsearch ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/my.keystore
Copy: /app/xPlore/config/indexserverconfig.xml to: /app/xPlore/config/indexserverconfig.xml.bakHttp
Primary instance: PrimaryDsearch
Host name: xplore_server_01
Http port: 9300, https port: 9302
Updated: /app/xPlore/config/indexserverconfig.xml
Copy: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIS(ConfigSSL.groovy:624)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL$_run_closure1.doCall(ConfigSSL.groovy:333)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.call(Closure.java:425)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1376)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1348)
        at org.codehaus.groovy.runtime.dgm$162.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at ConfigSSL.run(ConfigSSL.groovy:331)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

2. While configuring an IndexAgent in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIA(ConfigSSL.groovy:888)
        at ConfigSSL$configIA.callCurrent(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL.run(ConfigSSL.groovy:338)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

3. While configuring an IndexAgent in HTTPS: another one

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass Test4Pass+word$ -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200
    
Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference
      at java.util.regex.Matcher.appendReplacement(Matcher.java:808)
      at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
      at java.lang.String.replaceFirst(String.java:2119)
      at java_lang_String$replaceFirst.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
      at ConfigSSL.configIA(ConfigSSL.groovy:888)
      at ConfigSSL$configIA.callCurrent(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      at ConfigSSL.run(ConfigSSL.groovy:338)
      at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
      at groovy.lang.GroovyShell.run(GroovyShell.java:229)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

4. The solution?

You might have noticed above that each time the exception is thrown just after the backup of the file standalone.xml. So what is the Groovy script doing in this file exactly which can cause these errors? Well when configuring JBoss in HTTPS, there is only one thing to do in this file: add a new connector to enable the HTTPS and to point to the correct keystore/password. Based on this information, it is really easy to find the reason for all these exceptions. In addition to that, you might also have noticed that in the two first points above, the password of the keystore was “$JKS_Password” while in the last command, the password is in clear text (Test4Pass+word$). So what is that environment variable $JKS_Password? Well we had to use IQs to install all these environments with logging tools to review and validate the installation and since these logging tools are storing everything displayed to the console, we were using environment variable to store our password so it is not displayed on the console or on the history of the “xplore” user. This is done as follow:

[xplore@xplore_server_01 ~]$ stty -echo; read JKS_Password; stty echo
                                                         => Enter the password here, it will be invisible
[xplore@xplore_server_01 ~]$
[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word$
[xplore@xplore_server_01 ~]$

 

Basically the first command will ask you to type the password and when you press Enter, the prompt is returned and the environment variable $JKS_Password contains your password.

 

Now that this is clear, what is the solution to correct the errors mentioned above? Well the solution is the same for all errors above and this solution is quite simple: don’t use any annoying special characters in your passwords (like “$” or “\”). The problem with these characters is that the Groovy script isn’t able to manage them at the moment and therefore creating a password that contains one of these characters might result is errors and/or strange results as shown above.

 

Even if annoying special characters should be avoided to prevent these errors, you can continue to use some other special characters if you want to use a strong password like “+”, “-“, “?” and you can also continue to use the environment variable if you want. This is an example of the command that is working:

[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word
[xplore@xplore_server_01 ~]$ 
[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml
Copy: ./../../watchdog/config/dsearch-watchdog-config.xml to: ./../../watchdog/config/dsearch-watchdog-config.xml.bakHttp
Updated: ./../../watchdog/config/dsearch-watchdog-config.xml
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
SSL is enabled, clear jboss cache and start server to verify result.

 

Once this execution completed successfully, you can just continue the SSL setup as described in this blog linked at the top and you shouldn’t face any other issue.

 

As soon as you put a “$” in your password and use this password in the command line (or use the environment variable), you will in the end face an error that will depend on whether or not you used the environment variable to hide it. As an additional note, when I was working on these errors, I did a lot of tests and I already saw the script working (or so it seemed) even with a password containing a “$” (depending on how it is written, with single quotes, double quotes, escaped, aso…) when put directly in the command line. But when I tried to start the IndexAgent, it wasn’t able to start… Therefore I took a look at the file standalone.xml and it was corrupted! The Groovy script screwed up my standalone.xml file and I had to revert a backup to be able to restart it. Hopefully the script is smart enough to backup all files before doing anything…

 

In case you absolutely want a “$” in your password, then it is possible but you will have to do one step manually. First you need to execute the script with a DUMMY password without any special character:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass DUMMY_PASSWORD -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200
...

 

Once done, you can simply open the file standalone.xml and replace “DUMMY_PASSWORD” with the real password that contains a “$”. Restart the IndexAgent and this time, it should work. :)

 

Cet article Documentum story – SSL Password for the JKS of the DSearch & IndexAgent (xPlore) est apparu en premier sur Blog dbi services.

Oracle 12c – When did my row change?

Fri, 2016-10-28 01:51

My good old DEPT table always had 4 rows with 4 different departments. However, I have noticed that a new row was inserted into the DEPT table and the row 50 popped up, and I would like to know when it happened?

Before:

SQL> select  DEPTNO, DNAME, LOC from DEPT;

    DEPTNO DNAME          LOC
---------- -------------- -------------
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 OPERATIONS     BOSTON

After:

SQL> select  DEPTNO, DNAME, LOC from DEPT;

    DEPTNO DNAME          LOC
---------- -------------- -------------
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 OPERATIONS     BOSTON
        50 IT             L.A.

Unfortunately, I have no auditing in place, and supplemental logging is also not activated. So, auditing is not an option and LogMiner also not. I’m kind of running out of options to get the timestamp when the department 50 was inserted.

My last resort is the ORA_ROWSCN pseudocolumn. Due to the Oracle documentation, for each row, ORA_ROWSCN returns the conservative upper bound system change number (SCN) of the most recent change to the row. This pseudocolumn becomes useful, in case we want to determine approximately when a row was last updated.

ORA_ROWSCN is a pseudocolumn of any table that is not fixed or external, and be careful, it is not 100% precise, because Oracle tracks SCNs by transaction committed for the block in which the row resides. However, it is better then nothing for my use case.

SQL> select ora_rowscn, DEPTNO, DNAME, LOC from DEPT;

ORA_ROWSCN     DEPTNO DNAME          LOC
---------- ---------- -------------- -------------
   9708226         10 ACCOUNTING     NEW YORK
   9708226         20 RESEARCH       DALLAS
   9708226         30 SALES          CHICAGO
   9708226         40 OPERATIONS     BOSTON
   9708226         50 IT             L.A.

To convert now the SCN to a Timestamp, we can use the SCN_TO_TIMESTAMP function. It converts the SCN to a Timestamp with a precision of +/- 3 seconds.

SQL> select scn_to_timestamp(9708226) as timestamp from dual;

TIMESTAMP
---------------------------------------------------------------------------
27-OCT-16 11.40.07.000000000 AM

Due to the ORA_ROWSCN pseudocolumn, now I know at least, that the block where the department 50 was inserted was changed at the “27-OCT-16 11.40.07″.

If you want it more accurate, you need a feature called Row Level Dependency Tracking. This feature is activated by the keyword ROWDEPENDENCIES during the CREATE TABLE.

CREATE TABLE "SCOTT"."DEPT_RLDT" 
   (	"DEPTNO" NUMBER(2,0), 
	"DNAME" VARCHAR2(14 BYTE), 
	"LOC" VARCHAR2(13 BYTE), 
	 CONSTRAINT "PK_DEPT_RLDT" PRIMARY KEY ("DEPTNO")
  USING INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS"  ENABLE
   ) SEGMENT CREATION IMMEDIATE 
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ROWDEPENDENCIES ;

With this feature, each row in the table has a system change number (SCN) that represents a time greater than or equal to the commit time of the last transaction that modified the row.

You cannot change this setting after the table is created. Meaning, there is no “alter table … ROWDEPENDENCIES”.

This setting is useful primarily to allow for parallel propagation in replication environments, however, it can also be used to find out quickly the time, when a row was change.

Like always in life, nothing comes for free. The drawback is, that it increases the size of each row by 6 bytes. That’s why the default is NOROWDEPENDENCIES.

But once this feature is enable, like in the following example for my DEPT_RLDT table, you will get
a different SCN number for the new commited row (department 50).

SQL> select ora_rowscn, DEPTNO, DNAME, LOC from DEPT_RLDT;

ORA_ROWSCN     DEPTNO DNAME          LOC
---------- ---------- -------------- -------------
   9701315         10 ACCOUNTING     NEW YORK
   9701315         20 RESEARCH       DALLAS
   9701315         30 SALES          CHICAGO
   9701315         40 OPERATIONS     BOSTON
   9708244         50 IT             L.A.

SQL>

SCN 9701315 for deptno 10,20,30,40 and 9708244 for deptno 50. That’s cool.

SQL> select scn_to_timestamp(9708244) as timestamp from dual;

TIMESTAMP
---------------------------------------------------------------------------
27-OCT-16 11.40.24.000000000 AM

Due to the SCN to a Timestamp precision of +/- 3 seconds, it is still not 100%, but very close to it. Another important point is that the ORA_ROWSCN represents an upper bound SCN. We just know it has not changed after this SCN but may have been changed earlier.

So … does the ORA_ROWSCN replace auditing? No, not at all. But it can help to answer questions like the following: Please tell me when ROW xyz has been modified the last time and recover the table (using Flashback techniques) to that point.
Cheers,
William

 

Cet article Oracle 12c – When did my row change? est apparu en premier sur Blog dbi services.

12c GTT private statistics and cursor invalidation

Thu, 2016-10-27 15:27

Short summary of this post is that rolling invalidation do not occur when you gather statistics on Global Temporary Tables in 12c that have session statistic scope (which is the default) and this may cause too many hard parses. I’m sharing all details and comments are welcome.

When you gather statistics on a table, the goal is to get new plan if statistics have changed, so you can expect cursor invalidation. However, invalidating immediately all cursors that have a dependency with the table may cause a hard parse storm and this is why by default rolling invalidation occurs: invalidation of cursor will be planned randomly in a time window that follows next execution. 12c comes with a new feature, global temporary table private stats where execution plans are not shared between sessions. And there’s another feature where statistics gathering is automatic when you bulk insert into an empty table.

In both cases, by default, invalidation is not rolling but immediate. Let’s see examples.

No GTT -> rolling invalidation

Here is an example with a regular table to show rolling invalidation:

21:14:36 SQL> create
21:14:36 2 table DEMOGTT1 as select * from dual;
Table created.
 
21:14:38 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1');
PL/SQL procedure successfully completed.
 
21:14:39 SQL> insert into DEMOGTT1 select * from dual;
1 row created.
 
21:14:39 SQL> alter session set optimizer_mode=first_rows;
Session altered.
 
21:14:39 SQL> insert into DEMOGTT1 select * from dual;
1 row created.
 
21:14:39 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000008368BB98 3223759815 1 1 N 0
VALID 1 000000008368BB98 3223759815 1 1 N 0
 
21:14:40 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1',no_invalidate=>null);
PL/SQL procedure successfully completed.
 
21:14:41 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000008368BB98 3223759815 1 1 Y 0
VALID 1 000000008368BB98 3223759815 1 1 Y 0
 
21:14:41 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED 21:14:40

Statistics on non-GTT are shared and the dbms_stats using default no_invalidate do rolling invalidation.

GTT with session private stats -> immediate invalidation

Here is the same example with a Global Temporary Table:

21:13:06 SQL> create
21:13:06 2 global temporary
21:13:06 3 table DEMOGTT1 as select * from dual;
Table created.
...
21:13:09 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000008096DF10 3223759815 1 1 N 0
VALID 1 000000008096DF10 3223759815 1 1 N 0
 
21:13:10 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1',no_invalidate=>null);
PL/SQL procedure successfully completed.
 
21:13:11 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
INVALID_UNAUTH 0 000000008096DF10 3223759815 1 1 N 1
INVALID_UNAUTH 1 000000008096DF10 3223759815 1 1 N 1
 
21:13:11 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED
DEMOGTT1 SESSION 21:13:10

By default, gathered statistics are private to the session and you see that all cursors have been invalidated immediately. Next execution will need to hard parse.

GTT with shared stats -> no invalidation

When setting shared statistics on the GTT we come back to the 11g behavior:

21:28:52 SQL> create
21:28:52 2 global temporary
21:28:52 3 table DEMOGTT1 as select * from dual;
Table created.
 
21:28:52 SQL> exec dbms_stats.set_table_prefs(user,'DEMOGTT1','GLOBAL_TEMP_TABLE_STATS','SHARED');
PL/SQL procedure successfully completed.
...
21:28:55 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 0000000079782A08 3223759815 1 1 N 0
VALID 1 0000000079782A08 3223759815 1 1 N 0
 
21:28:56 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1',no_invalidate=>null);
PL/SQL procedure successfully completed.
 
21:28:57 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 0000000079782A08 3223759815 1 1 Y 0
VALID 1 0000000079782A08 3223759815 1 1 Y 0
 
21:28:57 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED 21:28:56

No invalidation: this is rolling invalidation

GTT with session stats but “_optimizer_use_gtt_session_stats”=false

Here is an exemple when disabling the private statistics feature:

21:15:36 SQL> create
21:15:36 2 global temporary
21:15:36 3 table DEMOGTT1 as select * from dual;
Table created.
 
21:15:36 SQL> alter session set "_optimizer_use_gtt_session_stats"=false;
Session altered.
...
21:15:38 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000007A373A08 3223759815 1 1 N 0
VALID 1 000000007A373A08 3223759815 1 1 N 0
 
21:15:39 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1',no_invalidate=>null);
PL/SQL procedure successfully completed.
 
21:15:41 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000007A373A08 3223759815 1 1 N 0
VALID 1 000000007A373A08 3223759815 1 1 N 0
 
21:15:41 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED
DEMOGTT1 SESSION 21:15:40

No invalidation here as in previous versions. But interesting thing is that I still have session statistics. The setting just disables its usage. But then, there were no invalidation and no rolling invalidation. Not sure how to interpret that…

Invalidation with online statistics gathering

In all those examples I’ve used dbms_stats with default no_invalidate. But in 12c statistics gathering can occur automatically during bulk insert. Let’s try that:

...
21:38:50 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 000000007A9D8860 3223759815 1 1 N 0
VALID 1 000000007A9D8860 3223759815 1 1 N 0
 
21:38:51 SQL> truncate table DEMOGTT1;
Table truncated.
21:38:52 SQL> insert /*+ append */ into DEMOGTT1 select * from dual;
1 row created.
 
21:38:53 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
INVALID_UNAUTH 0 000000007A9D8860 3223759815 1 1 N 1
INVALID_UNAUTH 1 000000007A9D8860 3223759815 1 1 N 1
21:38:53 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED
DEMOGTT1 SESSION 21:38:52

Same behaviour here. The online statistics gathering has gathered private statistics and invalidated all cursors.

NO_INVALIDATE=true

We can explicitly disable invalidation with no_invalidate=>true:

...
21:43:25 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 0000000075873D60 3223759815 1 1 N 0
VALID 1 0000000075873D60 3223759815 1 1 N 0
 
21:43:28 SQL> exec dbms_stats.gather_table_stats('DEMO','DEMOGTT1',no_invalidate=>true);
PL/SQL procedure successfully completed.
 
21:43:29 SQL> select object_status,child_number,address,hash_value,parse_calls,executions,is_rolling_invalid,invalidations from v$sql where sql_id='1pmuu9z02day7';
 
OBJECT_STATUS CHILD_NUMBER ADDRESS HASH_VALUE PARSE_CALLS EXECUTIONS IS_ROLLING_INVALID INVALIDATIONS
------------------- ------------ ---------------- ---------- ----------- ---------- ------------------ -------------
VALID 0 0000000075873D60 3223759815 1 1 N 0
VALID 1 0000000075873D60 3223759815 1 1 N 0
 
21:43:29 SQL> select table_name,scope,last_analyzed from user_tab_statistics where table_name='DEMOGTT1';
 
TABLE_NAME SCOPE LAST_ANA
------------------------------ ------- --------
DEMOGTT1 SHARED
DEMOGTT1 SESSION 21:43:28

Here, as requested, private statistics has been gathered but without cursor invalidation. However I’ll have new hard parse for my query because private statistics prevent sharing another cursor, but it’s not an invalidation of all cursors. The other sessions will continue to re-use their plan.

So what?

With those new features, we have the famous parsing dilemma again: do we want to avoid too many hard parses and share cursor with the risk of executing an execution plan that has been optimized for different data? Or do we prefer to optimize each query at the risk of more CPU consumption and shared pool contention? Given that 12c comes with adaptive dynamic sampling that can make hard parse longer, and sometimes very very long, all those new features may be gauged carefully.

If you want to avoid hard parses, you should set preferences to SHARED statistics and then gather statistics when the GTT is filled with the data you want to optimize for, and then lock it. If you don’t, then you are back to the problem that private statistics tries to solve: sharing a plan optimized for few rows and executed on thousands.

 

Cet article 12c GTT private statistics and cursor invalidation est apparu en premier sur Blog dbi services.

Documentum story – Status of IndexAgent not visible in HTTPS (xPlore)

Thu, 2016-10-27 02:00

In one of our project, we were using CS 7.2 P02 (upgraded later with P05) in correlation with xPlore 1.5 P02. With these versions, we wanted to setup the IndexAgents in HTTPS to have a completely secured environment. We choose to follow the xPlore documentation and use the Groovy script that EMC is now providing to facilitate the setup in HTTPS of the IndexAgents & Primary DSearch.

 

The first thing to note is that the documentation is describing some parts of how to setup the SSL in xPlore but it is not the complete setup and some other parts are missing! You can use one of my previous blog to know how to setup an IndexAgent in HTTPS using the Groovy Script: click here. The second thing to note is that after setting our IndexAgents in HTTPS, we faced an issue where the status of the IndexAgents weren’t available anymore (it was working in HTTP)! After following the steps described in the other blogs, please take a look at the explications below in case you are having the same issue.

 

So after doing what is described in the link above, the status of the IndexAgents in Documentum Administrator was “Not responding” and we weren’t able to start/stop them. Same behavior via iapi, the status command returned a 200 code. Unlike the HTTP response codes, the IndexAgents codes are:

  • 0 => Everything is running & working
  • 100 => The JBoss instance hosting the IndexAgent is running BUT the IndexAgent isn’t started (E.g.: not in the status “Start in Normal Mode”)
  • 200 => The JBoss instance hosting the IndexAgent isn’t running OR not responding

 

Now to check the status of the IndexAgent, you can do the following using iapi:

API> ?,c,select object_name, index_name from dm_ftindex_agent_config;
object_name                                      index_name
-----------------------------------------------  ------------------------------------------------
xplore_server_01_9200_IndexAgent                  DOCBASE_ftindex_01
(1 row affected)

API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,status
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES
 
  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 200

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES
 

API> exit
Bye

 

You can see above a status 200… I can assure you that the JBoss instance and IndexAgent are up, running & working properly (we had to start it using the GUI since it was the only way to check the status)! Then why is a status code 200 returned? Well that’s the purpose of this blog!

 

Like I said before, we choose to setup the SSL for the IndexAgents using the Groovy script and that’s the reason why we faced this issue in the first place… The Groovy script is setup to automatically update the file “standalone.xml” to add some configuration for the SSL. This is what the script will add:

[xplore@xplore_server_01 ~]$ grep -A4 "connector name=\"https\"" /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/standalone.xml
    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
        <ssl name="https" password="***********"
            certificate-key-file="/app/xPlore/dsearch/admin/./../../jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/my.keystore"
            cipher-suite="TLS_RSA_WITH_AES_128_CBC_SHA"/>
    </connector>

 

Now what is wrong with this configuration? It actually looks quite good… We were working on this specific issue for a few months with EMC without any special findings (no errors in logs, nothing in the DFC/RPC/HTTP_POST traces, aso…) when we noted that the Groovy script only added a “normal” Cipher to the standalone.xml file but our Content Servers were setup at that time using the default SSL mode of Documentum which is commonly named “Anonymous SSL”. As shown above, the SSL Cipher that the Groovy script added is “TLS_RSA_WITH_AES_128_CBC_SHA”. The problem is that the JBoss instance hosting the IndexAgent is restricting the Ciphers available but the docbase on the other side isn’t able to use this Cipher because it is using the “Anonymous SSL”. Therefore there is mismatch in the Ciphers available and the communication can’t be completed.

 

So what is the solution to that? Well we simply need to extend the Cipher list to find a Cipher that is available for both the docbase and the IndexAgent. To solve this issue, you just have to replace the “cipher-suite” as follow:

[xplore@xplore_server_01 ~]$ grep -A4 "connector name=\"https\"" /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/standalone.xml
    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
        <ssl name="https" password="***********"
            certificate-key-file="/app/xPlore/dsearch/admin/./../../jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/my.keystore"
            cipher-suite="TLS_DH_anon_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA"/>
    </connector>

 

With this setup, you just have to restart the JBoss instance hosting the IndexAgent, reinitialize the Content Server and after doing that, the status will be available again:

API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,status
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES

  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 100

SYSTEM ATTRIBUTES
 

APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES


API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,start
...
q1
API> next,c,q1
...
OK
API> dump,c,q1
...
USER ATTRIBUTES

  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 0

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES
 

API> exit
Bye

 

As you might know, the Cipher that we added contains “_anon_”. This means Anonymous (no authentication) and this also means that it is not the most secure of all Ciphers but this is necessary if you are using the default SSL mode in Documentum (without specifically setting up your own SSL Certificates). That’s why EMC recommends you to setup a non-Anonymous SSL mode but that’s a little bit more difficult to setup than just using the default one. Might be the subject of another blog :)

 

Cet article Documentum story – Status of IndexAgent not visible in HTTPS (xPlore) est apparu en premier sur Blog dbi services.

Documentum story – User gets kicked out of D2 when navigating in D2Config

Wed, 2016-10-26 04:30

On a customer site, one of the users complained about being kicked out each time he wanted to create new documents in D2. This issue is happening in a default deployment of D2 and D2Config in a WebLogic Domain.We found out that the user sessions for D2Config and D2 are conflicting together.
This issue occurs if D2Config and D2 applications are opened in the same browser using different TABs and if the user is navigating from D2 to D2Config and vice versa.
The error message is misleading as it shows a session time out and the user just signed in   .

D2 session timeout

Investigations:
Using a HTTP Header tracing tool, we saw that the JSESSIONID cookie which is the cookie storing the HTTP Session for the Java applications is changing when switching from one application to the other. This showed us that both Java Application were using the same session cookie which conduct to session lost.

Workaround or Fix:
An easy fix for this is to update the D2 weblogic.xml file included in the D2.war file with a section defining a new session cookie name as shown below:

<session-descriptor>
      <cookie-name>JSESSIONID_D2</cookie-name>
      <cookie-http-only>false</cookie-http-only>
</session-descriptor>

To proceed, follow the steps below:

  1. Extract the weblogic.xml file from the war file
    mkdir mytemp
    cd mytemp
    -- Put the D2.war file here
    jar xvf D2.war WEB-INF/weblogic.xml
  2. Edit the file and add the session-descriptor bloc above just after the Description closing TAG.
  3. Update the D2.war file with the new weblogic.xml
    jar uvf D2.war WEB-INF/weblogic.xml
  4. And finally redeploy the D2.war file to the WebLogic Server.

This fix has been submitted to and validated by EMC support.

 

Cet article Documentum story – User gets kicked out of D2 when navigating in D2Config est apparu en premier sur Blog dbi services.

Documentum story – Setup the DSearch & IndexAgent in HTTPS (xPlore)

Wed, 2016-10-26 02:00

In this blog, I will show you how to setup the Primary DSearch and IndexAgent in HTTPS for xPlore 1.5. The documentation about that is available on the EMC WebSite as always and the pdf name is: “xPlore 1.5 Installation Guide.pdf”. The reason why I wanted to wrote this blog is because the documentation is not too bad but there are still some missing parts and without these parts, your configuration will just not work properly. Moreover, I think it is better to have a concrete an complete example rather than just reading a PDF with some information spread on 40 different pages.

 

So let’s begin the configuration. The first thing to do is obviously to login to the Full Text Server where you xPlore 1.5 is installed. For this blog, I will use /app/xPlore as the installation folder of xPlore. I will also use a Self-Signed SSL Certificate with a Certificate Chain composed of a Root and Gold CA. So let’s import the Certificate Chain in xPlore (following commands assume all certificates are available under “/app/xPlore/jboss7.1.1/certs”):

[xplore@xplore_server_01 ~]$ /app/xPlore/java64/1.7.0_72/bin/keytool -import -trustcacerts -alias root_ca -keystore /app/xPlore/java64/1.7.0_72/jre/lib/security/cacerts -file /app/xPlore/jboss7.1.1/certs/Internal_Root_CA.cer
Enter keystore password:
[xplore@xplore_server_01 ~]$ /app/xPlore/java64/1.7.0_72/bin/keytool -import -trustcacerts -alias gold_ca -keystore /app/xPlore/java64/1.7.0_72/jre/lib/security/cacerts -file /app/xPlore/jboss7.1.1/certs/Internal_Gold_CA1.cer
Enter keystore password:

 

All java processes using /app/xPlore/java64/1.7.0_72/bin/java will now trust the Self-Signed SSL Certificate because the Certificate Chain is trusted. When this is done, shutdown all xPlore processes (Primary DSearch and IndexAgent(s)) and let’s configure the Primary DSearch in HTTPS:

[xplore@xplore_server_01 ~]$ /app/xPlore/scripts/startstop stop
  ** Indexagent_DOCBASE1 has been stopped successfully
  ** PrimaryDsearch has been stopped successfully
[xplore@xplore_server_01 ~]$ cd /app/xPlore/dsearch/admin
[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IS \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -isname PrimaryDsearch -ianame Indexagent_DOCBASE1

 

Some remarks:

  • “-enable” means that HTTPS will be enabled and HTTP will be disabled. If you want both to be enabled, use the “-dual” option instead
  • “-component” defines which component should be configured with this command. It can be “IS” (IndexServer), “IA” (IndexAgent) or “ALL” (IndexServer and IndexAgent)
  • “-isname” defines the name of the IndexServer/Primary DSearch that you installed
  • “-ianame” defines the name of the IndexAgent that you installed

 

Now what happen if you have more than one IndexAgent on the same server? Well the script isn’t smart enough for that and that’s the reason why I didn’t put “ALL” above but just “IS”. You might also noticed that I defined the “-ianame” parameter with “Indexagent_DOCBASE1”. This is because even if we are configuring the Primary DSearch in HTTPS, all IndexAgents have a reference in a configuration file that defines which port and protocol the IA should use to connect to the DSearch and if this isn’t setup properly, the IA will not be able to start.

 

Now the IndexServer is configured in HTTPS so let’s do the same thing for the IndexAgent:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent_DOCBASE1 -iaport 9200

 

As you can see above, this time no need to add the “-isname” parameter, it is not needed for the IndexAgent(s). Let’s say that you have a second IndexAgent for the docbase named DOCBASE2 , then you also have to execute the above command for this second indexAgent:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent_DOCBASE2 -iaport 9220

 

In case you didn’t know, yes each IndexAgent need at least 20 consecutive ports (so 9200 to 9219 for Indexagent_DOCBASE1 // 9220 to 9239 for Indexagent_DOCBASE2).

When configuring the IndexServer in HTTPS, I specified the “-ianame”. This is, like I said before, because there is a reference somewhere to the Protocol/Port used. This reference has been updated properly for Indexagent_DOCBASE1 normally but not for Indexagent_DOCBASE2. Therefore you need to do that manually:

[xplore@xplore_server_01 admin]$ grep -B1 -A10 dsearch_qrserver_protocol /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
        <parameter>
            <parameter_name>dsearch_qrserver_protocol</parameter_name>
            <parameter_value>HTTP</parameter_value>
        </parameter>
        <parameter>
            <parameter_name>dsearch_config_host</parameter_name>
            <parameter_value>xplore_server_01</parameter_value>
        </parameter>
        <parameter>
            <parameter_name>dsearch_config_port</parameter_name>
            <parameter_value>9300</parameter_value>
        </parameter>

 

Just open this file and update the few lines that I printed above by replacing “HTTP” with “HTTPS” and “9300” with “9302” and that’s it. If you have several IndexAgents, then you need to do that for all of them.

 

The next step is to login to the Content Server (e.g.: ssh dmadmin@content_server_01) and update some properties in the docbase:

[dmadmin@content_server_01 ~]$ iapi DOCBASE1 -Udmadmin -Pxxx
 

        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE1
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a8014087a started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> ?,c,select r_object_id from dm_ftengine_config where any lower(param_value) = lower('xplore_server_01');
r_object_id
----------------
083f245a800052ae
(1 row affected)

API> fetch,c,083f245a800052ae
...
OK
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : DSearch Fulltext Engine Configuration
  ...
  param_name                   [0]: dsearch_qrserver_protocol
                               [1]: dsearch_qrygen_mode
                               [2]: dsearch_qrserver_target
                               [3]: dsearch_qrserver_port
                               [4]: dsearch_config_port
                               [5]: dsearch_qrserver_host
                               [6]: dsearch_domain
                               [7]: dsearch_config_host
                               [8]: query_plugin_mapping_file
                               [9]: load_balancer_enabled
                              [10]: ft_wildcards_mode
  param_value                  [0]: HTTP
                               [1]: both
                               [2]: /dsearch/IndexServerServlet
                               [3]: 9300
                               [4]: 9300
                               [5]: xplore_server_01
                               [6]: DOCBASE1
                               [7]: xplore_server_01
                               [8]: /app/dctm/server/fulltext/dsearch/dm_AttributeMapping.xml
                               [9]: true
                              [10]: trailing_implicit
  ...

 

You might have noticed that I specified a WHERE clause on the select to find the r_object_id of the xPlore Server. That’s basically because in this case, there are two xPlore Servers in a HA setup (the parameter load_balancer_enabled is setup to true) but I just want to update the right object. So let’s update the parameters highlighted above (“dsearch_qrserver_protocol”, “dsearch_qrserver_port” and “dsearch_config_port”):

API> set,c,l,param_value[0]
SET> HTTPS
...
OK
API> set,c,l,param_value[3]
SET> 9302
...
OK
API> set,c,l,param_value[4]
SET> 9302
...
OK
API> save,c,l
...
OK

 

With these steps, we updated the dm_ftengine_config object. The next step is to also update the URL of the IndexAgent:

API> ?,c,select r_object_id from dm_server_config
r_object_id
----------------
3d3f245a80000102
3d3f245a80003796
(2 rows affected)

API> fetch,c,3d3f245a80000102
...
OK
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : DOCBASE1
  ...
  app_server_name              [0]: do_method
                               [1]: do_mail
                               [2]: do_bpm
                               [3]: xplore_server_01_9200_IndexAgent
                               [4]: xplore_server_02_9200_IndexAgent
  app_server_uri               [0]: https://content_server_01:9082/DmMethods/servlet/DoMethod
                               [1]: https://content_server_01:9082/DmMail/servlet/DoMail
                               [2]: https://content_server_01:9082/bpm/servlet/DoMethod
                               [3]: http://xplore_server_01:9200/IndexAgent/servlet/IndexAgent
                               [4]: http://xplore_server_02:9200/IndexAgent/servlet/IndexAgent
  ...

 

So we setup the IndexAgent installed on xplore_server_01 in HTTPS and therefore we also need to update the URL referenced in the docbase. That’s actually one of the things that aren’t in the official documentation at the moment. This is done as before:

API> set,c,l,app_server_uri[3]
SET> https://xplore_server_01:9202/IndexAgent/servlet/IndexAgent
...
OK
API> save,c,l
...
OK

 

As you saw above, this is an environment with two dm_server_config objects (two Content Servers) and two IndexAgents. Above, we setup the Primary DSearch and IndexAgent installed on xplore_server_01 in HTTPS. The dm_ftengine_config for this Primary DSearch has been updated and the URLs defined in one dm_server_config object have been updated too. But the same references are present in the second dm_server_config and therefore you also need to do that for the second one (3d3f245a80003796 in this case). Same steps so just repeat with the other r_object_id!

 

Ok so now all objects in the docbase have been updated successfully. Therefore return on the xPlore Server, clean the caches and start everything again:

[xplore@xplore_server_01 ~]$ rm -rf /app/xPlore/jboss7.1.1/server/DctmServer_*/tmp/work/*
[xplore@xplore_server_01 ~]$ /app/xPlore/scripts/startstop start
  ** PrimaryDsearch has been started successfully
  ** Indexagent_DOCBASE1 has been started successfully

 

As said before, some of these steps aren’t described/explained in the official documentation and that will lead you to a non-working situation… In addition to that, there are some bugs impacting the proper behavior of the Primary DSearch and/or the Index Agents when configured in HTTPS. We reported these bugs to EMC which was able to provide a fix for some of them and include that in a later patch but as you know it is not always possible to upgrade or patch your environment. For example with the CS 7.2 P02 or P05, the searches will NOT work against a DSearch in HTTPS (corrected in the P08 or P09 if I remember correctly) but I will not describe that in this blog. If you are facing an issue with the IndexAgents not responding in HTTPS, please check this blog.

 

Cet article Documentum story – Setup the DSearch & IndexAgent in HTTPS (xPlore) est apparu en premier sur Blog dbi services.

Documentum story – Thumbnail Server not starting if docbases are down

Tue, 2016-10-25 02:00

In this blog, I will talk about the Thumbnail Server. It’s a component of Documentum that you can install on the Content Server to generate… thumbnails! Basically what it does is that it will work in correlation with the ADTS/CTS in order to generate different kinds of previews of your document. For example jpeg_lres (low resolution) or jpeg_story (StoryBoard). You can define how many previews you want per document (1 preview per page, only the first page, aso…) and of which type. Then D2 can use these previews in a widget for the end-users to see what the document looks like. Apparently with D2 4.6, the Thumbnail Server and ADTS/CTS aren’t needed anymore to generate thumbnails but it was the case for D2 4.5 and previous versions.

 

That’s my first blog related to the Thumbnail Server because that’s actually a (the?) component of Documentum that is working pretty well without much issue so I absolutely wanted to explain the issue I faced and what has been done to solve that.

 

So let’s start with some background: I was working on a project where the TEST and PROD environments contain only one docbase (DOCBASE1) in addition to the Global Registry. In the DEV environment, there were three docbases for development purposes but the additional two (DOCBASE2 and DOCBASE3) were stopped for a few days because we were running a lot of Performance Tests with EMC and we wanted the results to reflect the TEST/PROD environments. At this point, the Thumbnail Server was working properly, previews were generated successfully, aso… Then to apply a change related to the Performance Tests, we had to restart the whole Content Server, including the Thumbnail Server since it has been installed on all Content Servers (HA environment). The change wasn’t related to the Thumbnail Server at all but we discovered a small bug because of this: after the restart, the Thumbnail Server wasn’t working anymore. Just like the JMS/ACS, there is a way to very quickly know if the Thumbnail Server is up & running or not and that can be checked by entering the following URL in a web browser: http(s)://content_server_01:port/thumbsrv/getThumbnail?

 

In our case, this URL wasn’t working anymore after the restart while it was working properly before and therefore I had to look at the Thumbnail log file. One important thing to note here is that the Thumbnail Server is bundled with Tomcat. For those of you who are used to work with Tomcat, your first reaction might be to open the file $TOMCAT_HOME/logs/catalina.out. For the Thumbnail, this would be the file: $DM_HOME/thumbsrv/container/logs/catalina.out. That’s what I did… But there were absolutely no useful information because only the Tomcat initialization is displayed in this file by default. The actual useful information for the Thumbnail Server are stored in the localhost log file… That’s the file I’m usually completely ignoring because there is less inside it than in the catalina.out file but apparently EMC took a difference approach!

 

So let’s take a look at what the log file is providing:

May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:23 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
  ...
May 04, 2016 2:10:05 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:11 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE2 - 0.0
May 04, 2016 2:10:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:24 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
  ...
May 04, 2016 2:13:06 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:12 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE3 - 0.0
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 2:13:20 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Failed to get a session for DOCBASE2: DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
SEVERE: getThumbnail: [DM_TS_E_INIT_FORMATS_MGR] Falied to initialize formats.
DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getDataFromDocbroker(ServerMapBuilder.java:171)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getMap(ServerMapBuilder.java:60)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.__AW_updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:180)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:110)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.__AW_newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.thumbsrv.docbase.DocbaseConnectionMgr.getSessionForSection(DocbaseConnectionMgr.java:197)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.__AW_loadFormatsFromRepositories(FormatMapperMgr.java:95)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.loadFormatsFromRepositories(FormatMapperMgr.java)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.<init>(FormatMapperMgr.java:62)
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:214)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: Marking servlet getThumbnail as unavailable
May 04, 2016 2:13:34 PM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /thumbsrv threw load() exception
javax.servlet.UnavailableException: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:221)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 

May 04, 2016 2:17:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:22:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:27:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:32:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:37:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:42:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:47:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable

 

Ok that’s quite a long trace and I already cut some parts because it was really too long. As you can see above, the very beginning of the startup is going on properly and then the Thumbnail Server is trying to contact the DOCBASE2. Hum why exactly is it trying to contact the DOCBASE2 while this docbase isn’t running? That’s a first strange thing. Then I cut a lot of lines but you can see that it is actually trying to do that for 3 minutes and it is not doing anything else during that time. Once the three minutes are over, it is trying to contact the DOCBASE3 for 3 minutes too (and failed again) and finally it contacted the DOCBASE1 which was the only docbase running at that time. This last one succeeded so the Thumbnail Server should have started properly… But actually it couldn’t because the first docbase that it tried fail (DOCBASE2) and that’s the docbase that will be used by the Thumbnail Server to open a session to retrieve some information.

 

So I did more tests to try to understand where the issue was and what could be done to solve that. First of all, the important thing to understand here is that the Thumbnail Server isn’t trying to contact all docbases ever installed on this Content Server. It will only try to contact the docbases that have been configured for it. To be more precise, the configuration of a docbase for the Thumbnail Server will update the file user.dat ($DM_HOME/thumbsrv/conf/user.dat) and add inside it the configuration for this specific docbase. When the Thumbnail Server will start, it will parse this file and see which docbases should be contacted during the startup.

 

Therefore my first test was to simply comment all lines related to DOCBASE2 and DOCBASE3 and then restart the Thumbnail Server. This is what I got:

May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 3:23:41 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_COMPLETE] Documentum Thumbnail Server initialization complete!

 

As you can see, the content is exactly the same except that the Thumbnail Server is now only contacting the DOCBASE1 (successfully again) but this time, the Thumbnail Server is up & running properly. Therefore commenting the lines in the file user.dat solved this issue… But that’s not enough. For me, this was clearly a bug and therefore I did more tests.

 

For the next test, I restored the lines related to DOCBASE2 and DOCBASE3 in the file user.dat and I tried to change the order of the lines inside this file… Because DOCBASE1 was the first docbase in this file, DOCBASE2 the second and DOCBASE3 the last one. So I thought that maybe the second docbase was the first to be initialized? And if the first docbase to be initialized (2nd in the file?) is responding, will the Thumbnail Server work?

 

Therefore I changed the order:

  • Switching DOCBASE1 in second position, DOCBASE2 in first position, DOCBASE3 still in last => Not working, same issue, DOCBASE2 is the first to be initialised
  • Switching DOCBASE1 in second position, DOCBASE2 in last position, DOCBASE3 in first position => Not working, same issue, DOCBASE3 is the first to be initialised

 

This test wasn’t successful because there is absolutely no logic behind which docbase will be initialized first… Therefore I restored the initial values (1 in first, 2 in second, 3 in last) and I performed a last test: stopping the DOCBASE1 and starting the DOCBASE2 so that there is still one docbase to be running which would be the first to be initialized. After doing that, I restarted a last time the Thumbnail Server and the first docbase to be initialized was indeed the DOCBASE2, which was running. Then it tried to initialise the DOCBASE1 and DOCBASE3 which weren’t running and therefore it failed. BUT in the end, the Thumbnail Server was able to start properly, even if it took 6 minutes + ~20 seconds instead of 20 seconds to start.

 

With all these results, I opened a Service Request on the EMC website and they were able to find the root cause for both issues (why it is looping for 3 minutes on nothing and why it is sometimes not able to start when one docbase is properly responding). In the end, they provided us a hotfix that has been incorporated in newer versions of the software normally and this hotfix fixed both issues properly.

 

 

Cet article Documentum story – Thumbnail Server not starting if docbases are down est apparu en premier sur Blog dbi services.

SQL Server 2016: Distributed availability groups and Cross Cluster migration

Mon, 2016-10-24 06:43

How to migrate an environment that includes availability groups from one Windows Failover Cluster to another one? This scenario is definitely uncommon and requires a good preparation. How to achieve this task depends mainly of your context. Indeed, we may use a plenty of scenarios according to the architecture in-place as well as the customer constraints in terms of maximum downtime allowed for example. Among all possible scenarios, there is a process called “cross-cluster migration scenario” that involves two Windows Failover Clusters side by side. In this blog post, I would like to focus on it and I will then show improvements in this field with SQL Server 2016.

Just to be clear, for the first scenario my intention is not to detail all the steps required to achieve a cross-cluster migration. You may refer to the whitepaper written by Microsoft here. To be honest, my feeling is the documentation may be improved because it includes sometimes a lack of specificity but it has the great merit of being a reality. I remember to navigate through this document, go down and go up back to the beginning several times by using the bookmarks in order to be confident with my understanding of the whole process. Anyway, let’s apply this procedure to a real customer case.

In my experience, I may confirm we haven’t customers with only one availability group installed on their environment. Most of time, we encountered customer cases which include several SQL Server instances and replicas on the same cluster node. I will demonstrate it with the following infrastructure (similar to most customer shops):

 

blog 107 - 0 - distributed ag cross cluster migration use case

We have 3 availability groups with the first two ones hosted on the same SQL14 instance and the last one on the SQL142 instance. The availability group architecture runs on the top of a Windows failover cluster (WSFC) – WIN201216CLUST – that includes two cluster nodes and a file share witness (FSW) not reported in the above picture. So a pretty common scenario at customer shops as I said previously. Without going into details of customer cases, the idea was to migrate all the current physical environment from the first datacenter (subnet 192.168.5.0) to the second datacenter on a virtual environment (subnet 192.168.50.0). As an aside, note that my customer subnets are not exactly the same and he used a different set of IP ranges but anyway it will help to set the scene.

So basically, according to the Microsoft documentation the migration process is divided in 4 main steps:

  • Preparation of the new WSFC environment (no downtime)
  • Preparation of the migration plan (no downtime).
  • Data migration (no downtime)
  • Resource migration (downtime)

Probably, the main advantage of using this procedure is the short outage that will occur during the last step (resource migration). In this way, we are more comfortable with previous preparation steps because they do not require downtime. Generally migration of data between two replicas is an important part of the migration process in terms of time and we are able to prepare smoothly the migration stuff between the two WSFCs.

 

Preparation of the new WSFC

Further points in the process have brought my attention. The first one concerns the preparation of the new WSFC environment (first step). Basically, we have to prepare the target environment that will host our existing availability groups and Microsoft warned us about the number of nodes (temporary nodes or not) regarding the overlapping among availability groups section and migration batches. During this preparation step we have also to set the corresponding cluster registry permissions to allow correct switching of the cluster context from the new installed replicas on the remote WSFC. At the first glance, I wondered why we have to perform such operation but the response became obvious when I tried to do a switch of the cluster from my new SQL Server instance and I faced the following error message:

ALTER SERVER CONFIGURATION SET HADR CLUSTER CONTEXT = 'WIN201216CLUST.dbi-services.test'

 

Msg 19411, Level 16, State 1, Line 1
The specified Windows Server Failover Clustering (WSFC) cluster,  ‘WIN201216CLUST.dbi-services.test’, is not ready to become the cluster context of AlwaysOn Availability Groups (Windows error code: 5).

The possible reason could be that the specified WSFC cluster is not up or that a security permissions issue was encountered. Fix the cause of the failure, and retry your ALTER SERVER CONFIGURATION SET HADR CLUSTER CONTEXT = ‘remote_wsfc_cluster_name’ command.

It seems that my SQL Server instance is trying unsuccessfully to get information from the registry hive of the remote WSFC in order to get a picture of the global configuration. As an aside, until the cluster context is not switched, we are not able to add the new replicas to the existing availability group. If we put a procmon trace (from sysinternals) on the primary cluster node, we may notice that executing the above command from the remote SQL Server instance implies the reading of the local HKLM\Cluster hive.

Well, after fixing the cluster permissions issues by using the PowerShell script provided by Microsoft, we may add the concerned replicas to our existing AG configuration. The operation must be applied on all the replicas from the same WSFC. According to the Microsoft documentation, I added then two replicas in synchronous replication and another one in asynchronous mode. A quick look at the concerned DMVs confirms that everything is ok

SELECT	
	g.name as ag_name,
	r.replica_server_name as replica_name,
	rs.is_local,
	rs.role_desc AS [role],
	rs.connected_state_desc as connection_state,
	rs.synchronization_health_desc as sync_state
FROM sys.dm_hadr_availability_replica_states as rs
JOIN sys.availability_groups as g
	on g.group_id = rs.group_id
JOIN sys.availability_replicas as r
	on r.replica_id = rs.replica_id

 

blog 107 - 1 - distributed ag cross cluster migration 1

 

SELECT 
	g.name as ag_name,
	r.replica_server_name as replica_name,
	DB_NAME(drs.database_id) AS [db_name],
	drs.database_state_desc as db_state,
	drs.is_primary_replica,
	drs.synchronization_health_desc as sync_health,
	drs.synchronization_state_desc as sync_state
FROM sys.dm_hadr_database_replica_states as drs
JOIN sys.availability_groups as g
	on g.group_id = drs.group_id
JOIN sys.availability_replicas as r
	on r.replica_id = drs.replica_id
ORDER BY r.replica_server_name

 

blog 107 - 2 - distributed ag cross cluster migration 2

If you are curious like me you may wonder how SQL Server deals with the two remote replicas? A quick look at the registry doesn’t give us any clue. But what about getting registry changed values after adding new replicas? Regshot tool was a good tool to use in my case to track changes between two registry snapshots:

blog 107 - 3 - distributed ag cross cluster migration 3

This is probably not an exhaustive list of added or modified keys but this output provides a lot of useful information to understand what’s happening to the cluster registry hive when adding remote replicas. The concerned resource is identified by the id 9bb8b518-2d1a-4705-a378-86f282d387da which corresponds to my DUMMY availability group. It makes sense to notice some changes at this level.

blog 107 - 4 - distributed ag cross cluster migration 4

I may formulate some assumptions here. Those registry changes are necessary to represent the complete picture of the new configuration (with ID 8B62AACE-6AFC-49B7-9369-590D5E832ED6). If we refer to the SQL Server error log we may identify easily each replica server name by its corresponding hexadecimal value.

 

Resource migration

However, resource migration is a critical part of the migration process because it will introduce downtime. The downtime duration depends mainly on the number of items to migrate. Migrating availability group includes bringing offline each availability group as well as dropping each corresponding listener at the source and then recreating the availability group configuration at the target. In other words, the more you have items to migrate, the longer this migration step might be.

We are also concerned by the availability group’s topology and migration batches. Indeed, according to the Microsoft documentation, we may not change the HADR context of the targeted SQL Server instance until we have migrated all the related availability groups preventing using them as new functional replicas. To understand the importance of migration batches, think about the following scenario: the HADR context to the remote cluster is enabled and you have just finished to migrate the first availability group. You then switch back the HADR to the local context but you forget to migrate the second availability. At this point, reverting the HADR context to the remote cluster is not possible because the concerned replica is no longer eligible.

Assuming I used a minimal configuration that includes only two target replicas as shown in the first picture (WSFC at the destination), I have at least to group DUMMY and DUMMY2 availability groups in one batch. DUMMY3 availability group may be migrated as a separate batch.

So basically, steps to perform the resources migration are as follows:

  • Stop application connectivity on each availability group
  • Bring offline each availability group (ALTER AVAILABILTY GROUP OFFLINE) and drop the corresponding listener
  • Set the HADR context to local for each cluster node
  • Recreate each availability group and the corresponding listener with the new configuration
  • Validate application connectivity

The migration script provided by Microsoft helps a lot in the generation of the availability group definition but we face some restrictions. Indeed, the script doesn’t retain the exact configuration of the old environment including the previous replication mode or the backup policy for example. This is an expected behavior and according to the Microsoft documentation it is up to you to maintain the existing configuration at the destination.

Finally after performing the last migration step here is the last configuration for my DUMMY availability group as follows:

blog 107 - 5 - distributed ag cross cluster migration 5

blog 107 - 6 - distributed ag cross cluster migration 6

 

So what about SQL Server 2016 in the context of cross-cluster migration?

By starting with SQL Server 2016, distributed availability groups are probably the way to go. I already introduced this feature in a previous blog and I would like to show you how interesting this feature is in this context. Well let’s first go back to the initial context. We basically have to perform the exact same steps compared to the first scenario but distributed availability groups may increase ? drastically the complexity of the entire process.

  • Preparation of the new WSFC environment (no downtime) – We no longer need to grant access on the cluster registry hive permissions to the SQL Server service account as well as switching the HADR context to the remote cluster
  • Preparation of the new availability groups on the destination WSFC including the corresponding listeners (no downtime) – We no longer need to take into account the migration batches in the migration plan
  • Data migration between the availability groups (no downtime)
  • Application redirection to the new availability groups (downtime) – We no longer need to switch back the HADR context to local nodes as well as recreating the different availability group’s configurations at this level.

In short, the migration of availability groups across WSFC with SQL Server 2016 requires less efforts and shorter downtime.

Here is the new scenario after preparing the new WSFC environment and applying data migration between availability groups in the both sides

blog 107 - 7 - distributed ag cross cluster migration 7

 

During the first phase, I prepared a new Windows Failover Cluster WIN2012162CLUST which will host empty availability groups DUMMY10, DUMMY20 and DUMMY30. Those availability groups will act as new containers when implementing distributed availability groups (respectively TEMPDB_DUMMY_AG, TEMPDB_DUMMY2AG and TEMPDB_DUMMY3AG). You may notice that I configured ASYNC replication mode between local and remote availability groups but regarding our context, synchronous replication mode remains a viable option.

 

blog 107 - 8 - distributed ag cross cluster migration 8

 

A quick look at the sys.dm_hadr_database_replica_states confirms all the distributed availability groups are working well. TEMPDB_DUMMY3_AG is not included in the picture below.

blog 107 - 9 - distributed ag cross cluster migration 9

At this step, availability groups (DUMMY10 and DUMMY20) on the remote WSFC cannot be accessed and are used only as standby waiting to be switched as new primaries.

In the last step (resources migration) here the new steps we have to execute

  • Failover all the availability groups context to the remote WSFC by using distributed availability group capabilities
  • Redirect applications to point to the new listener

And that’s all!

In the respect of the last point, we may use different approaches and here is mine regarding the context. We don’t want to modify application connection strings that may lead to extra steps from the application side. Listeners previously created with remote availability groups may be considered as technical listeners in order to achieve cross-cluster migration through distributed availability groups. Once this operation is done, we may reuse old listeners in the new environment and achieve almost transparent application redirection in this way.

One another important thing I have to cover is the distributed availability group process behavior. After digging into several failover tests, I noticed triggering a failover event from a distributed availability group will move the primary replica of each secondary availability group to PRIMARY but unfortunately, it will not switch automatically the role of the primary of each primary availability group to SECONDARY as expected. This situation may lead to a split brain scenario in the worst case. The quick workaround consists in forcing the primary availability group to be SECONDARY before initiating the failover process. Let me demonstrate a little bit:

/* ========= 0- DEMOTE OLD LOCAL AGS TO SECONDARY TO AVOID UPDATE ACTIVITY  ========= 
   =========    FROM THE OLD PRIMARIES (SPLIT BRAIN SCENARIO                ========= */

:CONNECT LST-DUMMY

ALTER AVAILABILITY GROUP [TEMPDB_DUMMY_AG] SET (ROLE = SECONDARY);
ALTER AVAILABILITY GROUP [TEMPDB_DUMMY2_AG] SET (ROLE = SECONDARY);
GO


/* ========= 1- FORCE FAILOVER DISTRIBUTED AVAILABILTY GROUPS ========= */

:CONNECT LST-DUMMY10

USE master;
GO

ALTER AVAILABILITY GROUP [TEMPDB_DUMMY_AG] FORCE_FAILOVER_ALLOW_DATA_LOSS;
GO

ALTER AVAILABILITY GROUP [TEMPDB_DUMMY2_AG] FORCE_FAILOVER_ALLOW_DATA_LOSS;
GO

The new situation is as follows:

blog 107 - 10 - distributed ag cross cluster migration 10

Let’s try to connect to the old primary availability group. Accessing the DUMMY database is no longer permitted as expected

Msg 976, Level 14, State 1, Line 1
The target database, ‘DUMMY’, is participating in an availability group and is currently
not accessible for queries. Either data movement is suspended or the availability replica is not
enabled for read access. To allow read-only access to this and other databases in the availability
group, enable read access to one or more secondary availability replicas in the group. 
For more information, see the ALTER AVAILABILITY GROUP statement in SQL Server Books Online.

We now have to get back the old listener for the new primary availability groups. In my case, I decided to drop the old availability groups in order to completely remove the availability group configuration from the primary WSFC and to make the concerned databases definitely inaccessible as well (RESTORING STATE).

/* ========= 3- DROP LISTENERS FROM THE OLD AVAILABILITY GROUPS  ========= */

:CONNECT WIN20121SQL16\SQL16

USE master;
GO

DROP AVAILABILITY GROUP [DUMMY];
GO

DROP AVAILABILITY GROUP [DUMMY2];
GO

 

Finally I may drop technical listeners and recreate application listeners by using the following script:

/* ========= 4- DROP TECHNICAL LISTENERS FROM THE NEW AVAILABILITY GROUPS  =========
   =========    ADD APPLICATION LISTENERS WITH NEW CONFIG                  ========= */
                      

:CONNECT WIN20122SQL16\SQL16

USE master;
GO

ALTER AVAILABILITY GROUP [DUMMY10] 
REMOVE LISTENER 'LST-DUMMY10';
GO

ALTER AVAILABILITY GROUP [DUMMY20] 
REMOVE LISTENER 'LST-DUMMY20';
GO

ALTER AVAILABILITY GROUP [DUMMY10]
ADD LISTENER 'LST-DUMMY' (WITH IP ((N'192.168.50.35', N'255.255.255.0')), PORT=1433);
GO

ALTER AVAILABILITY GROUP [DUMMY20]
ADD LISTENER 'LST-DUMMY' (WITH IP ((N'192.168.50.36', N'255.255.255.0')), PORT=1433);
GO

 

Et voilà!

 

Final thoughts

Cross-cluster migration is definitly a complex process regardless SQL Server 2016 improvements. However as we’ve seen in this blog post, the last SQL Server version may reduce the overall  complexity at the different migration steps. Personnally as a DBA, I don’t like to use custom registry modification stuff which may impact directly the WSFC level (required in the first migration model) because it may introduce some anxiousness and unexpected events. SQL Server 2016 provides a more secure way through distributed availability groups and includes all migration steps at the SQL Server level which make me more confident with the migration process.

Happy cross-cluster migration!

 

 

 

 

Cet article SQL Server 2016: Distributed availability groups and Cross Cluster migration est apparu en premier sur Blog dbi services.

Documentum story – Attempt to fetch object with handle 3c failed

Mon, 2016-10-24 02:00

Some time ago, I was working on the preparation of an upgrade of a Content Server and everything was working fine so I was about to begin but just before that I checked our monitoring interface for this environment to crosscheck and I saw the following alerts coming from the log file of the docbases installed on this CS:

2016-04-05T10:47:01.411651      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"

 

As you might know, an object with an ID starting with “3c” represents the dm_docbase_config and therefore I was a little bit afraid when I saw this alert. So first thing, I tried to open an idql session to see if the docbase was responding:

[dmadmin@content_server_01 log]$ iapi DOCBASE1 -Udmadmin -Pxxx


        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE1
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a800d7441 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_docbase_config
...
3c3f245a60000210
API> exit
Bye

 

Ok so apparently the docbase is working properly, we can access D2, DA, the idql is also working and even the dm_docbase_config object can be retrieved and seen but there is still the exact same error in the log file coming every 5 minutes exactly. So I took a look at the error message in detail because I didn’t want to scroll hundreds of lines for maybe nothing. The string description containing the object ID wouldn’t really be useful inside the log file to find what might be the root cause and the same thing apply for the DM_OBJ_MGR_E_FETCH_FAIL error, it would just print the exact same error again and again with just different timestamps. When thinking about that in my mind, I actually realised that the 2 or 3 error lines I was able to see on my screen were completely exact – except the timestamps – and that include the process ID that is throwing this error (second column on the log file).

 

With this new information, I tried to find all log entries related to this process ID:

[dmadmin@content_server_01 log]$ grep "21425\[21425\]" $DOCUMENTUM/dba/log/DOCBASE1.log | more
2016-04-04T06:58:09.315163      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 attempting to open"
2016-04-04T06:58:09.315282      21425[21425]    0000000000000000        [DM_SERVER_I_START_KEY_STORAGE_MODE]info:  "Docbase DOCBASE1 is using database for cryptographic key storage"
2016-04-04T06:58:09.315316      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 process identity: user(dmadmin)"
2016-04-04T06:58:11.400017      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Post Upgrade Processing."
2016-04-04T06:58:11.401753      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Base Types."
2016-04-04T06:58:11.404252      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRecovery."
2016-04-04T06:58:11.412344      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmACL."
2016-04-04T06:58:11.438249      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDocbaseIdMap."
2016-04-04T06:58:11.447435      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Error log streams."
2016-04-04T06:58:11.447915      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmUser."
2016-04-04T06:58:11.464912      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmGroup."
2016-04-04T06:58:11.480200      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmSysObject."
2016-04-04T06:58:11.515201      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmExprCode."
2016-04-04T06:58:11.524604      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmKey."
2016-04-04T06:58:11.533883      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueAssist."
2016-04-04T06:58:11.541708      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueList."
2016-04-04T06:58:11.551492      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueQuery."
2016-04-04T06:58:11.559569      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueFunc."
2016-04-04T06:58:11.565830      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmExpression."
2016-04-04T06:58:11.594764      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLiteralExpr."
2016-04-04T06:58:11.603279      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmBuiltinExpr."
2016-04-04T06:58:11.625736      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFuncExpr."
2016-04-04T06:58:11.636930      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmCondExpr."
2016-04-04T06:58:11.663622      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmCondIDExpr."
2016-04-04T06:58:11.707363      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDInfo."
2016-04-04T06:58:11.766883      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmScopeConfig."
2016-04-04T06:58:11.843335      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDisplayConfig."
2016-04-04T06:58:11.854414      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmNLSDDInfo."
2016-04-04T06:58:11.878566      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDomain."
2016-04-04T06:58:11.903844      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmAggrDomain."
2016-04-04T06:58:11.929480      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmMountPoint."
2016-04-04T06:58:11.957705      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLocation."
2016-04-04T06:58:12.020403      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Server Configuration."
2016-04-04T06:58:12.135418      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmPolicy."
2016-04-04T06:58:12.166923      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDCommonInfo."
2016-04-04T06:58:12.196057      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDAttrInfo."
2016-04-04T06:58:12.238040      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDTypeInfo."
2016-04-04T06:58:12.269202      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRelation."
2016-04-04T06:58:12.354573      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmForeignKey."
2016-04-04T06:58:12.387309      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmEvent."
2016-04-04T06:58:12.403895      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize DQL."
2016-04-04T06:58:12.405622      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFolder."
2016-04-04T06:58:12.433583      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDocument."
2016-04-04T06:58:12.480234      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Plugin Type."
2016-04-04T06:58:12.490196      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmNote."
2016-04-04T06:58:12.518305      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmComposite."
2016-04-04T06:58:12.529351      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmStorage."
2016-04-04T06:58:12.539944      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Common area."
2016-04-04T06:58:12.612097      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize File Store."
2016-04-04T06:58:12.675604      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Optical Store."
2016-04-04T06:58:12.717573      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Linked Stores."
2016-04-04T06:58:12.803227      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Stores."
2016-04-04T06:58:13.102632      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Blob Stores."
2016-04-04T06:58:13.170074      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External Store."
2016-04-04T06:58:13.242012      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External File Store."
2016-04-04T06:58:13.305767      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External URL."
2016-04-04T06:58:13.363407      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External Free."
2016-04-04T06:58:13.429547      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmContent."
2016-04-04T06:58:13.461400      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmiSubContent."
2016-04-04T06:58:13.508588      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmOutputDevice."
2016-04-04T06:58:13.630872      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmMethod."
2016-04-04T06:58:13.689265      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRouter."
2016-04-04T06:58:13.733289      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFederation."
2016-04-04T06:58:13.807554      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmAliasSet."
2016-04-04T06:58:13.871634      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Content Addressable Storage."
2016-04-04T06:58:13.924874      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Formats."
2016-04-04T06:58:13.995154      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Convert."
2016-04-04T06:58:13.998050      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Indices."
2016-04-04T06:58:14.025587      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Dump Files."
2016-04-04T06:58:14.107689      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Load Files."
2016-04-04T06:58:14.176232      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize In Box."
2016-04-04T06:58:14.225954      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed References."
2016-04-04T06:58:14.292782      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Registrations."
2016-04-04T06:58:14.319699      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Rights."
2016-04-04T06:58:14.330420      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Rights Domain."
2016-04-04T06:58:14.356866      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Business Activities."
2016-04-04T06:58:14.411545      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Business Processes."
2016-04-04T06:58:14.444264      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Packages."
2016-04-04T06:58:14.493478      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Workitems."
2016-04-04T06:58:14.521836      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Workflows."
2016-04-04T06:58:14.559605      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Audit Trail."
2016-04-04T06:58:14.639303      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Clean Old Links."
2016-04-04T06:58:14.640511      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Compute Internal Type Tag Cache."
2016-04-04T06:58:14.696040      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize LastActionProcs."
2016-04-04T06:58:14.696473      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize User Types."
2016-04-04T06:58:14.696828      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Start Up - Phase 1."
2016-04-04T06:58:15.186812      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Start Up - Phase 2."
2016-04-04T06:58:15.666149      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Crypto Objects."
2016-04-04T06:58:15.677119      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Queue Views."
2016-04-04T06:58:15.678343      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Port Info Views."
2016-04-04T06:58:15.679532      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Global Cache Finalization."
2016-04-04T06:58:16.939100      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize National Language Character Translation."
2016-04-04T06:58:16.941815      21425[21425]    0000000000000000        [DM_CHARTRANS_I_TRANSLATOR_OPENED]info:  "Translator in directory ($DOCUMENTUM/product/7.2/install/external_apps/nls_chartrans) was added succesfully initialized.  Translator specifics: (Chararacter Translator: , Client Locale: (Windows :(4099), Version: 4.0), CharSet: ISO_8859-1, Language: English_US, UTC Offset: 0, Date Format:%2.2d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d, Java Locale:en, Server Locale: (Linux :(8201), Version: 2.4), CharSet: UTF-8, Language: English_US, UTC Offset: 0, Date Format:%2.2d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d, Java Locale:en, Shared Library: $DOCUMENTUM/product/7.2/install/external_apps/nls_chartrans/unitrans.so)"
2016-04-04T06:58:16.942377      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize LDAP setup."
2016-04-04T06:58:16.961767      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed change-checking."
2016-04-04T06:58:17.022448      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Fulltext Plugin and Configuration."
2016-04-04T06:58:17.113147      21425[21425]    0000000000000000        [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info:  "Loaded FT Query Plugin: $DOCUMENTUM/product/7.2/bin/libDsearchQueryPlugin.so, API Interface version: 1.0, Build number: HEAD; Sep 14 2015 07:48:06, FT Engine version: xPlore version 1.5.0020.0048"
2016-04-04T06:58:17.122313      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content."
2016-04-04T06:58:17.125027      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content Map."
2016-04-04T06:58:17.125467      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content Digital Signatures."
2016-04-04T06:58:17.621156      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Acs Config List."
2016-04-04T06:58:17.621570      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLiteSysObject."
2016-04-04T06:58:17.623010      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmBatchManager."
2016-04-04T06:58:17.624369      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Partition Scheme."
2016-04-04T06:58:17.627552      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Authentication Plugins."
2016-04-04T06:58:17.631207      21425[21425]    0000000000000000        [DM_SESSION_I_AUTH_PLUGIN_LOADED]info:  "Loaded Authentication Plugin with code 'dm_krb' ($DOCUMENTUM/dba/auth/libkerberos.so)."
2016-04-04T06:58:17.631480      21425[21425]    0000000000000000        [DM_SESSION_I_AUTH_PLUGIN_LOAD_INIT]info:  "Authentication plugin ( 'dm_krb' ) was disabled. This is expected if no keytab file(s) at location ($DOCUMENTUM/dba/auth/kerberos).Please refer the content server installation guide."
2016-04-04T06:58:17.638885      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 opened"
2016-04-04T06:58:17.639005      21425[21425]    0000000000000000        [DM_SERVER_I_SERVER]info:  "Setting exception handlers to catch all interrupts"
2016-04-04T06:58:17.639043      21425[21425]    0000000000000000        [DM_SERVER_I_START]info:  "Starting server using service name:  DOCBASE1"
2016-04-04T06:58:17.810637      21425[21425]    0000000000000000        [DM_SERVER_I_LAUNCH_MTHDSVR]info:  "Launching Method Server succeeded."
2016-04-04T06:58:17.818319      21425[21425]    0000000000000000        [DM_SERVER_I_LISTENING]info:  "The server is listening on network address (Service Name: DOCBASE1_s, Host Name: content_server_01 :V4 IP)"
2016-04-04T06:58:17.821615      21425[21425]    0000000000000000        [DM_SERVER_I_IPV6_DISABLED]info:  "The server can not listen on IPv6 address because the operating system does not support IPv6"
2016-04-04T06:58:19.301490      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent master (pid : 21612, session 013f245a80000007) is started sucessfully."
2016-04-04T06:58:19.302601      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21613, session 013f245a8000000a) is started sucessfully."
2016-04-04T06:58:20.304937      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21626, session 013f245a8000000b) is started sucessfully."
2016-04-04T06:58:21.307256      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21639, session 013f245a8000000c) is started sucessfully."
2016-04-04T06:58:22.307448      21425[21425]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2016-04-04T06:58:22.325337      21425[21425]    0000000000000000        [DM_MQ_I_DAEMON_START]info:  "Message queue daemon (pid : 21655, session 013f245a80000456) is started sucessfully."

 

Ok so this is the first and second pages I got as a result and that’s actually when I realised that the process I was talking about above was in fact the docbase process… So I displayed the third page of the more command and I got the following:

2016-04-05T10:04:28.305238      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_CURSOR_FAIL]error:  "In operation Exec an attempt to create cursor failed; query was: 'SELECT * FROM DM_DOCBASE_CONFIG_RV dm_dbalias_B , DM_DOCBASE_CONFIG_SV dm_dbalias_C  WHERE (dm_dbalias_C.R_OBJECT_ID=:dmb_handle AND dm_dbalias_C.R_OBJECT_ID=dm_dbalias_B.R_OBJECT_ID) ORDER BY dm_dbalias_B.R_OBJECT_ID,dm_dbalias_B.I_POSITION'; error from database system was: ORA-03114: not connected to ORACLE"
2016-04-05T10:04:28.305385      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:04:28.317505      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:28.317591      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:58.329725      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:58.329884      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:28.339052      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:28.339143      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:49.077076      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
2016-04-05T10:05:49.077163      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
2016-04-05T10:06:23.461495      21425[21425]    013f245a80000002        [DM_SESSION_W_RESTART_AGENT_EXEC]warning:  "The agent exec program has stopped running.  It will be restarted."
2016-04-05T10:06:48.830854      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:11:52.533340      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:16:52.574766      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:21:52.546389      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:26:52.499108      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:31:52.232095      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:36:57.700202      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:42:01.198050      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:47:01.411651      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:52:02.242612      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:57:11.886518      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T11:02:13.133405      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T11:07:15.364236      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"

 

As you can see above, the first error occurred at “2016-04-05T10:04:28.305385″ and that’s actually 0,000147s (=0,1ms) after an error while trying to execute an SQL query on the database for the operation Exec… So this must be linked… The database errors stopped one minute after the first one so the DB was available again. I quickly verified that using sqlplus.

 

We opened a SR on the EMC website to work on it with them but as far as I know, no solution were found. The only workaround that we found is that a simple restart of the docbase will cleanup these errors from the docbase log file and they will not appear afterwards but that’s pretty annoying to restart the docbase while it is actually working properly (jobs are running, dfc clients are OK, aso…) just because there is one error message printed every 5 minutes in the log file that is flooding our monitoring tool.

 

The problem with this error is that it can happen frequently if the Network isn’t really reliable, if the Database Listener isn’t always responsive or if anything else prevent Documentum from reaching the Database while it is doing something with the dm_docbase_config objects… Something that we didn’t try yet is to re-initialize the Content Server, to see if it can help to restore a proper log file. I think I’m gonna try that next time this issue occurs!

Edit: Re-initialize the Content Server isn’t helping :(.

 

Cet article Documentum story – Attempt to fetch object with handle 3c failed est apparu en premier sur Blog dbi services.

2 x ODA X6-2S + Dbvisit Standby: Easy DR in SE

Fri, 2016-10-21 11:28

What’s common with Standard Edition, simplicity, reliability, high performance, and affordable price?
Dbvisit standby can be an answer because it brings Disaster Recovery to Standard Edition without adding complexity
ODA Lite (the new X6-2S and 2M) is another answer because you can run Standard Edition in those new appliance.
So it makes sense to bring them together, this is what I did recently at a customer.

I’ll not cover the reasons and the results here as this will be done later. Just sharing a few tips to set-up the following configuration: two ODA X6-2S runnimg 12c Standard Edition databases, protected by Dbvisit standby over two datacenters.

ODA repository

ODA X6 comes with a new interface to provision databases from command line (odacli) or GUI (https://oda:7093/mgmt/index.html). It’s a layer over the tools we usually use: it calls dbca in behind. What it does in addition is to log what has been done in a Java DB repository.

What is done is logged in the opt/oracle/dcs/log/dcs-agent.log:
2016-10-13 15:33:59,816 DEBUG [Database Creation] [] c.o.d.a.u.CommonUtils: run: cmd= '[su, -, oracle, -c, export PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin:/u01/app/oracle/product/12.1.0.2/dbhome_2/bin; export ORACLE_SID=MYNEWDB; export ORACLE_BASE=/u01/app/oracle; export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_2; export PWD=******** /u01/app/oracle/product/12.1.0.2/dbhome_2/bin/dbca -createDatabase -silent -gdbName MYNEWDB.das.ch -sid MYNEWDB -sysPassword ******* -systemPassword ******* -dbsnmpPassword ******* -asmSysPassword ******* -storageType ASM -datafileJarLocation /u01/app/oracle/product/12.1.0.2/dbhome_2/assistants/dbca/templates -emConfiguration DBEXPRESS -databaseConfType SINGLE -createAsContainerDatabase false -characterSet WE8MSWIN1252 -nationalCharacterSet AL16UTF16 -databaseType MULTIPURPOSE -responseFile NO_VALUE -templateName seed_noncdb_se2016-10-13_15-33-59.0709.dbc -initParams "db_recovery_file_dest_size=174080,db_unique_name=MYNEWDB" -recoveryAreaDestination /u03/app/oracle/fast_recovery_area/]'

Do I like it? Actually I don’t for two reasons. First reason is that I don’t want to learn new syntax every year. I know CREATE DATABASE from decades, I know DBCA for years. I just prefer to use those.
The second reason is that if you want to add a layer on something, you need to provide at least the same functionality and the same quality than the tool you call in behind. If you provide a command to create a database, then you must provide a command to delete it, even if the previous creation has failed. I’ve created a database which creation failed. The reason was that I changed the listener port, but the template explicitly sets local_listener to port 1521. Fortunately it calls DBCA and I know where are the logs. So my ODA repository has a database in failed status. The problem is that you can’t drop it (it doesn’t exist for DBCA) and you cannot re-create it (it exists for ODA). I’m not a developer, but when I write code I try to manage exceptions. At least they must implement a ‘force’ mode where errors are ignored when deleting something that does not exist.

So if you have the same problem, here is what I did:

  • Open a SR in the hope that they understand there’s something to fix in their code without asking me all log files to upload
  • create a database with same name, directly with DBCA, then drop it with ODACLI

Finally, My Workaround works and Their Oracle Support came with two solutions: create the database with another name or re-image the ODA!

But, when it doesn’t fail, the creation is very fast: from templates with datafiles, and datafiles in those very fast NVMe SSDs.

Create the standby

I don’t like this additional layer, but I have the feeling that it’s better than the ODA repository knows about my databases. The standby database is created with Dbvisit interface (I’m talking about real user friendly interface there, where errors are handled and you even have the possibility to resume a creation that failed). How to make it go to the ODA repository?

I see 3 possibilities.

The odacli has a “–register-database” option to register an already create database. But that does probably too much because it was designed to register databases created on previous ODAs with oakcli.

The odacli has a “–instanceonly” option which is there to register a standby database that will be created later with RMAN duplicate for example. Again this does too much as it creates an instance. I tried it and didn’t have the patience to make it work. When ODACLI encounters a problem, it doesn’t explain what’s wrong, but just show the command line help.

Finally what I did is create a database with ODACLI and the drop it (outside of ODACLI). This is ugly, but its the only way I got something where I understand exactly what is done. This is where I encountered the issue above, so my workflow was actually: create from ODACLI -> fails -> drom from DBCA -> re-create from ODACLI -> success -> drop

I didn’t drop it from DBCA because I wanted to keep the entry in ORATAB. I did it from RMAN:

RMAN> startup force dba mount
RMAN> drop database including backups noprompt;

Then, no problem to create the standby from Dbvisit GUI

Create a filesystem

I’ve created the database directly in ASM. I don’t see any reason to create an ACFS volume for them, especially for Standard Edition where you cannot use ACFS snapshots. It’s just a performance overhead (and with those no-latency disks, any CPU overhead counts) and a risk to remove a datafile as they are exposed in a filesystem with no reason for it.

However, Dbvisit needs a folder where to store the archived logs that are shipped to the standby. I can create a folder in in local filesystem, but I preferred to to create an ACFS filesystem for it.
I did it from ODACLI:


odacli create-dbstorage --dataSize 200 -n DBVISIT -r ACFS

This creates a 200GB filesystem mounted as /u02/app/oracle/oradata/DBVISIT/

Who starts the database?

Dbvisit comes with a scheduler that can start the databases in the required mode. But in ODA the resources are managed by Grid Infrastructure. So after creating the standby database you must modify its mount mode:

srvctl modify database -d MYNEWDB -startoption mount

Don’t forget to change the mount modes after a switchover or failover.

This can be scripted with something like: srvctl modify database -db $db -startoption $(/opt/dbvisit/standby/dbv_oraStartStop status $db| awk '/^Regular Database/{print "OPEN"}/^Standby Database/{print "MOUNT"}')

Keep it simple and test it

ODA is simple if you do what it has been designed for: run the database versions that are certified (currenty 11.2.0.4 and 12.1.0.2) adn don’t try to customize the configuration. Always test the switchover, so that you can rely on the protection. It’s easy with Dbvisit standby, either from GUI of command line. And be sure that your network can keep up with the redo rate. Again, this is easy to check from the GUI. Here is an exemple when testing the migration with Data Pump import:
DbvisitTransferLogSize

From public prices, and before any discount, you can get two ODA X6-2S plus perpetual licences for Oracle Database Standard Edition and Dbvisit standby for less than 90KUSD.
If you need more storage you can double the capacity for about additional 10KUSD for each ODA.
And if you think that ODA may need a DBA sometimes, have a look at our SLAs and you have a reliable and affordable system on your premises to store and process your data.

 

Cet article 2 x ODA X6-2S + Dbvisit Standby: Easy DR in SE est apparu en premier sur Blog dbi services.

Configure easily your Stretch Database

Fri, 2016-10-21 10:07

In this blog, I will present you the new Stretch Database feature in SQL Server 2016. It couples your SQL Server On-Premises database with an Azure SQL Database, allowing to stretch data from one ore more tables to Azure Cloud.
This mechanism offers to use low-cost hard drives available in Azure, instead of fast and expensive local solid state drives. Indeed SQL Database Server resources are solicited during data transfers and during remote queries (and not SQL Server on-premises).

First, you need to enable the “Remote Data Archive” option at the instance level. To verify if the option is enabled:
USE master
GO
SELECT name, value, value_in_use, description from sys.configurations where name like 'remote data archive'

To enable this option at the instance level:

EXEC sys.sp_configure N'remote data archive', '1';
RECONFIGURE;
GO

Now, you have to link your on-premises database with a remote SQL Database server:
Use AdventureWorks2014;
GO
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'masterPa$$w0rd'
GO
CREATE DATABASE SCOPED CREDENTIAL Stretch_cred
WITH IDENTITY = 'dbi' , SECRET = 'userPa$$w0rd' ;
GO
ALTER DATABASE AdventureWorks2014
SET REMOTE_DATA_ARCHIVE = ON
(
SERVER = 'dbisqldatabase.database.windows.net' ,
CREDENTIAL = Stretch_cred
) ;
GO

The process may take some time as it will create a new SQL Database in Azure, linked to your on-premises database. The credential entered to connect to your SQL Database server is defined in SQL Database. Previously you need to secure the credential by a database master key.

To view all the remote databases from your instance:
Select * from sys.remote_data_archive_databases

Now, if you want to migrate one table from your database ([Purchasing].[PurchaseOrderDetail] in my example), proceed as follows:
ALTER TABLE [Purchasing].[PurchaseOrderDetail] SET ( REMOTE_DATA_ARCHIVE ( MIGRATION_STATE = OUTBOUND) ) ;

Of course repeat this process for each table you want to stretch. You can still access to your data during the migration process.

To view all the remote tables from your instance:
Select * from sys.remote_data_archive_tables

To view the batch process of all the data being migrated: (indeed, you can filtrate by the a specific table)
Select * from sys.dm_db_rda_migration_status

It is also to easily migrate your data back:
ALTER TABLE [Purchasing].[PurchaseOrderDetail] SET ( REMOTE_DATA_ARCHIVE ( MIGRATION_STATE = INBOUND ) ) ;

Moreover, you can select rows to migration by using a filter function. Here is an example:
CREATE FUNCTION dbo.fn_stretchpredicate(@column9 datetime)
RETURNS TABLE
WITH SCHEMABINDING
AS
RETURN SELECT 1 AS is_eligible
WHERE @column9 > CONVERT(datetime, '1/1/2014', 101)
GO

Then when enable the data migration, specify the filter function:
ALTER TABLE [Purchasing].[PurchaseOrderDetail] SET ( REMOTE_DATA_ARCHIVE = ON (
FILTER_PREDICATE = dbo.fn_stretchpredicate(ModifiedDate),
MIGRATION_STATE = OUTBOUND
) )

Of course in Microsoft world, you can also use a wizard to set up this feature. The choice is up to you!

 

Cet article Configure easily your Stretch Database est apparu en premier sur Blog dbi services.

Documentum story – Unable to start a new Managed Server in SSL in WebLogic

Fri, 2016-10-21 02:00

Some time ago, I was creating a new Managed Server named msD2-02 on an existing domain of a WebLogic Server 12.1.3.0 created loooong ago and I faced a small issue that I will try to explain in this blog. This Managed Server will be used to host a D2 4.5 Application (Documentum Client) and I created it using the Administration Console, customized it, enabled the SSL with internal SSL Certificates, the SAML2 Single Sign-On, aso…

 

When I wanted to start it for the first time, I get an error showing that the user/password used was wrong… So I tried to recreate the boot.properties file from scratch, setting up the username/password in there and tried again: same error. What to do then? To be sure that the password was correct (even if I was pretty sure), I tried to copy the boot.properties file from another Managed Server and tried again but same result over and over. Therefore I tried a last time removing the boot.properties completely to enter the credentials during the startup:

[weblogic@weblogic_server_01 msD2-02]$ /app/weblogic/domains/DOMAIN/bin/startManagedWebLogic.sh msD2-02 t3s://weblogic_server_01:8443

JAVA Memory arguments: -Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=512m

CLASSPATH=/app/weblogic/Middleware/wlserver/server/lib/jcmFIPS.jar:/app/weblogic/Middleware/wlserver/server/lib/sslj.jar:/app/weblogic/Middleware/wlserver/server/lib/cryptoj.jar::/app/weblogic/Java/jdk1.8.0_45/lib/tools.jar:/app/weblogic/Middleware/wlserver/server/lib/weblogic_sp.jar:/app/weblogic/Middleware/wlserver/server/lib/weblogic.jar:/app/weblogic/Middleware/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/app/weblogic/Middleware/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar:/app/weblogic/Middleware/wlserver/common/derby/lib/derbyclient.jar:/app/weblogic/Middleware/wlserver/common/derby/lib/derby.jar:/app/weblogic/Middleware/wlserver/server/lib/xqrl.jar:/app/weblogic/domains/DOMAIN/lib/LB.jar:/app/weblogic/domains/DOMAIN/lib/LBJNI.jar:

PATH=/app/weblogic/Middleware/wlserver/server/bin:/app/weblogic/Middleware/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/app/weblogic/Java/jdk1.8.0_45/jre/bin:/app/weblogic/Java/jdk1.8.0_45/bin:/app/weblogic/domains/DOMAIN/D2/lockbox:/app/weblogic/domains/DOMAIN/D2/lockbox/lib/native/linux_gcc34_x64:/app/weblogic/Java/jdk1.8.0_45/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/weblogic/bin

***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
Starting WLS with line:
/app/weblogic/Java/jdk1.8.0_45/bin/java -server -Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=512m -Dweblogic.Name=msD2-02 -Djava.security.policy=/app/weblogic/Middleware/wlserver/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/weblogic/Middleware/wlserver/server/lib/cacerts  -Dcom.sun.xml.ws.api.streaming.XMLStreamReaderFactory.woodstox=true -Dcom.sun.xml.ws.api.streaming.XMLStreamWriterFactory.woodstox=true -Djava.io.tmpdir=/app/weblogic/tmp/DOMAIN/msD2-02 -Ddomain.home=/app/weblogic/domains/DOMAIN -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.protocolVersion=TLS1 -Dweblogic.security.disableNullCipher=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.security.allowCryptoJDefaultJCEVerification=true -Dweblogic.nodemanager.ServiceEnabled=true  -Djava.endorsed.dirs=/app/weblogic/Java/jdk1.8.0_45/jre/lib/endorsed:/app/weblogic/Middleware/wlserver/../oracle_common/modules/endorsed  -da -Dwls.home=/app/weblogic/Middleware/wlserver/server -Dweblogic.home=/app/weblogic/Middleware/wlserver/server   -Dweblogic.management.server=t3s://weblogic_server_01:8443  -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true  weblogic.Server
<Jun 14, 2016 11:52:43 AM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Jun 14, 2016 11:52:43 AM UTC> <Notice> <WebLogicServer> <BEA-000395> <The following extensions directory contents added to the end of the classpath:
/app/weblogic/domains/DOMAIN/lib/LB.jar:/app/weblogic/domains/DOMAIN/lib/LBJNI.jar.>
<Jun 14, 2016 11:52:44 AM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 25.45-b02 from Oracle Corporation.>
<Jun 14, 2016 11:52:44 AM UTC> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:
<Jun 14, 2016 11:52:54 AM UTC> <Warning> <Security> <BEA-090924> <JSSE has been selected by default, since the SSLMBean is not available.>
<Jun 14, 2016 11:52:54 AM UTC> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>
<Jun 14, 2016 11:52:54 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /app/weblogic/Middleware/wlserver/server/lib/cacerts.>
<Jun 14, 2016 11:52:54 AM UTC> <Info> <Management> <BEA-141298> <Could not register with the Administration Server: java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading the input.; nested exception is:
        javax.net.ssl.SSLException: Error using PKIX CertPathBuilder.>
<Jun 14, 2016 11:52:54 AM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.3.0.0  Wed May 21 18:53:34 PDT 2014 1604337 >
<Jun 14, 2016 11:52:55 AM UTC> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>
<Jun 14, 2016 11:52:55 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /app/weblogic/Middleware/wlserver/server/lib/cacerts.>
<Jun 14, 2016 11:52:55 AM UTC> <Alert> <Management> <BEA-141151> <The Administration Server could not be reached at https://weblogic_server_01:8443.>
<Jun 14, 2016 11:52:55 AM UTC> <Info> <Configuration Management> <BEA-150018> <This server is being started in Managed Server independence mode in the absence of the Administration Server.>
<Jun 14, 2016 11:52:55 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>
<Jun 14, 2016 11:52:55 AM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>
<Jun 14, 2016 11:52:55 AM UTC> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>
<Jun 14, 2016 11:52:55 AM UTC> <Notice> <Log Management> <BEA-170019> <The server log file /app/weblogic/domains/DOMAIN/servers/msD2-02/logs/msD2-02.log is opened. All server side log events will be written to this file.>
<Jun 14, 2016 11:52:57 AM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jun 14, 2016 11:52:57 AM UTC> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias alias_cert from the JKS keystore file /app/weblogic/domains/DOMAIN/certs/identity.jks.>
<Jun 14, 2016 11:52:57 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the JKS keystore file /app/weblogic/domains/DOMAIN/certs/trust.jks.>
<Jun 14, 2016 11:52:58 AM UTC> <Critical> <Security> <BEA-090403> <Authentication for user weblogic denied.>
<Jun 14, 2016 11:52:58 AM UTC> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 6 exceptions.  They are:
1. weblogic.security.SecurityInitializationException: Authentication for user weblogic denied.
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.security.SecurityService
3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.RemoteNamingService errors were found
4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.RemoteNamingService
5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.t3.srvr.T3InitializationService errors were found
6. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.t3.srvr.T3InitializationService

A MultiException has 6 exceptions.  They are:
1. weblogic.security.SecurityInitializationException: Authentication for user weblogic denied.
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.security.SecurityService
3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.RemoteNamingService errors were found
4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.RemoteNamingService
5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.t3.srvr.T3InitializationService errors were found
6. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.t3.srvr.T3InitializationService

        at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:88)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:269)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.security.SecurityInitializationException: Authentication for user weblogic denied.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:1023)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.postInitialize(CommonSecurityServiceManagerDelegateImpl.java:1131)
        at weblogic.security.service.SecurityServiceManager.postInitialize(SecurityServiceManager.java:943)
        at weblogic.security.SecurityService.start(SecurityService.java:159)
        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
        Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090303]Authentication Failed: User weblogic weblogic.security.providers.authentication.LDAPAtnDelegateException: [Security:090295]caught unexpected exception
        at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:257)
        at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Truncated. see log file for complete stacktrace
>
<Jun 14, 2016 11:52:58 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>
<Jun 14, 2016 11:52:58 AM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>
<Jun 14, 2016 11:52:58 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>

 

As you can see above, the WebLogic Managed Server is able to retrieve and read the SSL Keystores (identity and trust) so this apparently isn’t the issue which seems to be linked to a wrong username/password. Strange isn’t it?

 

All other Managed Servers are working perfectly, the applications are accessible in HTTPS, we can see the status of the servers via WLST/AdminConsole, aso… But this specific Managed Server isn’t able to start… After some reflexion, I thought at the Embedded LDAP! This is a completely new Managed Server and I tried to start it directly in HTTPS. What if this Managed Server isn’t able to authenticate the user weblogic because this user doesn’t exist in the Embedded LDAP of the Managed Server? Indeed during the first start, a Managed Server will try to automatically replicate the Embedded LDAP from the AdminServer which contains the primary Embedded LDAP. Just for information, we usually create a bunch of Managed Servers for Documentum during the domain creation and therefore all these Managed Servers are usually started at least 1 time in HTTP before setting up the SSL in the Domain: that’s the main difference between the existing Managed Servers and the new one and therefore I dug deeper in this direction.

 

To test my theory, I tried to replicate the Embedded LDAP manually. In case you don’t know how to do it, please take a look at this blog which explains that in details: click here. After doing that, the Managed Server msD2-02 was indeed able to start because it was able to authenticate the user weblogic but that doesn’t explain why the Embedded LDAP wasn’t replicated automatically in the first place…

 

So I checked more deeply the logs and actually the first strange message during startup is always the same:

<Jun 14, 2016 11:52:54 AM UTC> <Info> <Management> <BEA-141298> <Could not register with the Administration Server: java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading the input.; nested exception is:
        javax.net.ssl.SSLException: Error using PKIX CertPathBuilder.>

 

As said previously, all components are setup in HTTPS and only HTTPS. Therefore all communications are using an SSL Certificate. For this customer, we weren’t using a Self-Signed Certificate but a Certificate Signed by an internal Certificate Authority. As shown in the Info message, the Managed Server wasn’t able to register with the AdminServer with an SSL Exception… Therefore I checked the SSL Certificate, the Root and Gold Certificate Authority too but for me everything was working properly. The Admin Console is accessible in HTTPS, all Applications are accessible, the status of the Managed Servers are visible in the Administration Console and via WLST which shows that they are able to communicate internally too, aso… So what could be wrong? Well after checking the startup command of the Managed Server (and actually it is also mentioned in the startup logs), I found the following:

[weblogic@weblogic_server_01 servers]$ ps -ef | grep msD2-02 | grep -v grep
weblogic 31313     1  0 14:34 pts/2    00:00:00 /bin/sh ../startManagedWebLogic.sh msD2-02 t3s://weblogic_server_01:8443
weblogic 31378 31315 26 14:34 pts/2    00:00:35 /app/weblogic/Java/jdk1.8.0_45/bin/java -server
    -Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=512m -Dweblogic.Name=msD2-02
    -Djava.security.policy=/app/weblogic/Middleware/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true
    -Dweblogic.security.SSL.trustedCAKeyStore=/app/weblogic/Middleware/wlserver/server/lib/cacerts
    -Dcom.sun.xml.ws.api.streaming.XMLStreamReaderFactory.woodstox=true -Dcom.sun.xml.ws.api.streaming.XMLStreamWriterFactory.woodstox=true
    -Djava.io.tmpdir=/app/weblogic/tmp/DOMAIN/msD2-02 -Ddomain.home=/app/weblogic/domains/DOMAIN -Dweblogic.nodemanager.ServiceEnabled=true
    -Dweblogic.security.SSL.protocolVersion=TLS1 -Dweblogic.security.disableNullCipher=true -Djava.security.egd=file:///dev/./urandom
    -Dweblogic.security.allowCryptoJDefaultJCEVerification=true -Dweblogic.nodemanager.ServiceEnabled=true
    -Djava.endorsed.dirs=/app/weblogic/Java/jdk1.8.0_45/jre/lib/endorsed:/app/weblogic/Middleware/wlserver/../oracle_common/modules/endorsed
    -da -Dwls.home=/app/weblogic/Middleware/wlserver/server -Dweblogic.home=/app/weblogic/Middleware/wlserver/server
    -Dweblogic.management.server=t3s://weblogic_server_01:8443 -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server

 

What is this JVM parameter? Why does WebLogic defines a specific cacerts for this Managed Server and isn’t using the default one (included in Java)? Something is strange with this startup command…So I checked all other WebLogic Server processes and apparently ALL Managed Servers include this custom cacerts while the AdminServer doesn’t… Is that a bug?! Even if it makes sense to create a custom cacerts for WebLogic only, then why the AdminServer isn’t using it? This fact doesn’t make any sense and this is why we are facing this issue:
– All Managed Servers are using: /app/weblogic/Middleware/wlserver/server/lib/cacerts
– The AdminServer is using: /app/weblogic/Java/jdk1.8.0_45/jre/lib/security/cacerts

 

After checking the different startup scripts, it appears that this is define in the file startManagedServer.sh. Therefore this JVM parameter is only used by the Managed Server and therefore it is apparently a choice from Oracle (or something that has been forgotten…) to only start the Managed Servers with this option and not the AdminServer… Using different cacerts means that the SSL Certificates trusted by Java (default one) will be trusted by the AdminServer but it will not be the case for the Managed Servers. In our setup, we always add the Root and Gold Certificates (SSL Chain) in the default Java cacerts because it is the one used to allow the setup of our Domain and our Applications in SSL. This is working properly but that isn’t enough to allow the Managed Servers to start properly: you also need to take care of this second cacerts and that’s the reason why the new Managed Server wasn’t able to register to the AdminServer and therefore not able to replicate the Embedded LDAP.

 

So how to correct that? First, let’s export the Certificate Chain from the identity keystore and import that into the WebLogic cacerts too:

[weblogic@weblogic_server_01 servers]$ keytool -export -v -alias root_ca -file rootCA.der -keystore /app/weblogic/domains/DOMAIN/certs/identity.jks
[weblogic@weblogic_server_01 servers]$ keytool -export -v -alias gold_ca -file goldCA.der -keystore /app/weblogic/domains/DOMAIN/certs/identity.jks
[weblogic@weblogic_server_01 servers]$
[weblogic@weblogic_server_01 servers]$ keytool -import -v -trustcacerts -alias root_ca -file rootCA.der -keystore /app/weblogic/Middleware/wlserver/server/lib/cacerts
Enter keystore password:
[weblogic@weblogic_server_01 servers]$ keytool -import -v -trustcacerts -alias gold_ca -file goldCA.der -keystore /app/weblogic/Middleware/wlserver/server/lib/cacerts
Enter keystore password:

 

After doing that, you just have to remove the Embedded LDAP of this Managed Server to reinitialize it using the same steps as before but just do not copy the ldap from the AdminServer since we need to ensure that the automatic replication is working now. Then start the Managed Server one last time and verify that the replication is happening properly and therefore if the Managed Server is able to start or not. For me, everything was now working properly, so that’s a victory! :)

 

Cet article Documentum story – Unable to start a new Managed Server in SSL in WebLogic est apparu en premier sur Blog dbi services.

Enterprise Manager 13.1.0.0 does not display correct values for memory

Thu, 2016-10-20 08:03

I recently had problems with Enterprise Manager 13.1.0.0, receiving such alerts:

EM Event Critical hostname Memory Utilization is 93,205 % crossed warning (80%) or critical (90%)

When we have a look at the EM 13c console for the host:

me1

On the system the free -m command displays:

oracle@host:~/24437699/ [agent13c] free -m
             total       used       free     shared    buffers     cached
Mem:         48275      44762       3512          0        205      37483
-/+ buffers/cache:       7073      41201
Swap:         8189       2397       5791

Em 13c does not take into account the buffer / cached component.

In fact the memory calculation has changed from EM 12.1.0.5 and EM 13.1.0.0.  According to Metalink Note 2144976.1:

“While the total Memory available in the host target is displayed correctly after applying the latest PSU # 23030165 (Agent-Side 13.1.0.0.160429), the formula used for Memory Utilization is (100.0 * (realMem-freeMem) / realMem) and does not consider Buffers / Cached component for the calculation.”

To solve the problem we have to patch the OMS and the different agents:

For the oms: use the patch 23134365

For the agents : use the patch 24437699

Watch out, when you want to apply the 23134365 patch for oms, we have to install the latest version of omspatcher. We download  Patch 19999993 of Release 13.1.0.0.0 from MOS.

We backup the OMSPatcher directory in the $ORACLE_HOME oms13c environment:

oracle:OMS_HOME:/ [oms13c] mv OMSPatcher/ OMSPatcher_save

then we copy and unzip the p19999993_131000_Generic.zip from the $ORACLE_HOME directory:

oracle:$OMS_HOME/ [oms13c] unzip p19999993_131000_Generic.zip
Archive:  p19999993_131000_Generic.zip
   creating: OMSPatcher/
   creating: OMSPatcher/oms/
  inflating: OMSPatcher/oms/generateMultiOMSPatchingScripts.pl
   creating: OMSPatcher/jlib/
  inflating: OMSPatcher/jlib/oracle.omspatcher.classpath.jar
  inflating: OMSPatcher/jlib/oracle.omspatcher.classpath.unix.jar
  inflating: OMSPatcher/jlib/omspatcher.jar
  inflating: OMSPatcher/jlib/oracle.omspatcher.classpath.windows.jar
   creating: OMSPatcher/scripts/
   creating: OMSPatcher/scripts/oms/
   creating: OMSPatcher/scripts/oms/oms_child_scripts/
  inflating: OMSPatcher/scripts/oms/oms_child_scripts/omspatcher_wls.bat
  inflating: OMSPatcher/scripts/oms/oms_child_scripts/omspatcher_jvm_discovery
  inflating: OMSPatcher/scripts/oms/oms_child_scripts/omspatcher_jvm_discovery.bat
  inflating: OMSPatcher/scripts/oms/oms_child_scripts/omspatcher_wls
  inflating: OMSPatcher/scripts/oms/omspatcher
  inflating: OMSPatcher/scripts/oms/omspatcher.bat
  inflating: OMSPatcher/omspatcher
   creating: OMSPatcher/wlskeys/
  inflating: OMSPatcher/wlskeys/createkeys.cmd
  inflating: OMSPatcher/wlskeys/createkeys.sh
  inflating: OMSPatcher/omspatcher.bat
  inflating: readme.txt
  inflating: PatchSearch.xml

We check the OMSPatcher version:

oracle:/ [oms13c] ./omspatcher version
OMSPatcher Version: 13.6.0.0.1
OPlan Version: 12.1.0.2.2
OsysModel build: Wed Oct 14 06:21:23 PDT 2015
 
OMSPatcher succeeded.

We download from Metalink the p23134265_131000_Generic-zip file, and we run:

oracle@host:/home/oracle/23134365/ [oms13c] omspatcher apply -analyze
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
OMSPatcher version : 13.6.0.0.1
OUI version        : 13.6.0.0.0
Running from       : /u00/app/oracle/product/13.1.0.0/middleware
Log file location  : /u00/app/oracle/product/13.1.0.0/middleware/
cfgtoollogs/omspatcher/opatch2016-09-26_11-06-30AM_1.log
 
OMSPatcher log file: /u00/app/oracle/product/13.1.0.0/middleware/
cfgtoollogs/omspatcher/23134365/omspatcher_2016-09-26_11-06-34AM_analyze.log
 
Please enter OMS weblogic admin server URL(t3s://hostname:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>
 
Configuration Validation: Success
 
Running apply prerequisite checks for sub-patch(es) "23134365" 
and Oracle Home "/u00/app/oracle/product/13.1.0.0/middleware"...
Sub-patch(es) "23134365" are successfully analyzed for Oracle Home 
"/u00/app/oracle/product/13.1.0.0/middleware"

Complete Summary
================

OMSPatcher succeeded.

We stop the oms and we run:

oracle@hostname:/home/oracle/23134365/ [oms13c] omspatcher apply
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.
 
OMSPatcher version : 13.6.0.0.1
OUI version        : 13.6.0.0.0
Running from       : /u00/app/oracle/product/13.1.0.0/middleware
 
Please enter OMS weblogic admin server URL(t3s://hostname:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>
 
Configuration Validation: Success
…

OMSPatcher succeeded.

We finally restart the OMS:

oracle@hostname:/home/oracle/ [oms13c] emctl start oms

Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Already Started
BI Publisher Server is Up

 

Now we apply the patch to the agents:

After downloaded and unzipped the p24437699_131000_Generic.zip, we stop the management agent and we run:

oracle@hostname:/home/oracle/24437699/ [agent13c] opatch apply
Oracle Interim Patch Installer version 13.6.0.0.0
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
 
Oracle Home       : /u00/app/oracle/product/13.1.0.0/agent/agent_13.1.0.0.0
Central Inventory : /u00/app/oraInventory
OPatch version    : 13.6.0.0.0
OUI version       : 13.6.0.0.0

OPatch detects the Middleware Home as "/u00/app/oracle/product/13.1.0.0/agent"
 
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   24437699
 
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Backing up files...
Applying interim patch '24437699' to 
OH '/u00/app/oracle/product/13.1.0.0/agent/agent_13.1.0.0.0'
 
Patching component oracle.sysman.top.agent, 13.1.0.0.0...
Patch 24437699 successfully applied.
 
OPatch succeeded.

Finally we restart the agent with the emctl start agent command.

After the patches have been applied, the memory used displayed is correct:

me2

me3

And we do not receive critical alerts anymore :=)

 

 

Cet article Enterprise Manager 13.1.0.0 does not display correct values for memory est apparu en premier sur Blog dbi services.

Documentum story – Replicate an Embedded LDAP manually in WebLogic

Thu, 2016-10-20 02:00

In this blog, I will talk about the WebLogic Embedded LDAP. This LDAP is created by default on all AdminServers and Managed Servers of any WebLogic installation. The AdminServer always contains the Primary Embedded LDAP and all other Servers are synchronized with this one. This Embedded LDAP is the default security provider database for the WebLogic Authentication, Authorization, Credential Mapping and Role Mapping providers: it usually contains the WebLogic users, groups, and some other stuff like the SAML2 setup, aso… So basically a lot of stuff configured under the “security realms” in the WebLogic Administration Console. This LDAP is based on files that are stored under “$DOMAIN_HOME/servers/<SERVER_NAME>/data/ldap/”.

 

Normally the Embedded LDAP is automatically replicated from the AdminServer to the Managed Servers during startup but this can fail for a few reasons:

  • AdminServer not running
  • Problems in the communications between the AdminServer and Managed Servers
  • aso…

 

Oracle usually recommend to use an external RDBMS Security Store instead of the Embedded LDAP but not all information are stored in the RDBMS and therefore the Embedded LDAP is always used, at least for a few things. More information on this page: Oracle WebLogic Server Documentation.

 

So now in case the automatic replication isn’t working properly, for any reason, or if a manual replication is needed, how can it be done? Well that’s pretty simple and I will explain that below. I will also use a home made script in order to quickly and efficiently start/stop one, several or all WebLogic components. If you don’t have such script available, then please adapt the steps below to manually stop and start all WebLogic components.

 

So first you need to stop all components:

[weblogic@weblogic_server_01 ~]$ $DOMAIN_HOME/bin/startstop stopAll
  ** Managed Server msD2-01 stopped
  ** Managed Server msD2Conf-01 stopped
  ** Managed Server msDA-01 stopped
  ** Administration Server AdminServer stopped
  ** Node Managed NodeManager stopped
[weblogic@weblogic_server_01 ~]$ ps -ef | grep weblogic
[weblogic@weblogic_server_01 ~]$

 

Once this is done, you need to retrieve the list of all Managed Servers installed/configured in this WebLogic Domain for which a manual replication is needed. For me, it is pretty simple, they are printed above in the start/stop command but otherwise you can find them like that:

[weblogic@weblogic_server_01 ~]$ cd $DOMAIN_HOME/servers
[weblogic@weblogic_server_01 servers]$ ls | grep -v "domain_bak"
AdminServer
msD2-01
msD2Conf-01
msDA-01

 

Now that you have the list, you can proceed with the manual replication for each and every Managed Server. First backup the Embedded LDAP and then replicate it from the Primary (in the AdminServer as explained above):

[weblogic@weblogic_server_01 servers]$ current_date=$(date "+%Y%m%d")
[weblogic@weblogic_server_01 servers]$ 
[weblogic@weblogic_server_01 servers]$ mv msD2-01/data/ldap msD2-01/data/ldap_bck_$current_date
[weblogic@weblogic_server_01 servers]$ mv msD2Conf-01/data/ldap msD2Conf-01/data/ldap_bck_$current_date
[weblogic@weblogic_server_01 servers]$ mv msDA-01/data/ldap msDA-01/data/ldap_bck_$current_date
[weblogic@weblogic_server_01 servers]$ 
[weblogic@weblogic_server_01 servers]$ cp -R AdminServer/data/ldap msD2-01/data/
[weblogic@weblogic_server_01 servers]$ cp -R AdminServer/data/ldap msD2Conf-01/data/
[weblogic@weblogic_server_01 servers]$ cp -R AdminServer/data/ldap msDA-01/data/

 

When this is done, just start all WebLogic components again:

[weblogic@weblogic_server_01 servers]$ $DOMAIN_HOME/bin/startstop startAll
  ** Node Manager NodeManager started
  ** Administration Server AdminServer started
  ** Managed Server msDA-01 started
  ** Managed Server msD2Conf-01 started
  ** Managed Server msD2-01 started

 

And if you followed these steps properly, the Managed Servers will now be able to start normally with a replicated Embedded LDAP containing all recent changes coming from the Primary Embedded LDAP.

 

Cet article Documentum story – Replicate an Embedded LDAP manually in WebLogic est apparu en premier sur Blog dbi services.

Datawarehouse ODS load is fast and easy in Enterprise Edition

Wed, 2016-10-19 14:56

In a previous post, tribute to transportable tablespaces (TTS), I said that TTS is also used to move data quickly from operational database to a datawarehouse ODS. For sure, you don’t transport directly from the production database because TTS requires that the tablespace is read only. But you can transport from a snapshot standby. Both features (transportable tablespaces and Data Guard snapshot standby) are free in Enterprise Edition without option. Here is an exemple to show that it’s not difficult to automate

I have a configuration with the primary database “db1a”

DGMGRL> show configuration
 
Configuration - db1
 
Protection Mode: MaxPerformance
Members:
db1a - Primary database
db1b - Physical standby database
 
Fast-Start Failover: DISABLED
 
Configuration Status:
SUCCESS (status updated 56 seconds ago)
 
DGMGRL> show database db1b
 
Database - db1b
 
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 0 Byte/s
Real Time Query: ON
Instance(s):
db1
 
Database Status:
SUCCESS

I’ve a few tables in the tablespace USERS and this is what I want to transport to ODS database:

SQL> select segment_name,segment_type,tablespace_name from user_segments;
 
SEGMENT_NAME SEGMENT_TY TABLESPACE
------------ ---------- ----------
DEPT TABLE USERS
EMP TABLE USERS
PK_DEPT INDEX USERS
PK_EMP INDEX USERS
SALGRADE TABLE USERS

Snapshot standby

With Data Guard it is easy to open temporarily the standby database. Just convert it to a snapshot standby with a simple command:


DGMGRL> connect connect system/oracle@//db1b
DGMGRL> convert database db1b to snapshot standby;
Converting database "db1b" to a Snapshot Standby database, please wait...
Database "db1b" converted successfully

Export

Here you can start to do some Extraction/Load but better to reduce this window where the standby is not in sync. The only thing we will do is export the tablespace in the fastest way: TTS.

First, we put the USERS tablespace in read only:

SQL> connect system/oracle@//db1b
Connected.
 
SQL> alter tablespace users read only;
Tablespace altered.

and create a directory to export metadata:

SQL> create directory TMP_DIR as '/tmp';
Directory created.

Then export is easy

SQL> host expdp system/oracle@db1b transport_tablespaces=USERS directory=TMP_DIR
Starting "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01": system/********@db1b transport_tablespaces=USERS directory=TMP_DIR
 
Processing object type TRANSPORTABLE_EXPORT/TABLE_STATISTICS
Processing object type TRANSPORTABLE_EXPORT/STATISTICS/MARKER
Processing object type TRANSPORTABLE_EXPORT/PLUGTS_BLK
Processing object type TRANSPORTABLE_EXPORT/POST_INSTANCE/PLUGTS_BLK
Processing object type TRANSPORTABLE_EXPORT/TABLE
Processing object type TRANSPORTABLE_EXPORT/CONSTRAINT/CONSTRAINT
Master table "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_TRANSPORTABLE_01 is:
/tmp/expdat.dmp
******************************************************************************
Datafiles required for transportable tablespace USERS:
/u02/oradata/db1/users01.dbf
Job "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01" successfully completed at Wed Oct 19 21:03:36 2016 elapsed 0 00:00:52

I’ve the metadata in /tmp/expdat.dmp and the data in /u02/oradata/db1/users01.dbf. I copy this datafile directly in his destination for the ODS database:

[oracle@VM118 ~]$ cp /u02/oradata/db1/users01.dbf /u02/oradata/ODS/users01.dbf

This is physical copy, which is the fastest data movement we can do.

I’m ready to import it into my ODA database, but I can already re-sync the standby database because I extracted everything I wanted.

Re-sync the physical standby

DGMGRL> convert database db1b to physical standby;
Converting database "db1b" to a Physical Standby database, please wait...
Operation requires shut down of instance "db1" on database "db1b"
Shutting down instance "db1"...
Connected to "db1B"
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires start up of instance "db1" on database "db1b"
Starting instance "db1"...
ORACLE instance started.
Database mounted.
Connected to "db1B"
Continuing to convert database "db1b" ...
Database "db1b" converted successfully
DGMGRL>

The duration depends on the time to flashback the changes (and we did no change here as we only exported) and the time to apply the redo stream generated since the convert to snapshot standby (which duration has been minimized to its minimum).

This whole process can be automated. We did that at several customers and it works well. No need to change anything unless you have new tablespaces.

Import

Here is the import to the ODS database and I rename the USERS tablespace to ODS_USERS:

SQL> host impdp system/oracle transport_datafiles=/u02/oradata/db2B/users02.dbf directory=TMP_DIR remap_tablespace=USERS:ODS_USERS
Master table "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_01" successfully loaded/unloaded
Starting "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_01": system/******** transport_datafiles=/u02/oradata/ODS/users01.dbf directory=TMP_DIR remap_tablespace=USERS:ODS_USERS
Processing object type TRANSPORTABLE_EXPORT/PLUGTS_BLK
Processing object type TRANSPORTABLE_EXPORT/TABLE
Processing object type TRANSPORTABLE_EXPORT/CONSTRAINT/CONSTRAINT
Processing object type TRANSPORTABLE_EXPORT/TABLE_STATISTICS
Processing object type TRANSPORTABLE_EXPORT/STATISTICS/MARKER
Processing object type TRANSPORTABLE_EXPORT/POST_INSTANCE/PLUGTS_BLK
Job "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_01" completed with 3 error(s) at Wed Oct 19 21:06:18 2016 elapsed 0 00:00:10

Everything is there. You have all your data in ODS_USERS. You can have other data/code in this database. Only the ODS_USERS tablespace have to be dropped to be re-imported. You can have your staging tables here adn even permanent tables.

12c pluggable databases

In 12.1 it is even easier because the multitenant architecture gives the possibility to transport the pluggable databases in one command, through file copy or database links. It is even faster because metadata are transported physically with the PDB SYSTEM tablespace. I said multitenant architecture here, and didn’t mention any option. Multitenant option is needed only if you want multiple PDBs managed by the same instance. But if you want the ODS database to be an exact copy of the operational database, then you don’t need any option to unplug/plug.

In 12.1 you need to put the source in read only, so you still need a snapshto standby. And from my test, there’s no problem to convert it back to physical standby after a PDB has been unplugged. In next release, we may not need a standby because it has been announced that PDB can be cloned online.

I’ll explain the multitenant features available without any option (in 12c current and next release) at Oracle Geneva office on 23rd of November:

CaptureBreakfastNov23
Do not hesitate to register by e-mail.

 

Cet article Datawarehouse ODS load is fast and easy in Enterprise Edition est apparu en premier sur Blog dbi services.

Feedback on my session at Oracle Open World 2016

Wed, 2016-10-19 11:00

I was a speaker at Oracle Open World and received the feedback and demographic data. this session took place on the Sunday, which is the User Group Forum day about Multitenant, defining what is the multitenant architecture and which features it brings to us even wen we dont’ have the multitenant option. Unfortunately, I cannot upload the slides before the next 12c release is available. If you missed the session or want to hear it in my native language, I’ll give it in Geneva on 23rd of November at Oracle Switzerland office.

Here is the room when we was setting up de demo on my laptop, but from demographic statistics below 84 people attended (or planned to attend) my session.

2016-09-18 10.28.07

Feedback survey

Depending on the conferences, the percentage of people that fill the feedback goes from low to very low. Here 6 people gave feedback wich is 7% or the attendees:

Number of Respondents: 6
Q1: How would you rate the content of the session? (select a rating of 1 to 3, 3 being the best): 2.67
Q2: How would you rate the speaker(s) of the session? (select a rating of 1 to 3, 3 being the best): 2.83
Q3: Overall, based on content and speakers I would rate this session as (select a rating of 1 to 3, 3 being the best): 2.67

Thanks for this. But most important is the quality than the quantity. I received only one comment and this one is very important for me because it can help me to improve my slide:

Heavy accent was difficult to understand at times where I lost interest/concentration. Ran through slides too quick (understanding time constraints). Did not allow image capturing (respected). Did provide examples which was nice. Advised slides will be downloadable…to be seen.

The accent is not a surprise. It’s an international event a lot of people coming from all around the world may have accent that is difficult to understand. I would love to speak English more clearly but I know that my French accent is there, and my lack of vocabulary as well. That will be hard to change. But the remark about the slides is very pertinent. I usually put lot of material in my presentations: lot of slides, lot of texts, lot of demos. My idea is that you don’t need to read all what is written. It is there to read it later when you download the slides (I expected the 12cR2 to be fully available for OOW when I prepared the slides). And it’s also there in case my live demos fails so that I’ve the info on the slides, but I usually skip them quickly when all was seen in the demo.

But thanks to this comment, I understand that reading the slides is important when you don’t get what I say, and having too much text makes it difficult to follow. For future presentations, I’ll remove text from slides and put it as powerpoint presenter notes, made available in the pdf.

So thanks for the one that has written this comment. I’ll improve that. And don’t hesitate to ping me to know when the slides can be downloadable, and maybe I can already share a few ones.

Demographic data

Open World gives some demographic data about attendees. As at the Sunday Session you don’t have to scan you badge, I suppose it’s about people that registered and may not have been there. But intention counts ;)

About countries: we were in US so that’s the main country represented here. Next comes 6 people from Switzerland, the country where I live and work:

CaptureOOW16demoCountries84

When we register to OOW we fill-in the industry we are working on. The most represented in the room were Financial, Heathcare, High Tech:

CaptureOOW16demoIndustries84

And the job title which is a free text have several values, which makes it difficult to aggregate:

CaptureOOW16demoJobtitles84

That’s no surprise that there were a lot of DBAs. I’m happy to see some managers/supervisors interested in technical sessions.
My goal for future events is to get more attention from developers because a database is not a black box storage for data, but a full software where data is processed.

I don’t think that 84 people were in that room actually, there were several good sessions at the same time, as well as the bridge run.

oo1Csp1eTvXgAABNcH

This is the kind of slides where there’s lot of text but I go fast. Actually I had initially 3 slides about this point (feature usage detection, multitenant and CON_IDs). I removed some, and kept one with too much text. When I remove slides, I usually post a blog about what I’ll not have time to detail.

Here are those posts:

http://blog.dbi-services.com/unplugged-pluggable-databases/
http://blog.dbi-services.com/how-to-check-multitenant-option-feature-usage/

Thanks

My session was part of part of the stream which was selected by the EMEA Oracle Usergroup Community. Thanks a lot to EOUC. They have good articles in their newly created magazine www.oraworld.org. Similar name but nothing to do with the team of worldwide OCMs and ACEs publishing for years as Oraworld Team.

 

Cet article Feedback on my session at Oracle Open World 2016 est apparu en premier sur Blog dbi services.

Documentum xPlore – ftintegrity tool not working

Wed, 2016-10-19 08:38

I’ve been patching around some xPlore servers for a while and recently I went into an issue regarding the ftintegrity tool. Maybe you figured it out as well, for xPlore 1.5 Patch 15 the ftintegrity tool available in $DSEARCH_HOME/setup/indexagent/tools was corrupted by the patch.
I think for some reason the libs were changed and the tool wasn’t able to load anymore. I asked EMC and they said it was a known bug which will be fixed in next release.

So I came to patch it again when the Patch 17 went out (our customer processes doesn’t allow us to patch every month, so I skipped the Patch 16). After patching, I directly started the ftintegrity tool in order to check that everything was fixed, and…. no.

In fact yes, but you have something to do before making it work. The error you have is like ‘Could not load because config is NULL’, or ‘dfc.properties not found’. I found these error kinda strange, so I wondered if the ftintegrity tool script was patched as well. And the answer is no, the script is still the same but the jar libraries have been changed, that means that the script is pointing to the wrong libraries and it can’t load properly.

Thus the solution is simple, I uninstalled the Index Agent and installed it again right after. The ftintegrity tool script was re-created with the good pointers to the new libraries. Little tip: If you have several Index Agents and don’t want to re-install them all, you may want to copy the content of the updated ftintegrity tool script and paste it into the other instances (do not forget to adapt it because it may point to different docbases).

To summary, if you have issues regarding the execution of the ftintegrity tool, check the libraries call in the script and try to re-install it in order to have the latest one.

 

Cet article Documentum xPlore – ftintegrity tool not working est apparu en premier sur Blog dbi services.

Pages