Skip navigation.

Vikram Das

Syndicate content
Blog dedicated to Oracle Applications (E-Business Suite) Technology; covers Apps Architecture, Administration and third party bolt-ons to Apps
Updated: 15 hours 46 sec ago

R12.2 adapcctl.sh: opmnctl: not found after fresh install / upgrade

Tue, 2014-03-11 19:51


This problem occurs during rapidwiz where it fails to create apache instance and the error is logged in deploytech.log:from deploytech.log:

ERROR: Failed to create the EBS domain.
Failed to deploy the E-Business Suite domainOracle would usually advise you to re-install. However I found a way to do it by tracking the rapidwiz log files to see what all scripts it calls to create it manually:    Create missing apache instance, not created by rapidwiz Reference :
. ./EBSapps.env runcd $FMW_HOME/opmnctl/bin./opmnctl createinstance -oracleInstance $FMW_HOME/webtier/instances/EBS_web_ebs122_OHS1 -adminhost admin.justanexample.com -adminPort 7002 -adminRegistration OFF
Creating Oracle Instance directories...DoneRecording OPMN ports reservations...DoneBootstrapping OPMN configuration files...DoneInstantiating opmnctl for direct usage...DoneSkipping instance registrationCommand succeeded.
./opmnctl createcomponent -componentType OHS -componentName EBS_web_ebs122 -oracleInstance $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1
Creating empty component directories...DoneProvisioning OHS files for EBS_web_ebs122  Copying OHS files from ORACLE_HOME to ORACLE_INSTANCE locations  Customizing httpd.conf  Adding component's process control to OPMN...DoneSkipping EBS_web_ebs122 component registration.Command succeeded.

cd $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/bin./opmnctl registerinstance –adminhost admin.justanexample.com -adminPort 7002
Command requires login to weblogic admin server (alpcmfapq11.comfin.ge.com):  Username: weblogic  Password:
Deploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl admin.justanexample.com:7002 -username weblogic -name NonJ2EEManagement -source $FMW_HOME/webtier/opmn/applications/NonJ2EEManagement.ear -nostage -deploy -upload -noexit Task 0 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.Task 0 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.Target state: deploy completed on Server AdminServer
DoneRegistering instanceCommand succeeded.
Copy these *.conf files to $FMW_Home/webtier/instances/EBS_web_ebs122_OHS1/config/OHS/EBS_web_ebs122 obtained from Vision instance :

admin.conf
apps.conf
custom.conf
httpd.conf
mod_wl_ohs.conf
oracle_apache.conf
oracle_apache_ssl.conf
security2.conf
security2_dmz.conf
ssl.conf
ssl_terminator.conf
trusted.conf
url_fw.conf
url_fw_ws.conf
webgate.conf
Global replace the server name and port in the files above.
Run adconfig to regenerate these files
Instantiate the Oracle HTTP Server configuration files admin.conf, ssl.conf, mod_wl_ohs.conf on Run File System.
Note: Since the templates are being instantiated, all customizations will be lost and have to be restored explicitly.
Perform the instantiation with the following commands (the backslash at the end of each line is being used to divide each command across several screen lines for readability). When prompted for the APPS password, press .

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/ssl_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/mod_wl_ohs_conf_FMW.tmp

$ADJVAPRG oracle.apps.ad.autoconfig.InstantiateFile \
-e $CONTEXT_FILE \
-d $FND_TOP/admin/driver/ohstmpl.drv \
-tmpl $FND_TOP/admin/template/admin_conf_FMW.tmp

perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=$CONTEXT_FILE \
-outfile=$INST_TOP/logs/appl/rgf/TXK/test.log

apps.conf generated successfully

mod_wl_ohs.conf generated successfully
  
Categories: APPS Blogs

R12.2 Dictionary Corrupted error during adop phase=prepare

Mon, 2014-03-10 23:07
Do not follow the document 1576086.1.  I followed it and increased my invalid counts from 1000 to 100,000 and wasted 10 hours to get back to where I started.  Instead follow these instructions:


Reference : Ora-04023 Reported while Accessing Valid Objects (Doc ID 1610514.1)Dictionary Corrupted: Details are as follows: OWBSYS          HAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          WB_RTI_OBJECT_CATALOG                          OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          UAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          UAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          UAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          UAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
PUBLIC          WB_OLAP_AW_PRECOMPUTE                          OWBSYS          WB_OLAP_AW_PRECOMPUTE                          TS mismatch: 23-SEP-10 05:17:34 26-NOV-12 18:48:25
OWBSYS          ALL_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          RAB_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          HAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          HAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
OWBSYS          HAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19

please contact Oracle Support and request them to open a bug against Oracle Application Install(166), Component Online Patching(OP)         [ERROR]     Prepare phase completed with errors/warnings. Please check logfiles        Log file: /erpodpa1/erpapp/fs_ne/EBSapps/log/adop/3/adop_20140306_005850.log
select owner,object_type,object_namefrom dba_objectswhere object_name in('HAB_RT_EXEC_PROC_RUN_METRICS','WB_RTI_OBJECT_CATALOG','UAB_RT_EXEC_PROCESS','UAB_RT_EXEC_PROCESS_RUNS','UAB_RT_EXEC_PROC_RUN_COUNTS','UAB_RT_EXEC_PROC_RUN_METRICS','WB_OLAP_AW_PRECOMPUTE','ALL_RT_INSTALLATIONS','RAB_RT_INSTALLATIONS','HAB_RT_EXEC_PROCESS','HAB_RT_EXEC_PROCESS_RUNS','HAB_RT_EXEC_PROC_RUN_COUNTS');

Select against a view  fails with following error.
SQL> Select * from . ;
ORA-04023: could not be validated or authorized
This is issue is due to a timestamp inconsistency in the data dictionary. This timestamp inconsistency will lead to the problem when we  populate the relevant cached objects and later try to select from the affected views.
Run following script to find out the objects which are having timestamp discrepencies.
Connect as / as sysdba
SQL> @?/rdbms/admin/utldtchk.sql
This script is available from 11g onwards.
Recompile all the objects manually returned by this script.Conn owbsys/owbsysalter view ALL_RT_INSTALLATIONS compile;alter view RAB_RT_INSTALLATIONS compile;alter view HAB_RT_EXEC_PROCESS compile;alter view HAB_RT_EXEC_PROCESS_RUNS compile;alter view HAB_RT_EXEC_PROC_RUN_COUNTS compile;alter view HAB_RT_EXEC_PROC_RUN_METRICS compile;alter view UAB_RT_EXEC_PROCESS compile;alter view UAB_RT_EXEC_PROCESS_RUNS compile;alter view UAB_RT_EXEC_PROC_RUN_COUNTS compile;alter view UAB_RT_EXEC_PROC_RUN_METRICS compile;alter function WB_OLAP_AW_PRECOMPUTE compile;drop public synonym WB_OLAP_AW_PRECOMPUTE;CREATE PUBLIC SYNONYM WB_OLAP_AW_PRECOMPUTEFOR OWBSYS.WB_OLAP_AW_PRECOMPUTE;
It will remove the timestamp inconsistency in data dictionary and resolve this issue.
Verify by re-running the script, it should return zero rows.
SQL> @?/rdbms/admin/utldtchk.sql
If you still see occurances, bounce the database and rerun the above script.
Categories: APPS Blogs

ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table. ORA-06512: at "APPS.AD_ZD_ADOP", line 523

Wed, 2014-02-26 21:37
I reported this error to Oracle on an SR:

The adop phase=prepare fails before generating any log files. Whatever we get on the screen is what we have. I'll paste it for you again:

$ adop phase=prepare

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Please wait. Validating credentials...


RUN file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml
PATCH file system context file: $INST_TOP/appl/admin/tsoadba1_alpgetapp049d.xml


************* Start of session *************
version: 12.2.0
started at: Sun Feb 23 2014 20:55:38

APPL_TOP is set to $APPL_TOP
[STATEMENT] Using 48 workers (Default: 48, Recommended maximum limit: 696)
[EVENT] [START 2014/02/23 20:55:48] Verify SSH
Logfile location $INST_TOP/logs/appl/rgf/TXK/verifyssh.log
xml output = $INST_TOP/logs/appl/rgf/TXK/out.xml
[EVENT] [END 2014/02/23 20:55:54] Verify SSH
[ERROR] Failed to execute sql statement :
begin
AD_ZD_ADOP.LOCK_SESSIONS_TABLE('server1',60,0);
end;

[ERROR] SQLPLUS error: buffer=
SQL*Plus: Release 10.1.0.5.0 - Production on Sun Feb 23 20:55:59 2014
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> SQL> Connected.
SQL> begin
*
ERROR at line 1:
ORA-20010: ERROR: Unable to acquire lock on ad_adop_sessions table.
ORA-06512: at "APPS.AD_ZD_ADOP", line 523
ORA-06512: at line 2
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

[UNEXPECTED]Unable to acquire lock on sessions table

adop exiting with status = 2 (Fail)

Oracle asked me to confirm whether I had run adconfig on both run and patch environments. I had forgotten to run adconfig on patch environment.  After running adconfig on the patch environment, I retried and it worked.


Categories: APPS Blogs

Product Restrictions during R12.2 online patching

Thu, 2014-02-20 23:34
This information is from page 75 of Oracle® E-Business Suite Maintenance Guide Release 12.2 Part No. E22954-13, published in July 2013 available here  http://docs.oracle.com/cd/V39571_01/current/acrobat/122ebsmt.zip

For patches that have manual steps, the patch readme file instructs you to use Oracle Patch Application Assistant (PAA) to create customized instructions for your system. PAA consolidates and displays only the relevant manual steps for all the patches you want to apply, including steps that you have completed. It also automatically merges the contents of individual patch readme files for a merged patch.

Create Checklist of Product Functionality Disabled in Online Patching Cycle:
During an online patching cycle, the following product restrictions will apply. Before you commence patching, you should therefore ensure there will be no requirement for any these actions or features until the cycle is complete.
  • Payroll
    • Users will not be able to define Fast Formulas or use the Fast Formula Assistant.
    • Users will not be able to perform dynamic trigger maintenance.
    • Users will not be able to create, update, or delete US Cities.
    • Data Pump meta-mapper generator will be disabled.
    • The Japanese Balance Dimensions concurrent program will be deferred to after the cutover phase is complete.
    • Pension Calculation Setup cannot be used.
    • US localization earnings and deduction setup cannot be used.
    • Tax Withholding Rules Setup cannot be used.
    • Wage Attachment Earnings Rules Setup cannot be used.
    • Garnishment Rules Setup cannot be used.
    • Quick Paint Reports cannot be used.
    • Quantum Program Update Installer execution is unavailable.
  • Order Management:
    • Creation of a new Defaulting Condition in the Attribute Defaulting Rules form is disabled, unless the same seeded condition already exists for a given attribute.
  • Warehouse Management:
    • WMS Rule creation is restricted.
  • Inventory:
    • Concurrent program "Generate Stock Locator Flexfield Definition for Mobile Transactions" will be disabled.
  • Public Sector Financials International:
    • Users will not be able to run the following concurrent programs:
      • Subledger Security: Apply Security
      • Subledger Security: Import/Export Data Fix
  • Subledger Accounting:
    • Users will not be able to Validate the Application Accounting definitions.
  • Accounts Receivable:
    • Users will not be able to create new Transaction Sources.
  • Incentive Compensation:
    • Transaction collection process for new mappings will not be available and any changed mapping will continue to use previous mapping rules.
    • Users will not be able to run the "Synchronize Classification Rulesets" program.
    • Users will not be able to use the "Formula Generation" feature.
    • Users will not be able to specify new formulas or changes to compensation rules.
  • Oracle Demand Planning:
    • Demand plans will not be available for users.
Categories: APPS Blogs

R12.2 upgrade - icxr12pd.sql takes a very long time

Sat, 2014-02-08 17:19
During an R12.2 upgrade, the worker icxr12pd.sql takes very long time.  This is document in My Oracle Support Article ICXR12PD.Sql Runs Slowly During Upgrade to 12.1.1 (Doc ID 864898.1)
 
Cause: The pre-upgrade steps for iProcurement were not implemented as documented in the upgrade guide
Though this article talks about R12.1.1, it also applies to R12.2.  If you read the 11i to R12.2 upgrade guide ( http://docs.oracle.com/cd/V39571_01/current/acrobat/122upg11i.pdf ). it has specifically mentioned on page 285:

iProcurement
Completing these tasks could substantially reduce the downtime required for your
upgrade.
The Catalog Data Pre-upgrade Process
This pre-upgrade process is strongly recommended if you are upgrading iProcurement
from 11.5.9 or 11.5.10 to Release 12. It pre-processes bulk-loaded content to reduce the
actual time required for the upgrade and to ensure the upgrade process runs smoothly.
You can run it multiple times. If exceptions are found, then make corrections and re-run
the program until no exceptions are noted. Running this program does not require users
to log off the system. Note: Exceptions noted and not fixed before the upgrade will not
be available in the iProcurement Catalog. Specifically, this program shortens the time it
takes to run these upgrade scripts:
• -icxiftug.sql: updates POR_TEMPLATE_INFO
• -icxr12in.sql: general setup upgrade script needed for iProcurement
• -poxujpoh.sql:updates PO_HEADERS_ALL (for example, create_language)
• -poxukpol.sql: updates PO_LINES_ALL (for example, ip_category_id)
• -poxukrt.sql: updates PO_REQEXPRESS_LINES_ALL (for example, ip_catetory_id)
• -icxr12ug.sql: migrates iProcurement data model to PO
• -icxr12rt.sql: populates the requisition templates in iProcurement intermedia index
tables
• -icxr12pd.sql: populates the Purchasing documents in iProcurement intermedia
index tables: blanket purchase agreements (BPAs), global blanket agreements
(GBPAs), and quotes
• -icxr12mi.sql: populates the master items in iProcurement intermedia index tables
• -poxukfi.sql: purchasing script to approve the GBPAs created during iProcurement
migration
• -poxukat.sql: purchasing script to update the attachments
• -icxr12fi.sql: final upgrade script, which upgrades favorite lists, purges the BPAs
and GBPAs that are not approved, and creates the intermedia index


Make sure that you complete the tasks described in the upgrade guide, during the pre-upgrade phase, much before actual cutover downtime starts.

If you haven't done this and proceed to upgrade, it is going to take a long time for all the workers which are going to run icxr12pd.sql.  Based on my analysis of v$sql_longops, this is the insert query most fired during this worker's execution:

INSERT INTO AP_INV_DISTS_TARGET T1 (BATCH_ID, INVOICE_ID, INVOICE_LINE_NUMBER,
INVOICE_DISTRIBUTION_ID, DISTRIBUTION_LINE_NUMBER, LINE_TYPE_LOOKUP_CODE,
DESCRIPTION, DIST_MATCH_TYPE, ORG_ID, DIST_CODE_COMBINATION_ID, ACCOUNTING_DATE,
PERIOD_NAME, ACCRUAL_POSTED_FLAG, CASH_POSTED_FLAG, AMOUNT_TO_POST,
BASE_AMOUNT_TO_POST, POSTED_FLAG, ACCOUNTING_EVENT_ID, SET_OF_BOOKS_ID, AMOUNT,
BASE_AMOUNT, EXCHANGE_DATE, ROUNDING_AMT, QUANTITY_VARIANCE, BASE_QUANTITY_VARIANCE,
MATCH_STATUS_FLAG, ENCUMBERED_FLAG, PACKET_ID, USSGL_TRANSACTION_CODE,
USSGL_TRX_CODE_CONTEXT, REVERSAL_FLAG, PARENT_REVERSAL_ID, CANCELLED_FLAG, INCOME_TAX_REGION,
TYPE_1099, STAT_AMOUNT, CHARGE_APPLICABLE_TO_DIST_ID, PREPAY_AMOUNT_REMAINING,
PREPAY_DISTRIBUTION_ID, PREPAY_TAX_PARENT_ID, PARENT_INVOICE_ID, CORRECTED_QUANTITY,
PO_DISTRIBUTION_ID, RCV_TRANSACTION_ID, UNIT_PRICE, MATCHED_UOM_LOOKUP_CODE, QUANTITY_INVOICED,
FINAL_MATCH_FLAG, RELATED_ID, ASSETS_ADDITION_FLAG, ASSETS_TRACKING_FLAG, PROJECT_ID, TASK_ID, EXPENDITURE_T

In our test environment, we have 25 million + rows in ap_inv_dists_target.  We have already waited for 8 hours for all the workers to complete.  Here's the status of the queries:

set pages 300 lines 300
col "Username" for a12
col "QC/Slave" for A8
col "Slaveset" for A8
col "Slave INST" for A9
col "QC SID" for A6
col "QC INST" for A6
col "operation_name" for A30
col "target" for A30

select
decode(px.qcinst_id,NULL,username,
' - '||lower(substr(pp.SERVER_NAME,
length(pp.SERVER_NAME)-4,4) ) )"Username",
decode(px.qcinst_id,NULL, 'QC', '(Slave)') "QC/Slave" ,
to_char( px.server_set) "SlaveSet",
to_char(px.inst_id) "Slave INST",
substr(opname,1,30) operation_name,
substr(target,1,30) target,
sofar,
totalwork,
units,
start_time,
timestamp,
decode(px.qcinst_id, NULL ,to_char(s.sid) ,px.qcsid) "QC SID",
to_char(px.qcinst_id) "QC INST"
from gv$px_session px,
gv$px_process pp,
gv$session_longops s
where px.sid=s.sid
and px.serial#=s.serial#
and px.inst_id = s.inst_id
and px.sid = pp.sid (+)
and px.serial#=pp.serial#(+)
order by
decode(px.QCINST_ID, NULL, px.INST_ID, px.QCINST_ID),
px.QCSID,
decode(px.SERVER_GROUP, NULL, 0, px.SERVER_GROUP),
px.SERVER_SET,
px.INST_ID
/

Username QC/Slave SlaveSet Slave INS OPERATION_NAME TARGET SOFAR TOTALWORK UNITS START_TIME TIMESTAMP QC SID QC INS
------------ -------- -------- --------- ------------------------------ ------------------------------ ---------- ---------- -------------------------------- --------------- --------------- ------ ------
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Gather Table's Index Statistic 0 0 Indexes 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335
APPS QC 1 Table Scan AP.AP_INVOICE_DISTRIBUTIONS_AL 1570329 1570329 Blocks 08-FEB-14 2335

7 rows selected.
Categories: APPS Blogs

weblogic.store.PersistentStoreFatalException: The persistent file store "WLS_DIAGNOSTICS" canno t open file WLS_DIAGNOSTICS000000.DAT

Tue, 2014-01-07 20:47
After bouncing an EBS R12.2 VM, weblogic adminserver wouldn't start up.

I examined $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log and found this error:

BEA-280060 - The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
        at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
        at weblogic.store.io.file.direct.DirectFileChannel.(DirectFileChannel.java:54)
        at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
        at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
        at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
        at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
        at weblogic.store.io.file.Heap.open(Heap.java:325)
        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
        at weblogic.diagnostics.lifecycle.ArchiveLifecycleImpl.initialize(ArchiveLifecycleImpl.java:94)
        at weblogic.diagnostics.lifecycle.DiagnosticFoundationService.start(DiagnosticFoundationService.java:108)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

A search on support.oracle.com for keywords
resulted in this hit:

E-WL: WebLogic Fails to Start with Error: “The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException" (Doc ID 859622.1)

WORKAROUNDS:
1. This is a recurrent workaround. This workaround has to be applied every time the server(s) will be restarted.

Delete the *.dat file under
$PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\default\ and $PS_HOME\webserv\domain\servers\PIA (or WebLogicAdmin)\data\store\diagnostics\
  
Specifically for EBS R12.2 I did this:

cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/default
rm *.DAT
 cd $EBS_DOMAIN_HOME/servers/AdminServer/data/store/diagnostics
rm *.DAT

adadminsrvctl.sh start

After keying in apps, system and weblogic passwords, it finally started up.

Have to see if this is a recurrent workaround or fixes the issue for good.

Categories: APPS Blogs