Adop phase=fs_clone failing with error [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on

Issue Details: - 

adop phase=fs_clone failed.

 applmgr@oraebsdemo ~

EBSPOC: adop phase=fs_clone

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Warning: WebLogic Admin Server is not running.

Trying to start Admin Server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Starting WLS Admin Server...

Refer /u01/apps/fs1/inst/apps/EBSPOC_oraebsdemo/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/apps/fs1/FMW_Home/user_projects/domains/EBS_domain_EBSPOC/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/apps/fs1/inst/apps/EBSPOC_oraebsdemo/logs/appl/admin/log/adadminsrvctl.txt for more information ...

Initializing.

    Run Edition context  : /u01/apps/fs1/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml

    Patch edition context: /u01/apps/fs2/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml

    Patch file system free space: 858.38 GB

Validating system setup.

    Node registry is valid.

    [WARNING]   MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

Checking for existing adop sessions.

    No pending session exists.

    Starting new adop session.

===========================================================================

ADOP (C.Delta.12)

Session ID: 3

Node: oraebsdemo

Phase: fs_clone

Log: /u01/apps/fs_ne/EBSapps/log/adop/3/20211009_180424/adop.log

===========================================================================

Validating configuration on node: [oraebsdemo].

    Log: /u01/apps/fs_ne/EBSapps/log/adop/3/20211009_180424/fs_clone/validate/oraebsdemo

    [UNEXPECTED]Error occurred running "perl /u01/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/apps/fs1/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml -patchctxfile=/u01/apps/fs2/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml -phase=fs_clone -logloc=/u01/apps/fs_ne/EBSapps/log/adop/3/20211009_180424/fs_clone/validate/oraebsdemo -promptmsg=hide"

    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on oraebsdemo

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 1 (Fail)

applmgr@oraebsdemo ~

EBSPOC:

Cause:- 

Run below to verify the patch context file entry, if no rows selected then follow the solution section.

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

no rows selected

SQL> 

Solution:-

To fix this issue test the following to upload the missing PATCH context file to the database:

1. Get the value of your patch context file.

    1.1. Source your PATCH file system

    1.2. Echo the value of $CONTEXT_FILE and take note of it. You will use this value in step #3.

2. Source the RUN filesystem

3. Execute the following command on the run filesystem to upload the patch context file to the database.

$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \

action=upload \

contextfile=/u01/apps/fs2/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml \

logfile=/tmp/patchctxupload.log

Note: the context file parameter should point to the physical location of the patch context file in your system

4. Run the following query and it should return one entry for the recently uploaded context file.

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

PATH

--------------------------------------------------------------------------

/u01/apps/fs2/inst/apps/EBSPOC_oraebsdemo/appl/admin/EBSPOC_oraebsdemo.xml

-- Rerun the adop phase=fs_clone, it completed successfully.

OPatch failed with error code 73, The Central Inventory is corrupted:

Details of error as follows - 

oracle@orademodb01 ~

ORCL: opatch lsinv

Oracle Interim Patch Installer version 12.2.0.1.21

Copyright (c) 2021, Oracle Corporation.  All rights reserved.

Oracle Home       : /ora/db/tech_st/12.1.0.2

Central Inventory : /home/oracle/oraInventory

   from           : /ora/db/tech_st/12.1.0.2/oraInst.loc

OPatch version    : 12.2.0.1.21

OUI version       : 12.1.0.2.0

Log file location : /ora/db/tech_st/12.1.0.2/cfgtoollogs/opatch/opatch2021-09-28_16-16-01PM_1.log

List of Homes on this system:

  Home name= ORCL_Home, Location= "/ora/db/tech_st/12.1.0.2"

LsInventorySession failed: RawInventory gets null OracleHomeInfo

OPatch failed with error code 73

oracle@orademodb01 ~

Solution:

Turned out my inventory had not been updated with my new Oracle Home. When I looked in the inventory.xml file on my server, there was no entry for the installation there.

To fix this, add the new OH to your inventory. From your OH that is missing, do the following:

oracle@orademodb01 /ora/db/tech_st/12.1.0.2

ORCL: cat oraInst.loc

inventory_loc=/home/oracle/oraInventory

inst_group=oinstall

oracle@orademodb01 /ora/db/tech_st/12.1.0.2

ORCL: cat /etc/oraInst.loc

inventory_loc=/home/oracle/oraInventory

inst_group=oinstall

oracle@orademodb01 /ora/db/tech_st/12.1.0.2

ORCL: cd $ORACLE_HOME/oui/bin

oracle@orademodb01 /ora/db/tech_st/12.1.0.2/oui/bin

ORCL: ./runInstaller -invPtrLoc /ora/db/tech_st/12.1.0.2/oraInst.loc -attachHome ORACLE_HOME=/ora/db/tech_st/12.1.0.2 ORACLE_HOME_NAME="ORCL"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 15517 MB    Passed

The inventory pointer is located at /ora/db/tech_st/12.1.0.2/oraInst.loc

'AttachHome' was successful.

oracle@orademodb01 /ora/db/tech_st/12.1.0.2/oui/bin

EBS 12.2.10 Upgrade (Patch 30399999) error

 While applying EBS 12.2.10 patch 30399999 got the below error.

ATTENTION: All workers either have failed or are waiting:

FAILED: file fnd_web_resource.xdf on worker 1.

FAILED: file fnd_web_product_family.xdf on worker 4.

FAILED: file fnd_web_used_resource.xdf on worker 5.

FAILED: file fnd_web_product_family_tl.xdf on worker 7.

FAILED: file fnd_web_application.xdf on worker 8.

Fix:

Login to sqlplus as apps user and execute below :

exec AD_ZD_SEED.UPGRADE('FND_WEB_PRODUCT_FAMILY');

exec AD_ZD_SEED.UPGRADE('FND_WEB_APPLICATION');

exec AD_ZD_SEED.UPGRADE('FND_WEB_RESOURCE');

exec AD_ZD_SEED.UPGRADE('FND_WEB_PRODUCT_FAMILY_TL');

exec AD_ZD_SEED.UPGRADE('FND_WEB_USED_RESOURCE');

Once you applied the fix, restart all failed workers and the patch got completed successfully. 

How to check online patching is enabled from backend ?

 If you wanted to check online patching is enabled or not from backend, run the below query as apps user.

sqlplus apps/apps

SQL> select decode(editions_enabled, 'Y','Yes','No') "Online Patching Enabled?"

from fnd_oracle_userid au, dba_users du

where au.oracle_username = du.username and au.read_only_flag = 'U';  

Onl

---

Yes

DB Startup issue after memory change in the server level

Faced an issue after changing the DB memory parameters (SGA and PGA) and got the below error while startup.

ORA-27106: System Pages Not Available To Allocate Memory

Issue: Found that the "USE_LARGE_PAGES" setting to "ONLY"

Fix: Parameter USE_LARGE_PAGES set to "TRUE" and started database, DB came up. 

How to check who logged in Oracle Apps(EBS) Front End ?

 It is very common for DBA to check who logged into EBS application in order to take application down for some maintenance ... etc and make sure the system is idle.

Here is the query to see who is there on the system from backend, connect to database as APPS user and run the query.

I ran this query to see who logged in last one day, based on your requirement you can change it for hours as well.

 col user_name for a40

 select USER_NAME, to_char(LAST_LOGON_DATE, 'DD-MON-YYYY HH24:MI:SS') from fnd_user where last_logon_date > sysdate -1 order by USER_NAME;

USER_NAME        TO_CHAR(LAST_LOGON_DATE,'D

-----------      --------------------------

ABC.DEF          21-JUN-2021 09:48:28

GHI.JKL          21-JUN-2021 08:51:31

MNO.PQR          21-JUN-2021 10:18:14

Workflow Override Email Address

It is very common requirement to stop workflow notifications going out of the system, particularly from Development instances and Test instances. Oracle have given a functionality “workflow override email address” to achieve this requirement.

In many companies this task is done by Database Administrators (DBA) but incase you need to do it yourself you follow the below steps.  There are two ways to setup this: 

1. Login to Oracle Apps preferably with a user who is "Workflow System Administrator".

Navigate to “Workflow Administrator Web Applications” Responsibility
Expand "Oracle Applications Manager"

Click "Workflow Manager"

Click "Service Components"

Click “Workflow Notification Mailer”

Click "Set Override Address"

Once you provide the email address you will receive a verification code for the first time which you need to enter it back in the above page to complete the process. Verification field will be enabled after you provide the email address.
2.  From backend, connect to the database using apps user and run below update statement.

update fnd_svc_comp_param_valsset    parameter_value = '<email address>'where  parameter_value = 'NONE'and    parameter_id =( select parameter_idfrom   fnd_svc_comp_params_tlwhere  display_name = 'Test Address');
commit;

OTA – not up and Running in 12.1.3

 

We were in the process of creating Purchase Orders from iProc Requisitions and to send the Auto-approved Purchase Orders through OSN. We created OSN accounts and then realized that the PO’s didn’t hit the OSN. We did check the Work-flow agents and listeners were Enabled and running fine. As part of the trouble-shooting the issue, we found that the OTA (OXTA) was not up and running.

SQL –

select machine,action, decode(count(*),0,’Error: OTA is Not Running’,’OTA is Running’)
from gv$session
where action like ‘%OXTA%’
group by machine, action;

The above query returned no rows.

Reason –

1. ASADMIN password didn’t either match in the system-jazn-data.xml ($INST_TOP/ora/10.1.3/j2ee/oafm/config/system-jazn-data.xml) as of ASADMIN set through the User Management
2. If there was an Upgrade the passwords got changed

Fix –

1. Check and confirm that dbc file specified in $INST_TOP/ora/10.1.3/j2ee/oafm/application-deployments/oafm/data-sources.xml file exists and is correct.
2.  Confirm that the ASADMIN user setup is correct as below

a. Set profile Applications SSO Login Types  (APPS_SSO_LOCAL_LOGIN) for ASADMIN user to ‘Local’.

b. Change the ASADMIN password from User Management as follows (note: in this example password is set to welcome)

– Log onto Oracle E-Business Suite using sysadmin/<sysadmin_pw>.
– Select the User Management responsibility in the Navigator.
– Click the Users link from the navigation menu to open the User Maintenance window.
– Locate ‘ASADMIN’ user by entering information in the search area to retrieve the ‘ASADMIN’ user.
– Click the Update icon next to the ASADMIN user to open the Update User window.
– Remove the Active to date field and click Apply in case is not already removed
– Click the Reset Password icon next to the ASADMIN user to open the Reset Password window.
– Enter new password twice and click Submit

c. Confirm the definition for ASADMIN user in $INST_TOP/ora/10.1.3/j2ee/oafm/application-deployments/oafm/data-sources.xml is defined correctly:

user=”ASADMIN”
password=”->ASADMIN”

Note: The password above must be set with exact value as above because ->ASADMIN is an indirection, it indicates AS 10.1.0.3 that the password of ASADMIN must be taken from system-jazn-data.xml

d. Change password for ASADMIN user in $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/config/system-jazn-data.xml to be !(! in front is mandatory)

Mention: ! in front of password is mandatory and this sign means that password will be encrypted

e. Shutdown and restart the OAFM container and re-test:

sh $ADMIN_SCRIPTS_HOME/adoafmctl.sh stop

sh $ADMIN_SCRIPTS_HOME/adoafmctl.sh start

References –

1. NOTE: 1O87499.1 – Cannot Start OXTA After Upgrade To R12.1.x
2. NOTE:418926.1 – How to Configure OTA For XML Gateway in Release 12
3. NOTE:419839.1 – How to enable Apache, OC4J and OPMN logging in Oracle Applications R12

Checking Status of all Concurrent Managers from backend.

 There are two ways you can check the concurrent managers running in Oracle apps server.

1. From Oracle Apps Front End :-

Below is the navigation to check currently running concurrent managers in the server:

System Administrator >> Concurrent : Manager >> Administer.

2. Query to get the info of all the concurrent managers running in the Oracle apps DB:-

SELECT DECODE (
          CONCURRENT_QUEUE_NAME,
          'FNDICM', 'Internal Manager',
          'FNDCRM', 'Conflict Resolution Manager',
          'AMSDMIN', 'Marketing Data Mining Manager',
          'C_AQCT_SVC', 'C AQCART Service',
          'FFTM', 'FastFormula Transaction Manager',
          'FNDCPOPP', 'Output Post Processor',
          'FNDSCH', 'Scheduler/Prereleaser Manager',
          'FNDSM_AQHERP', 'Service Manager: AQHERP',
          'FTE_TXN_MANAGER', 'Transportation Manager',
          'IEU_SH_CS', 'Session History Cleanup',
          'IEU_WL_CS', 'UWQ Worklist Items Release for Crashed session',
          'INVMGR', 'Inventory Manager',
          'INVTMRPM', 'INV Remote Procedure Manager',
          'OAMCOLMGR', 'OAM Metrics Collection Manager',
          'PASMGR', 'PA Streamline Manager',
          'PODAMGR', 'PO Document Approval Manager',
          'RCVOLTM', 'Receiving Transaction Manager',
          'STANDARD', 'Standard Manager',
          'WFALSNRSVC', 'Workflow Agent Listener Service',
          'WFMLRSVC', 'Workflow Mailer Service',
          'WFWSSVC', 'Workflow Document Web Services Service',
          'WMSTAMGR', 'WMS Task Archiving Manager',
          'XDP_APPL_SVC', 'SFM Application Monitoring Service',
          'XDP_CTRL_SVC', 'SFM Controller Service',
          'XDP_Q_EVENT_SVC', 'SFM Event Manager Queue Service',
          'XDP_Q_FA_SVC', 'SFM Fulfillment Actions Queue Service',
          'XDP_Q_FE_READY_SVC', 'SFM Fulfillment Element Ready Queue Service',
          'XDP_Q_IN_MSG_SVC', 'SFM Inbound Messages Queue Service',
          'XDP_Q_ORDER_SVC', 'SFM Order Queue Service',
          'XDP_Q_TIMER_SVC', 'SFM Timer Queue Service',
          'XDP_Q_WI_SVC', 'SFM Work Item Queue Service',
          'XDP_SMIT_SVC', 'SFM SM Interface Test Service')
          AS "Concurrent Manager's Name",
       max_processes AS "TARGET Processes",
       running_processes AS "ACTUAL Processes"
  FROM apps.fnd_concurrent_queues
 WHERE CONCURRENT_QUEUE_NAME IN
          ('FNDICM',
           'FNDCRM',
           'AMSDMIN',
           'C_AQCT_SVC',
           'FFTM',
           'FNDCPOPP',
           'FNDSCH',
           'FNDSM_AQHERP',
           'FTE_TXN_MANAGER',
           'IEU_SH_CS',
           'IEU_WL_CS',
           'INVMGR',
           'INVTMRPM',
           'OAMCOLMGR',
           'PASMGR',
           'PODAMGR',
           'RCVOLTM',
           'STANDARD',
           'WFALSNRSVC',
           'WFMLRSVC',
           'WFWSSVC',
           'WMSTAMGR',
           'XDP_APPL_SVC',
           'XDP_CTRL_SVC',
           'XDP_Q_EVENT_SVC',
           'XDP_Q_FA_SVC',
           'XDP_Q_FE_READY_SVC',
           'XDP_Q_IN_MSG_SVC',
           'XDP_Q_ORDER_SVC',
           'XDP_Q_TIMER_SVC',
           'XDP_Q_WI_SVC',
           'XDP_SMIT_SVC');