Wednesday, August 30, 2017

AccessGate Login Fails with HTTP 500 Error after ADOP Cycle

I was facing the subject problem after successful ADOP cycle, Single Sign On SSO does not work and throws HTTP 500 Error at login.
Below shows that ADOP cycle was successful,
But when we try to access the application we get below error,
EBS 12.2 AccessGate Login Fails with HTTP 500 Error 'Request Failed for : /accessgate/ssologin?, Resp Code : [500]' On Secondary Nodes After A Complete ADOP Patching Cycle As File is Not Updated On Multiple Filesystems (fs2)

In some cases fs_clone can fail to copy the file to the other filesystem fs2 for the additional apps nodes.
For example, on the secondary apps tiers using a different filesystem (fs2), file will have null settings for some parameters:
# +======================================================================+
# |    Copyright (c) 2005, 2014 Oracle and/or its affiliates.           |
# |                         All rights reserved.                         |
# |                           Version 12.0.0                             |
# +======================================================================+
# $Header: oaea_wls_properties.tmp 120.0.12020000.1 2014/07/15 12:36:37 mmanku noship $
# This file lists all the  parameters required to start acessgate application
# /u03/eruat/fs1/EBSapps/appl/fnd/12.0.0
# ###############################################################
# Do not edit settings in this file manually. They are managed
# automatically and will be overwritten when AutoConfig runs.
# For more information about AutoConfig, refer to the Oracle
# E-Business Suite Setup Guide.
# ###############################################################

In certain cases for an ONLINE patching cycle, the file $INST_TOP/appl/admin/ does not get copied from RUN to PATCH.
The issue is documented in the following unpublished defect:
Workaround Fix:
To resolve the issue test the following steps in a development instance and then migrate accordingly:
1. Ensure a valid backup exists for the instance where the solution is being testing.
2. Copy file $INST_TOP/appl/admin/ from the original filesystem (fs1) over to the additional filesystems (fs2), to migrate the proper node settings to
3. Restart the servers to ensure the file changes are processed.
4. Retest the login and confirm the error is resolved.
Following should be the content of the file
This is the latest update as of 28-August-2017 for this issue, there is a patch for this defect 25720619 that is planned to be released in the next AD / TXK Delta patchset, with no release date information currently available.

For any queries on this article, kindly email me

Monday, July 31, 2017

Install Webgate, Deploy AccessGate, Register EBS with OAM

In this article we will see how to Install webgate, Deploy Access Gate and Register EBS with OAM.

1. Install Webgate and apply patch to Webgate Home Directory
Firstly copy the webgate patch p21118593 and webgate software to node 1 EBS server and then install using script

# scp -rp webgate appluat@ed-olraclin3:/home/appluat/ssoPatches
# -script=SetOAMReg -installWebgate=yes -webgatestagedir=/home/appluat/ssoPatches/webgate

Now apply WEBGATE Patch to webgate HOME Directory
[appluat@ed-olraclin3 FMW_Home]$ export ORACLE_HOME=/u03/eruat/fs1/FMW_Home/Oracle_OAMWebGate1
[appluat@ed-olraclin3 FMW_Home]$ export PATH=/u03/eruat/fs1/FMW_Home/Oracle_OAMWebGate1/OPatch/:$PATH
[appluat@ed-olraclin3 ssoPatches]$ unzip
[appluat@ed-olraclin3 21118593]$ pwd
[appluat@ed-olraclin3 21118593]$ opatch apply

2. Deploy Access Gate
[appluat@ed-olraclin3 21118593]$ wget (test the connection)--2017-06-18 10:44:45--
Resolving (
Connecting to (||:7014... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-18 10:44:45 ERROR 404: Not Found.

On EBS Node 1 (ed-olraclin3) server execute the below command,
# perl $AD_TOP/patch/115/bin/ ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate -SSOServerURL= -managedsrvname=oaea_server1 -managedsrvport=6801 -logfile=/tmp/deployeag.log

On EBS Node 2 (ed-olraclin4) server execute the below command,
# perl $AD_TOP/patch/115/bin/ ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate -SSOServerURL= -managedsrvname=oaea_server2 -managedsrvport=6802 -logfile=/tmp/deployeag.log
On EBS Node 1 (ed-olraclin3) server execute the below command,
#perl $FND_TOP/patch/115/bin/ -contextfile=$CONTEXT_FILE -configoption=addMS
On EBS Node 2 (ed-olraclin4) server execute the below command,
#perl $FND_TOP/patch/115/bin/ -contextfile=$CONTEXT_FILE -configoption=addMS

--> Now Start Accessgate managed servers and make sure that accessgate + datasource are up and running. Test the datasource too from weblogic console datasources link.
3. Register EBS with OAM
Execute below command on EBS Node 1 ed-olraclin3 server
# -script=SetOAMReg -registeroam=yes -oamHost= -oamUserName=weblogic -ldapUrl=ldap:// -oidUserName=cn=orcladmin -skipConfirm=yes -ldapSearchBase=cn=Users,dc=qa,dc=qia,dc=net -ldapGroupSearchBase=cn=Groups,dc=qa,dc=qia,dc=net

Execute below command on EBS Node 2 ed-olraclin4 server -script=SetOAMReg -registeroam=yes -oamHost= -oamUserName=weblogic -ldapUrl=ldap:// -oidUserName=cn=orcladmin -ldapSearchBase=cn=Users, dc=qa,dc=qia,dc=net - ldapGroupSearchBase=cn=Groups, dc=qa,dc=qia,dc=net -authScheme=EBSAuthScheme -authSchemeMode=reference

Note: If you are setting up SSO for EBS then firstly you should be ready with OID, OAM setup and then do this article steps, set required EBS profile settings and finally restart all EBS Services and test EBS Single Sign On SSO.
Moreover, kindly follow my previously posted articles to do setup for OID, OAM, Windows Native Authentication WNA Zero Sign-on, etc.

For any further queries please email me on

Tuesday, June 20, 2017

Cloning Oracle EBS 12.2.5 Delta 7+

Run the script on existing ERP system that will be cloned to new system.
After script on primary server, COPY ONLY /u01/erpsource/fs1/EBSapps from Primary to Clone server.

# perl appsTier

Now zip the EBSapps directory and copy to the target server where we will do the cloning,
#cd /u03/erpprd/fs1
#tar -cvf EBSapps.tar EBSapps
#cd /u03/erpprd/fs1
#gzip EBSapps.tar

Copy the EBSapps.gz file to the target server
#gunzip EBSapps.gz
#tar -xvf EBSapps.tar

On target server make sure all Pre-requisites are available that are mentioned in this Oracle Document,
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)
High level below are few pre-requisites,

systemctl disable bluetooth.service
systemctl disable libvirtd.service
systemctl disable NetworkManager.service
systemctl disable lvm2-monitor.service
systemctl disable lvm2-lvmetad.service
systemctl disable rdma.service
systemctl disable lvm2-lvmetad.socket
systemctl disable lvm2-lvmpolld.socket
systemctl stop firewalld.service
systemctl diabled firewalld.service
systemctl disabled firewalld.service
systemctl disable firewalld.service

Check pre-requisite RPMs for Database and also install the below rpm for application tier,

• compat-libstdc++-296-2.96-144.0.2.el7.i686
• openmotif21-2.1.30-11.EL7.i6861
• xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
Additionally, the following rpms need to be installed from the Oracle Linux 7 or RHEL 7 distribution media on the application tier:
• libXrender-0.9.8-2.1.el7.i686
• perl-File-CheckTree-4.42-3.el7.noarch
If not using the pre-install rpm, the following packages must be installed from the Oracle Linux 7 or RHEL 7 distribution media on both the application and database tiers:
• binutils-
• compat-libstdc++-33-3.2.3-71.el7.i686
• gcc-4.8.2-16.el7.x86_64
• gcc-c++-4.8.2-16.el7.x86_64
• gdbm-1.10-8.el7.i686
• gdbm-1.10-8.el7.x86_64
• glibc-2.17-55.el7.i686
• glibc-2.17-55.el7.x86_64
• glibc-common-2.17-55.el7.x86_64
• glibc-devel-2.17-55.el7.i686
• glibc-devel-2.17-55.el7.x86_64
• libgcc-4.8.2-16.el7.i686
• libgcc-4.8.2-16.el7.x86_64
• libstdc++-devel-4.8.2-16.el7.i686
• libstdc++-devel-4.8.2-16.el7.x86_64
• libstdc++-4.8.2-16.el7.i686
• libstdc++-4.8.2-16.el7.x86_64
• libXi-1.7.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.x86_64
• libXtst-1.2.2-2.1.el7.i686
• libaio-0.3.109-12.el7.i686
• libaio-0.3.109-12.el7.x86_64
• libgomp-4.8.2-16.el7.x86_64
• make-3.82-21.el7.x86_64
• redhat-lsb-4.1-24.0.1.el7.x86_64
• sysstat-10.1.5-4.el7.x86_64
• util-linux-2.23.2-16.el7.x86_64
Additionally, the following RPMs are required for the database tier:
• compat-libcap1-1.10-7.el7.x86_64
• compat-libstdc++-33-3.2.3-71.el7.x86_64
• elfutils-libelf-devel-0.158-3.el7.x86_64
• libaio-devel-0.3.109-12.el7.i686
• libaio-devel-0.3.109-12.el7.x86_64
• xorg-x11-utils-7.5-13.1.el7.x86_64

?Install the below listed prerequisites RPM from DVD for Oracle Database
yum install binutils -y
yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install zlib-devel -y
yum install elfutils-libelf-devel -
  • Perform the following command (as root on your system) to create a required link to the Berkeley DB library
[root@erplx0 lib]# scp -r -p vebsap2:/usr/lib
# cd /usr/lib                                                                      
# ln -s

  • DBb tier cloning
cd /u01/oracle/
./runInstaller -clone -silent -ignorePreReq ORACLE_HOME="/u01/oracle/" ORACLE_HOME_NAME="oracledb_home1" ORACLE_BASE="/u01/oracle" OSOPER_GROUP=oinstall

  • init.ora file check for these important parameters
service_names    string      erptest, ebs_patch

# perl appsuser=APPS appspass=apps
# ./ contextfile=/u01/oracle/
WARNING: [AutoConfig Error Report]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/              INSTE8_PRF         1
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/              INSTE8_APPLY       1

Update TNSNAMES.ORA file and correct the ERPTEST tnsentry and then run the below two .sh scripts manually,
. /u01/oracle/
. /u01/oracle/

[oraerp@db02 bin]$ perl appsuser=APPS appspass=apps
Starting context file generation for db tier..
Using JVM from /u01/oracle/ to execute java programs..
The log file for this adbldxml session is located at:
UnsatisfiedLinkError exception loading native library: njni11
java.lang.UnsatisfiedLinkError: no njni11 in java.library.path
Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: db02
Enter Port of Database server: 1522
Enter SID of Database server: erptest
Enter Database Service Name: erptest
Enter the value for Display Variable:
Invalid input, reenter the value for Display Variable: 0.0
The context file has been created at:

On Application Tier for below are the options,
Target System Base Directory : /u01/erptest
Target System Instance Home Directory [/u01/erptest] :
Do you want to preserve the Display [erplx0:0.0] (y/n)  : n
Target System Display [erplx0:0.0] :
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

[root@erplx0 etc]# cd /u01/erpt/
[root@erplx0 erptest]# mkdir oraInventory
chown -R appltst:oinstall oraInventory

as root user, create new file /etc/oraInst.loc with below content.
[root@erplx0 etc]# vi oraInst.loc
[root@erplx0 ~]# usermod -g oinstall -G oinstall appltst
[root@erplx0 ~]# id appltst
uid=1001(appltst) gid=54322(oinstall) groups=54322(oinstall)
  • To clean the oracle_home entries from oraInventory run the below
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1


              2 X Y N X Y X C              lxap1
              4 Y Y Y Y Y X C              lxap1
              5 Y Y Y Y Y X C              lxap1
              6 Y Y Y Y Y X C              lxap1
              7 Y Y Y Y Y X C              lxap1
              8 Y Y Y Y Y X C              lxap1
              9 Y Y Y Y Y X C              lxap1
             10 Y Y Y Y Y X C             lxap1
             10 Y Y Y Y Y X C             lxap2
             11 Y Y Y Y Y X C             lxap1
             11 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap1
             12 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             lxap1
             13 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             irec
             14 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             lxap2
             15 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             irec
             16 Y Y Y Y Y X C             lxap1
             16 Y Y Y Y Y X C             lxap2
             16 Y Y Y Y Y X C             irec
             17 Y Y Y Y Y X C             lxap1
             17 Y Y Y Y Y X C             lxap2
             17 Y Y Y Y Y X C             irec
             18 Y Y Y Y Y X C             lxtst0
             19 Y Y Y Y Y X C             lxtst0
             20 Y Y Y Y Y X C             lxtst0
             21 Y Y Y Y Y X C             lxtst0
33 rows selected.

As can be seen above last online patching session (11) was done successfully till cleanup stage (marked as Y)

SQL> SELECT abbreviation,codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');
ad                             C.8
txk                            C.8
Above means we are on delta8 patch level, hence we can use the 'dualfs' option while cloning from Delta 7 onwards.

#cd /u01/erptest/fs1/EBSapps/comn/clone/bin
#perl appsTier dualfs

1. Change APPS and Weblogic passwords
2. update all context files ----> https to http in both FS1 & FS2
3. fnd_conc_clone.setup_clean;
4. run autoconfig on patch FS (.EBSapps.env patch)--->if patch edition auto config error then disable ebs_logon trigger and later enable it.
5. run autoconfig on run FS (.EBSapps.env run)

  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system
[applerp@erplx0 ~]$ more /u01/erptest/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml | grep https
<EndUserMonitoringURL oa_var="s_endUserMonitoringURL"></EndUserMonitoringURL>
  <externURL oa_var="s_external_url"></externURL>
    <login_page oa_var="s_login_page"></login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">https</webentryurlprotocol>


<EndUserMonitoringURL oa_var="s_endUserMonitoringURL"></EndUserMonitoringURL>
  <externURL oa_var="s_external_url"></externURL>
    <login_page oa_var="s_login_page"></login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">http</webentryurlprotocol>

vi /u01/erptst/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
vi /u01/erptest/fs2/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
:%s/https/http/g -- then change http to http for only this line---><httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
. /u01/erptst/EBSapps.env patch
. /u01/erptst/EBSapps.env run
---->IF THIS FAILS THEN SQL> ALTER TRIGGER EBS_LOGON DISABLE; THEN RUN AUTOCONFIG AGAIN and then enable the trigger and then run autoconfig from run file system.
SQL> select owner,trigger_name,trigger_type,status from dba_triggers where trigger_name='EBS_LOGON';
OWNER                          TRIGGER_NAME                   TRIGGER_TYPE STATUS
SYSTEM                         EBS_LOGON                      AFTER EVENT ENABLED

  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system

1. Shut down all application tier services except the Admin Server.
On the primary node, run the command:
 $ <ADMIN_SCRIPTS_HOME>/ -skipNM -skipAdmin
 On all secondary nodes, run the command:
2. Change the Oracle WebLogic Server Administration User password by performing the following steps on the run file system of the primary node.
 Source the environment on the run file system.
 Run the commands appropriate for your platform:
 On UNIX, run the command:
 $ perl $FND_TOP/patch/115/bin/ -action=updateAdminPassword
The above command is available only if TXK DELTA 7 is applied.
If that is not the case, the password need to changed from Console and few steps need to be performed
3. Start all services on all nodes, using the command:
  • Do not do the below on cloned instance as it is done only on production. On clone instance it is not required to enable the workflow mailer after cloning for test environments. 
Setup this profile for mailer:
WF: Workflow Mailer Framework Web Agent:
Please change the value to appropriate server.
Also run the below script to setup the Workflow Admin role
SQL> update wf_resources
where name = 'WF_ADMIN_ROLE';

########################TROUBLESHOOTING 1##############
  • 12.2 patch edition auto config error  INSTE8_APPLY 1
To run autoconfig from patch file system you must disable trigger ebs_login prior to running autoconfig
1. alter trigger ebs_logon disable;
2. source patch file system and run autoconfig successfully.
3. alter trigger ebs_logon enable;
4. source patch file system and then connect as SYSTEM user using sqlplus

########################TROUBLESHOOTING 2##############
Enter the APPS password :
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object
chmod: cannot access /u01/erptest/fs1/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt: No such file or directory
In /etc/fstab file make xfs file system 32bit. (REFERENCE:
/dev/oracle_vg1/u01_v                           /u01                    xfs     rw,inode32      0 0
check for jre and jdk 32 bit folders, ./java -version

########################TROUBLESHOOTING 3##############
ERROR: while running ./runInstaller -deinstall option we are getting below error,
The Java RunTime Environment was not found at /tmp/OraInstall/jdk/jre/bin/java
oraparam.ini file under the software staged directory path /software/FR11121/Disk1/install/linux64 had an entry JRE_SCRATCH_LOC
 It was set to
UPDATE oraparam.ini to use another jre or fix the corrupted jre files in "/u01/erpt/fs1/EBSapps/10.1.2/jdk" folder.

########################TROUBLESHOOTING 4##############
After installing oracle required patches run the below command as root user,
ldconfig -v

check the below
[root@erplx0 lib]# ls -ltr libdb*
-rwxr-xr-x 1 root root 1691492 Sep 11  2014
-rwxr-xr-x 1 root root 1575444 Sep 11  2014
-rwxrwxrwx 1 root root    5825 Aug  3  2016
lrwxrwxrwx 1 root root      12 Apr  3 14:33 ->

if link is not there then create it as below,
# ln -s
check if this file exists or not /usr/lib/

########################TROUBLESHOOTING 5##############
Error that FMW_Home already exists the,
cd /u01/erptest/fs1/FMW_Home/oracle_common/oui/bin
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
>>Then remove all folders except fs1 and re-run script.
[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erpt/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.

########################TROUBLESHOOTING 6##############
R12.2.x ADOP : [ERROR] ETCC not run in the database node DBHOSTNAME

This table TXK_TCC-RESULTS is created in APPLSYS schema, by the latest version of the script delivered by 17537119.

select owner,table_name from dba_tables where table_name='TXK_TCC_RESULTS'

 1. Login to the database as APPS schema.
 2. Check the table TXK_TCC_RESULTS and verify that the information is same as FND_NODES.
select * from  FND_NODES;
 3. If the value in APPLSYS.TXK_TCC_RESULTS are not the same (for example: either Production database name after clone), perform the following steps:
 4. Create a backup table
create table APPS.TXK_TCC_RESULTS_backup as select * from APPLSYS.TXK_TCC_RESULTS;
 5. Truncate the APPLSYS.TXK_TCC_RESULTS table:
truncate table  APPLSYS.TXK_TCC_RESULTS;
 6. Download the ETCC Patch - 17537119.
 7.unzip the patch, run the following in Database Node
It will prompt for database(RDBMS) context file.
 8. Now check the APPLSYS.TXK_TCC_RESULTS again
If the information in the table is correct, proceed with adop phase.
 9. Also verify the v$parameter for "_system_trig_enabled",
SQL>select value from v$parameter where name='_system_trig_enabled';

########################TROUBLESHOOTING 7##############
Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone

Patch context file is missing in the FND_OAM_CONTEXT_FILES table. Line 1291 of line (see error below) is expecting the full path of CONTEXT_FILE name from the patch file system. However, the command returns no results.
To fix this issue test the following to upload the missing PATCH context file to the database:
1. Get the value of your patch context file.
    1.1. Source your PATCH file system
    1.2. Echo the value of $CONTEXT_FILE and take note of it. You will use this value in step #3.
2. Source the RUN filesystem
3. Execute the following command on the run filesystem to upload the patch context file to the database.
$ADJVAPRG action=upload contextfile=<contextfilehere> logfile=/tmp/patchctxupload.log
For Example:
$ADJVAPRG action=upload contextfile=/data1/oracle/EBSDEV/fs2/inst/apps/EBSDEV_ebsapp01/appl/admin/EBSDEV_ebsapp01.xml logfile=/tmp/patchctxupload.log
Note: the context file parameter should point to the physical location of the patch context file in your system
4. Run the following query and it should return one entry for the recently uploaded context file.
SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';
For any queries related to this article you can email me

Wednesday, May 31, 2017

Oracle EBS R12.2.x LOG FILES

Below are log file locations that Oracle EBS 12.2 Applications DBA need to access quite frequently.
Green color boxes indicates location where log files are.

Log files for Online patching (adop)
The adop log files are located on the non-editioned file system (fs_ne), under:

This log directory will contain patch logs, patch worker logs and other patch related log files created for specific purposes
Also, some patch tasks may create separate log files in the same directory.

Online pathcing filesystem cloning process related log files are found under:

Log files for Autoconfig process
On Applicaion Tier: $INST_TOP/admin/log/
On Database Tier: $ORACLE_HOME/appsutil/log//

Log files for start/stop of services from $ADMIN_SCRIPTS_HOME
In below directory you will find log files related to start/stop process of oacore, forms, apache,opmn, weblogic admin server/node manager:

Log/Out files for Concurrent programs/managersThe path is defined by same variables that we used earlier for 11i/R12.1 . Log/Out files are stored in Non-Editioned filesystem.
Log files: $APPLCSF/$APPLLOG (or $NE_BASE/inst//logs/appl/conc/log)
Out files: $APPLCSF/$APPLOUT (or $NE_BASE/inst//logs/appl/conc/out)

Log files for OPMN and OHS processesBelow directory contains log files related OPMN process(opmn.log), OPMN Debug
logs(debug.log), HTTP Transaction logs (access.log), security settings related logs.


Log file for Weblogic Node Manager
Log file is generated by Node Manager and contains data for all domains that are controlled by Node Manager on a given physical machine.

Log file for Weblogic Oracle Management Service log file
Initial settings AdminServer and Domain level information is written in this log file

Log files for server processes initiated through Weblogic
Stdout and stderr messages generated by the server instance (server instnaces like forms, oafm, oacore etc) at NOTICE severity level or higher are written by Weblogic Node Manager to below directory.

I hope you find this article useful. Kindly subscribe to it and if you have any queries send it to

Sunday, April 30, 2017

Granting Privileges On An Object in EBS 12.2.X

In 12.2 When grants are given to "Stub objects" it invalidates all dependent stub objects. "stub objects" are simply pointers to an actual object definition from an ancestor edition that is still being inherited by the given edition.
One of the most important new features of Oracle 11 (11gR2) release 2 is the ability to create object versions, making it easy to fix bugs in applications, allowing you to change procedures, triggers, views and packages without affecting application usage.

SQL> Select sys_context('userenv','current_edition_name') "Current_Edition"  from dual;

SQL> select object_name, object_type, edition_name from user_objects_ae where object_name ='PER_ALL_PEOPLE_F';
PER_ALL_PEOPLE_F SYNONYM               V_20170213_2045
PER_ALL_PEOPLE_F NON-EXISTENT      V_20170213_0218

ad_zd.grant_privs (created by patch 19674458:R12.AD.C) brings a new API for providing grants to objects with minimal invalidation. Henceforth granting in 12.2.x, should be using only ad_zd.grant_privs instead of 'grant' (DDL) directly

SQL> grant select on FND_CONCURRENT_REQUESTS TO apps;

X_PERMISSIONS in varchar2, /* comma-seperated permission list */
X_OBJECT_NAME in varchar2, /* name of APPS object or APPS synonym */
X_GRANTEE in varchar2, /* grantee schema or role */
X_OPTIONS in varchar2 default NULL); /* grant options, example: 'WITH GRANT OPTION' */

SQL> exec AD_ZD.grant_privs('SELECT', 'FND_RESPONSIBILITY', 'MSC',X_GRANT_TO_TABLE=>TRUE)PL/SQL procedure successfully completed.

SQL> exec AD_ZD.revoke_privs('SELECT','FND_RESPONSIBILITY','MSC');

The concept here is to execute the GRANT statement for a given editioned object in the edition containing the actual copy of that editioned object.  This is an unfortunate consequence of Edition-based Redefinition. This is fixed in Delta 6 with a new API: AD_ZD.GRANT_PRIVS.  The AD_ZD.GRANT_PRIVS procedure is meant to operate on EBS APPS "logical" objects, meaning the object present or exposed in the APPS schema and identified by its name or synonym in the APPS schema.  Such an object could be:
 1. An editioned code object,  actually present in the APPS schema.
 2. An effectively-editioned table (APPS Synonym -> Editioning View -> Table).
The GRANT_PRIVS API is specialized to the task of granting permissions on EBS objects (exposed in APPS) to other users.  For grants on editioned custom objects, you could make synonym to the custom object in APPS, and then run GRANT_PRIVS on that synonym name.  We could look at extending the support in GRANT_PRIVS to directly handle custom objects not exposed to APPS.

Workaround 1: Move to the Delta 6 code level (or higher) and grant privileges using AD_ZD.GRANT_PRIVS procedure instead of direct DDL

Workaround 2: Start an online patching cycle (adop phase=prepare), connect to the patch edition (source EBSapps.env patch), execute native DB grant statements in the patch edition, then complete the patching cycle (finalize, cutover, cleanup).

Patch 21841299 is now available, and recommended to uptake the latest changes,
Patch 19674458:R12.AD.C is available.
This patch creates a new API for providing grants to objects with minimal invalidation.
This patch prereqs R12.AD.C.DELTA5.  In addition, it is recommended that customer also apply 19581770:R12.AD.C, BUNDLE FIXES FOR R12.AD.C.DELTA.5 (18283295), before applying this patch.

Granting Privileges On An Object May Cause Invalidations (Doc ID 1987947.1)

For any queries, kindly email me at

Thursday, March 30, 2017

Recreate EBS 12.2.x Weblogic Domain

If you are facing errors mentioned in below Knowledge Base documents then the recommended solution from Oracle Support is to recreate the EBS Weblogic Domain.

Script Failed To Start AdminServer With "Cannot connect to Node Manager" Error (Doc ID 2086471.1)
After R12.2.4 Upgrade Found OPMNCTL Missing While Starting Services (Doc ID 1953456.1)

After applying AD/TXK Patches on 12.2.0, adop phase=cutover is failing on starting Apache OHS -- timing out (Doc ID 1952008.1)
12.2 E-Business Suite Users Report Error '10.4.5 404 Not Found From RFC 2068 Hypertext Transfer Protocol -- HTTP/1.1' When Attempting To Login After An Upgrade (Doc ID 2240807.1)

Working on EBS 12.2 is not same as how it was with 11i or R12.1.x, especially due to the tight dependency on weblogic. If you are a Fusion middleware DBA, you would know how it works and what is the relevance of the domain. The same things applies for 12.2.x also, all configuration comes under domain directory and if anything messed up there will cause a lot of issues and even you will not be able to start up anything as you have EBS depending completely on weblogic managed servers.
I have faced several issues working with 12.2 and issues with config.xml or passwords or while cloning the production to test, that might get complicated, so in this case you should think about the method of recreating the whole domain cleanly using the context file which is the base for creating all configuration files.
On Database Tier
1. Log into the database tier.
2. Source the environment.
3. Ensure that database and listener are up and running.
On Applications Tier
1. Source the RUN environment
2. $FND_TOP/bin/ -script=ChkEBSDependecies -server=ALL_SERVERS
3. cd $FND_TOP/patch/115/bin

Below prompts are shown by the script:
  Enter the full path of Applications Context File [DEFAULT - ]: Complete path of the RUN context file
  Enter the server start mode for the domain [DEFAULT - prod]: prod
  Enter the APPS schema password :
  Enter weblogic admin server password :

4. Now Start all middle tier services using $ADMIN_SCRIPTS_HOME/
5. This will fix the  issue in run filesystem. To propagate the change to patch filesystem run fs_clone.
The above command will create the whole EBS domain.
You can take backup of the existing domain before running above command for safety purpose or to compare files.

For any queries please email me on, and do not forget to subscribe to this blog so I will send you free AppsDBA/CoreDBA/FusionMiddleware Articles/Docs.

Monday, February 27, 2017

Change Weblogic password in EBS 12.2.x

The EBS WebLogic Domain uses Node Manager to control the Administration Server and the managed servers. For the EBS WebLogic Domain, the Node Manager and Oracle WebLogic Server Administration User password must be same, otherwise the AD control scripts will not work properly.

Important: If you need to change the Administration User password,you must change the Node Manager password first. If you do not do this, the WebLogic Server configuration change will not be detected and the next online patching cycle may fail.

The instructions that follow should be performed on the run file system. The password change will be automatically propagated to the patch file system during the next adop prepare phase or fs_clone operation.

1. Shut down all application tier services except the Admin Server.
On the primary node, run the command:
 $ <ADMIN_SCRIPTS_HOME>/ -skipNM -skipAdmin
 On all secondary nodes, run the command:

2. Change the Oracle WebLogic Server Administration User password by performing the following steps on the run file system of the primary node.
 Source the environment on the run file system.
 Run the commands appropriate for your platform:
 On UNIX, run the command:
 $ perl $FND_TOP/patch/115/bin/ -action=updateAdminPassword
The above command is available only if TXK DELTA 7 is applied.
If that is not the case, the password need to changed from Console and few steps need to be performed

3. Start all services on all nodes, using the command:

Steps for Changing  the weblogic password pre TXK DELTA 7
1)  Access Administrative console
2) Login using the current weblogic admin username and  password.
3) Select the lock and edit button and click domain
4) Click on Security TAB and then click on advanced tab
5) Provide the new password which you want to configure for weblogic admin user   in the node manager password field
6) Click on “save” and Verify settings updated successfully and then click on “Activate changes”
7) Now click on “Security Realms” and Click on “myrealm”
8)select “users and groups” and select weblogic User and then provide the new password for the weblogic user
9) Make sure the settings has been updated successfully

For any queries please email me on

Tuesday, January 31, 2017

Relocating RAC database and instances across nodes

Presuming you have an RAC database named RACNVT with an instance RACNVT1 running on raclinux1 node and you wish to relocate the instance permanently or temporally to raclinux2 node due to a prolonged maintenance scheduled on the raclunux1 node, or maybe the node resources are heavily consumed and you want to reduce the usage by relocating the instance. We are going to demonstrate here how to relocate an instance from one node to another.
The following output demonstrates where the RAC database and its instances are running:

[oracle@raclinux1 ~]$ ./
HA Resource Target                                              State
ora.RACNVT.RACNVT1.inst        ONLINE      ONLINE on raclinux1
ora.RACNVT.db                            ONLINE      ONLINE on raclinux2
ora.raclinux1.ASM1.asm                ONLINE      ONLINE on raclinux1
ora.raclinux1.LISTENER_RACLINUX1.lsnr   ONLINE   ONLINE on raclinux1
ora.raclinux1.LISTENER_RCONFIG_RACLINUX1.lsnr  ONLINE   ONLINE on raclinux1
ora.raclinux1.RACDB_RACLINUX1.lsnr   ONLINE   ONLINE on raclinux1
ora.raclinux1.gsd                            ONLINE        ONLINE on raclinux1
ora.raclinux1.ons                            ONLINE        ONLINE on raclinux1                             ONLINE       ONLINE on raclinux1
ora.raclinux2.ASM2.asm                 ONLINE       ONLINE on raclinux2
ora.raclinux2.LISTENER_RACLINUX2.lsnr    ONLINE   ONLINE on raclinux2
ora.raclinux2.RACDB_RACLINUX2.lsnr         ONLINE   ONLINE on raclinux2
ora.raclinux2.gsd                             ONLINE       ONLINE on raclinux2
ora.raclinux2.ons                             ONLINE       ONLINE on raclinux2                              ONLINE       ONLINE on raclinux2

This output confirms that the RAC database RACNVT is registered on the raclinux2 node and its instance RACNVT1 is running on the raclinux1 node, which is the expected behavior in an RAC environment. Assuming for any valid reasons you wish to relocate the instance from raclinux1 node to raclinux2 node, you will need to complete the following sequence on the raclinux2 node:

1. Prepare an spfile on raclinux2 for the RACNVT2 instance, and make any necessary changes. (We will change the instance name to match the node number.)
2. Create a password file for the RACNVT2 instance on raclinux2 node.
3. Ensure that the disk groups used for RACNVT database are mounted on raclinux2 node, via an ASM instance.
4. Create a Listener on raclinux2 node with the required port.
Relocating the instance
Once we have performed the above steps, let's move on to relocating the instance
using the examples mentioned below.
First, stop the instance on raclinux1 node using the following command:

#srvctl stop instance -d RACNVT -i RACNVT1

After the instance is stopped successfully, remove it from the raclinux1 node using the following command:

#srvctl remove instance -d RACNVT -i RACNVT1
Remove instance RACNVT1 from the database RACNVT? (y/[n]) y
If the RACNVT1 instance is not stopped properly when you attempt to remove it from
the node, the following cluster error will be shown:
PRKP-1023 : The instance {0} is still running.RACNVT

Adding the instance example
After successfully removing the instance from the node, use the following example
to add the instance on the raclinux2 node:
#srvctl add instance –d RACNVT –i RACNVT2 –n raclinux2
As we mentioned earlier, while adding the instance, you have the flexibility to
change its name according to the standards that you follow in your environment.
We changed the instance name from RACNVT1 to RACNVT2. However, ensure that
you have made the changes accordingly to your passwordfile and instance_name
parameter in the spfile to reflect the new name.
Once the instance is added successfully, bring up the instance using the following
cluster command:
#srvctl start instance -d RACNVT –I RACNVT2
Using the following cluster command, you can verify that the database/instance has
been successfully moved to raclinux2 node:
[oracle@raclinux2 ~]$ ./

HA Resource Target State
ora.RACNVT.RACNVT2.inst ONLINE ONLINE on raclinux2

For any questions on this topic, please email me on