Wednesday, August 30, 2017

AccessGate Login Fails with HTTP 500 Error after ADOP Cycle

I was facing the subject problem after successful ADOP cycle, Single Sign On SSO does not work and throws HTTP 500 Error at login.
Below shows that ADOP cycle was successful,
But when we try to access the application we get below error,
EBS 12.2 AccessGate Login Fails with HTTP 500 Error 'Request Failed for : /accessgate/ssologin?, Resp Code : [500]' On Secondary Nodes After A Complete ADOP Patching Cycle As File oaea_wls.properties is Not Updated On Multiple Filesystems (fs2)

In some cases fs_clone can fail to copy the file oaea_wls.properties to the other filesystem fs2 for the additional apps nodes.
For example, on the secondary apps tiers using a different filesystem (fs2), file oaea_wls.properties will have null settings for some parameters:
# +======================================================================+
# |    Copyright (c) 2005, 2014 Oracle and/or its affiliates.           |
# |                         All rights reserved.                         |
# |                           Version 12.0.0                             |
# +======================================================================+
# $Header: oaea_wls_properties.tmp 120.0.12020000.1 2014/07/15 12:36:37 mmanku noship $
#
# This file lists all the  parameters required to start acessgate application
# /u03/eruat/fs1/EBSapps/appl/fnd/12.0.0
# ###############################################################
#
# Do not edit settings in this file manually. They are managed
# automatically and will be overwritten when AutoConfig runs.
# For more information about AutoConfig, refer to the Oracle
# E-Business Suite Setup Guide.
#
# ###############################################################
SSO_SERVER_RELEASE=
SSO_SERVER_URL=
SSO_SERVER_TYPE=
LOG_CONFIG_FILE=DEFAULT
APPL_SERVER_ID=
OAM_LOGOUT_URL=
CONNECTION_REF=
WEBGATE_LOGOUT=
oracle.apps.fnd.sso.WebEntries=DEFAULT

In certain cases for an ONLINE patching cycle, the file $INST_TOP/appl/admin/oaea_wls.properties does not get copied from RUN to PATCH.
The issue is documented in the following unpublished defect:
Bug 25720619 - ERROR 500 INTERNAL SERVER ERROR TRYING TO LOGIN AFTER PATCH CYCLE
Workaround Fix:
To resolve the issue test the following steps in a development instance and then migrate accordingly:
1. Ensure a valid backup exists for the instance where the solution is being testing.
2. Copy file $INST_TOP/appl/admin/oaea_wls.properties from the original filesystem (fs1) over to the additional filesystems (fs2), to migrate the proper node settings to oaea_wls.properties
3. Restart the servers to ensure the file changes are processed.
4. Retest the login and confirm the error is resolved.
Following should be the content of the file oaea_wls.properties
This is the latest update as of 28-August-2017 for this issue, there is a patch for this defect 25720619 that is planned to be released in the next AD / TXK Delta patchset, with no release date information currently available.

For any queries on this article, kindly email me samiappsdba@gmail.com

Monday, July 31, 2017

Install Webgate, Deploy AccessGate, Register EBS with OAM

In this article we will see how to Install webgate, Deploy Access Gate and Register EBS with OAM.


1. Install Webgate and apply patch to Webgate Home Directory
Firstly copy the webgate patch p21118593 and webgate software to node 1 EBS server and then install using txkrun.pl script


# scp -rp p21118593_111230_Linux-x86-64.zip webgate appluat@ed-olraclin3:/home/appluat/ssoPatches
# txkrun.pl -script=SetOAMReg -installWebgate=yes -webgatestagedir=/home/appluat/ssoPatches/webgate

Now apply WEBGATE Patch to webgate HOME Directory
[appluat@ed-olraclin3 FMW_Home]$ export ORACLE_HOME=/u03/eruat/fs1/FMW_Home/Oracle_OAMWebGate1
[appluat@ed-olraclin3 FMW_Home]$ export PATH=/u03/eruat/fs1/FMW_Home/Oracle_OAMWebGate1/OPatch/:$PATH
[appluat@ed-olraclin3 ssoPatches]$ unzip p21118593_111230_Linux-x86-64.zip
[appluat@ed-olraclin3 21118593]$ pwd
/home/appluat/ssoPatches/21118593
[appluat@ed-olraclin3 21118593]$ opatch apply

2. Deploy Access Gate
[appluat@ed-olraclin3 21118593]$ wget http://uatoam.qa.qia.net:7014 (test the connection)--2017-06-18 10:44:45--  http://uatoam.qa.qia.net:7014/
Resolving uatoam.qa.qia.net (uatoam.qa.qia.net)... 192.168.1.10
Connecting to uatoam.qa.qia.net (uatoam.qa.qia.net)|192.168.1.10|:7014... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-06-18 10:44:45 ERROR 404: Not Found.



On EBS Node 1 (ed-olraclin3) server execute the below command,
# perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate -SSOServerURL=http://uatoam.qa.qia.net:7014 -managedsrvname=oaea_server1 -managedsrvport=6801 -logfile=/tmp/deployeag.log


On EBS Node 2 (ed-olraclin4) server execute the below command,
# perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-create-oaea_resources -contextfile=$CONTEXT_FILE -deployApps=accessgate -SSOServerURL=http://uatoam.qa.qia.net:7014 -managedsrvname=oaea_server2 -managedsrvport=6802 -logfile=/tmp/deployeag.log
On EBS Node 1 (ed-olraclin3) server execute the below command,
#perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=addMS -accessgate=ed-olraclin3.qa.qia.net:6801
On EBS Node 2 (ed-olraclin4) server execute the below command,
#perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=$CONTEXT_FILE -configoption=addMS -accessgate=ed-olraclin4.qa.qia.net:6802


--> Now Start Accessgate managed servers and make sure that accessgate + datasource are up and running. Test the datasource too from weblogic console datasources link.
3. Register EBS with OAM
Execute below command on EBS Node 1 ed-olraclin3 server
# txkrun.pl -script=SetOAMReg -registeroam=yes -oamHost=http://uatadmin.qa.qia.net:7012 -oamUserName=weblogic -ldapUrl=ldap://uatoid.qa.qia.net:3060 -oidUserName=cn=orcladmin -skipConfirm=yes -ldapSearchBase=cn=Users,dc=qa,dc=qia,dc=net -ldapGroupSearchBase=cn=Groups,dc=qa,dc=qia,dc=net



Execute below command on EBS Node 2 ed-olraclin4 server
#txkrun.pl -script=SetOAMReg -registeroam=yes -oamHost=http://uatadmin.qa.qia.net:7012 -oamUserName=weblogic -ldapUrl=ldap://uatoid.qa.qia.net:3060 -oidUserName=cn=orcladmin -ldapSearchBase=cn=Users, dc=qa,dc=qia,dc=net - ldapGroupSearchBase=cn=Groups, dc=qa,dc=qia,dc=net -authScheme=EBSAuthScheme -authSchemeMode=reference




Note: If you are setting up SSO for EBS then firstly you should be ready with OID, OAM setup and then do this article steps, set required EBS profile settings and finally restart all EBS Services and test EBS Single Sign On SSO.
Moreover, kindly follow my previously posted articles to do setup for OID, OAM, Windows Native Authentication WNA Zero Sign-on, etc.


For any further queries please email me on samiappsdba@gmail.com

Tuesday, June 20, 2017

Cloning Oracle EBS 12.2.5 Delta 7+

Run the adpreclone.pl script on existing ERP system that will be cloned to new system.
After adpreclone.sh script on primary server, COPY ONLY /u01/erpsource/fs1/EBSapps from Primary to Clone server.

cd $ADMIN_SCRIPTS_HOME
# perl adpreclone.pl appsTier

Now zip the EBSapps directory and copy to the target server where we will do the cloning,
#cd /u03/erpprd/fs1
#tar -cvf EBSapps.tar EBSapps
#cd /u03/erpprd/fs1
#gzip EBSapps.tar

Copy the EBSapps.gz file to the target server
#gunzip EBSapps.gz
#tar -xvf EBSapps.tar

On target server make sure all Pre-requisites are available that are mentioned in this Oracle Document,
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)
High level below are few pre-requisites,

systemctl disable bluetooth.service
systemctl disable libvirtd.service
systemctl disable NetworkManager.service
systemctl disable lvm2-monitor.service
systemctl disable lvm2-lvmetad.service
systemctl disable rdma.service
systemctl disable lvm2-lvmetad.socket
systemctl disable lvm2-lvmpolld.socket
systemctl stop firewalld.service
systemctl diabled firewalld.service
systemctl disabled firewalld.service
systemctl disable firewalld.service

Check pre-requisite RPMs for Database and also install the below rpm for application tier,

• compat-libstdc++-296-2.96-144.0.2.el7.i686
• openmotif21-2.1.30-11.EL7.i6861
• xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
Additionally, the following rpms need to be installed from the Oracle Linux 7 or RHEL 7 distribution media on the application tier:
• libXrender-0.9.8-2.1.el7.i686
• perl-File-CheckTree-4.42-3.el7.noarch
If not using the pre-install rpm, the following packages must be installed from the Oracle Linux 7 or RHEL 7 distribution media on both the application and database tiers:
• binutils-2.23.52.0.1-16.el7.x86_64
• compat-libstdc++-33-3.2.3-71.el7.i686
• gcc-4.8.2-16.el7.x86_64
• gcc-c++-4.8.2-16.el7.x86_64
• gdbm-1.10-8.el7.i686
• gdbm-1.10-8.el7.x86_64
• glibc-2.17-55.el7.i686
• glibc-2.17-55.el7.x86_64
• glibc-common-2.17-55.el7.x86_64
• glibc-devel-2.17-55.el7.i686
• glibc-devel-2.17-55.el7.x86_64
• libgcc-4.8.2-16.el7.i686
• libgcc-4.8.2-16.el7.x86_64
• libstdc++-devel-4.8.2-16.el7.i686
• libstdc++-devel-4.8.2-16.el7.x86_64
• libstdc++-4.8.2-16.el7.i686
• libstdc++-4.8.2-16.el7.x86_64
• libXi-1.7.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.x86_64
• libXtst-1.2.2-2.1.el7.i686
• libaio-0.3.109-12.el7.i686
• libaio-0.3.109-12.el7.x86_64
• libgomp-4.8.2-16.el7.x86_64
• make-3.82-21.el7.x86_64
• redhat-lsb-4.1-24.0.1.el7.x86_64
• sysstat-10.1.5-4.el7.x86_64
• util-linux-2.23.2-16.el7.x86_64
Additionally, the following RPMs are required for the database tier:
• compat-libcap1-1.10-7.el7.x86_64
• compat-libstdc++-33-3.2.3-71.el7.x86_64
• elfutils-libelf-devel-0.158-3.el7.x86_64
• libaio-devel-0.3.109-12.el7.i686
• libaio-devel-0.3.109-12.el7.x86_64
• xorg-x11-utils-7.5-13.1.el7.x86_64

?Install the below listed prerequisites RPM from DVD for Oracle Database
yum install binutils -y
yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install zlib-devel -y
yum install elfutils-libelf-devel -
  • Perform the following command (as root on your system) to create a required link to the Berkeley DB library
[root@erplx0 lib]# scp -r -p libdb-4.7.so vebsap2:/usr/lib
# cd /usr/lib                                                                      
# ln -s libdb-4.7.so libdb-4.3.so

  • DBb tier cloning
cd /u01/oracle/11.2.0.4/erphome1/oui/bin
./runInstaller -clone -silent -ignorePreReq ORACLE_HOME="/u01/oracle/11.2.0.4/erphome1" ORACLE_HOME_NAME="oracledb_home1" ORACLE_BASE="/u01/oracle" OSOPER_GROUP=oinstall

  • init.ora file check for these important parameters
service_names    string      erptest, ebs_patch

# perl adbldxml.pl appsuser=APPS appspass=apps
# ./adconfig.sh contextfile=/u01/oracle/11.2.0.4/erphome1/appsutil/erptest_erplx0.xml
  • IF AUTOCONFIG ON DATABASE TIER GIVES BELOW ERROR THEN EXECUTE THE .sh SCRIPTS MANUALLY.
WARNING: [AutoConfig Error Report]
  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0
      afdbprf.sh              INSTE8_PRF         1
  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0
      adcrobj.sh              INSTE8_APPLY       1

SOLUTION
Update TNSNAMES.ORA file and correct the ERPTEST tnsentry and then run the below two .sh scripts manually,
. /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0/afdbprf.sh
. /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0/adcrobj.sh

[oraerp@db02 bin]$ perl adbldxml.pl appsuser=APPS appspass=apps
Starting context file generation for db tier..
Using JVM from /u01/oracle/11.2.0.4/erphome1/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/u01/oracle/11.2.0.4/erphome1/appsutil/log/adbldxml_04171416.log
UnsatisfiedLinkError exception loading native library: njni11
java.lang.UnsatisfiedLinkError: no njni11 in java.library.path
Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: db02
Enter Port of Database server: 1522
Enter SID of Database server: erptest
Enter Database Service Name: erptest
Enter the value for Display Variable:
Invalid input, reenter the value for Display Variable: 0.0
The context file has been created at:
/u01/oracle/11.2.0.4/erphome1/appsutil/erptest_db02.xml

On Application Tier for adcfgclone.sh below are the options,
Target System Base Directory : /u01/erptest
Target System Instance Home Directory [/u01/erptest] :
Do you want to preserve the Display [erplx0:0.0] (y/n)  : n
Target System Display [erplx0:0.0] :
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

[root@erplx0 etc]# cd /u01/erpt/
[root@erplx0 erptest]# mkdir oraInventory
chown -R appltst:oinstall oraInventory

as root user, create new file /etc/oraInst.loc with below content.
[root@erplx0 etc]# vi oraInst.loc
inventory_loc=/u01/erpt/oraInventory
inst_group=oinstall
[root@erplx0 ~]# usermod -g oinstall -G oinstall appltst
[root@erplx0 ~]# id appltst
uid=1001(appltst) gid=54322(oinstall) groups=54322(oinstall)
  • To clean the oracle_home entries from oraInventory run the below
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1

SQL> SELECT ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

ADOP_SESSION_ID P A F C C A S ABANDON_FLAG NODE_NAME
-----------------------------------------------------
              2 X Y N X Y X C              lxap1
              4 Y Y Y Y Y X C              lxap1
              5 Y Y Y Y Y X C              lxap1
              6 Y Y Y Y Y X C              lxap1
              7 Y Y Y Y Y X C              lxap1
              8 Y Y Y Y Y X C              lxap1
              9 Y Y Y Y Y X C              lxap1
             10 Y Y Y Y Y X C             lxap1
             10 Y Y Y Y Y X C             lxap2
             11 Y Y Y Y Y X C             lxap1
             11 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap1
             12 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             lxap1
             13 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             irec
             14 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             lxap2
             15 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             irec
             16 Y Y Y Y Y X C             lxap1
             16 Y Y Y Y Y X C             lxap2
             16 Y Y Y Y Y X C             irec
             17 Y Y Y Y Y X C             lxap1
             17 Y Y Y Y Y X C             lxap2
             17 Y Y Y Y Y X C             irec
             18 Y Y Y Y Y X C             lxtst0
             19 Y Y Y Y Y X C             lxtst0
             20 Y Y Y Y Y X C             lxtst0
             21 Y Y Y Y Y X C             lxtst0
33 rows selected.

As can be seen above last online patching session (11) was done successfully till cleanup stage (marked as Y)

SQL> SELECT abbreviation,codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');
ABBREVIATION                   CODELEVEL
ad                             C.8
txk                            C.8
Above means we are on delta8 patch level, hence we can use the 'dualfs' option while cloning from Delta 7 onwards.

#cd /u01/erptest/fs1/EBSapps/comn/clone/bin
#perl adcfgclone.pl appsTier dualfs

POST CLONE STEPS
1. Change APPS and Weblogic passwords
2. update all context files ----> https to http in both FS1 & FS2
3. fnd_conc_clone.setup_clean;
4. run autoconfig on patch FS (.EBSapps.env patch)--->if patch edition auto config error then disable ebs_logon trigger and later enable it.
5. run autoconfig on run FS (.EBSapps.env run)


  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system
[applerp@erplx0 ~]$ more /u01/erptest/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml | grep https
<EndUserMonitoringURL oa_var="s_endUserMonitoringURL">https://erplx0.ad.councilmolqia.net:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif</EndUserMonitoringURL>
  <externURL oa_var="s_external_url">https://erplx0.ad.councilmolqia.net:8001</externURL>
    <login_page oa_var="s_login_page">https://erplx0.ad.councilmolqia.net:8001/OA_HTML/AppsLogin</login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">https</webentryurlprotocol>

MODIFY THE CONTEXT FILES OF FS1 AND FS2 AS BELOW

<EndUserMonitoringURL oa_var="s_endUserMonitoringURL">http://erplx0.ad.councilmolqia.net:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif</EndUserMonitoringURL>
  <externURL oa_var="s_external_url">http://erplx0.ad.councilmolqia.net:8001</externURL>
    <login_page oa_var="s_login_page">http://erplx0.ad.councilmolqia.net:8001/OA_HTML/AppsLogin</login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">http</webentryurlprotocol>

vi /u01/erptst/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
vi /u01/erptest/fs2/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
:%s/https/http/g -- then change http to http for only this line---><httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
  • AFTER ABOVE CHANGE FND_CONC_CLONE.SETUP CLEAN AND THEN RUN THE AUTOCONFIG AS BELOW,
THEN RUN THE AUTO CONFIG ON BOTH 'PATCH' AND 'RUN' FILE SYSTEMS.
. /u01/erptst/EBSapps.env patch
cd $ADMIN_SCRIPTS_HOME
sh adautocfg.sh
. /u01/erptst/EBSapps.env run
cd $ADMIN_SCRIPTS_HOME
sh adautocfg.sh
---->IF THIS FAILS THEN SQL> ALTER TRIGGER EBS_LOGON DISABLE; THEN RUN AUTOCONFIG AGAIN and then enable the trigger and then run autoconfig from run file system.
SQL> select owner,trigger_name,trigger_type,status from dba_triggers where trigger_name='EBS_LOGON';
OWNER                          TRIGGER_NAME                   TRIGGER_TYPE STATUS
SYSTEM                         EBS_LOGON                      AFTER EVENT ENABLED

  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system

  • CHANGE WEBLOGIC PASSWORD
1. Shut down all application tier services except the Admin Server.
On the primary node, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh -skipNM -skipAdmin
 On all secondary nodes, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh
2. Change the Oracle WebLogic Server Administration User password by performing the following steps on the run file system of the primary node.
 Source the environment on the run file system.
 Run the commands appropriate for your platform:
 On UNIX, run the command:
 $ perl $FND_TOP/patch/115/bin/txkUpdateEBSDomain.pl -action=updateAdminPassword
The above command is available only if TXK DELTA 7 is applied.
If that is not the case, the password need to changed from Console and few steps need to be performed
3. Start all services on all nodes, using the command:
 $ <ADMIN_SCRIPTS_HOME>/adstrtal.sh
  • Do not do the below on cloned instance as it is done only on production. On clone instance it is not required to enable the workflow mailer after cloning for test environments. 
Setup this profile for mailer:
WF: Workflow Mailer Framework Web Agent:  http://erp.ad.councilmolqia.net:8011
Please change the value to appropriate server.
Also run the below script to setup the Workflow Admin role
SQL> update wf_resources
set text='FND_RESP|FND|COUNCILMOLQIA__WF_ADMIN|STANDARD'
where name = 'WF_ADMIN_ROLE';
SQL> COMMIT;

########################TROUBLESHOOTING 1##############
  • 12.2 patch edition auto config error
txkGenADOPWrapper.pl  INSTE8_APPLY 1
Solution:
To run autoconfig from patch file system you must disable trigger ebs_login prior to running autoconfig
1. alter trigger ebs_logon disable;
2. source patch file system and run autoconfig successfully.
3. alter trigger ebs_logon enable;
4. source patch file system and then connect as SYSTEM user using sqlplus

########################TROUBLESHOOTING 2##############
ERROR:
Enter the APPS password :
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object
chmod: cannot access /u01/erptest/fs1/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt: No such file or directory
SOLUTION:
In /etc/fstab file make xfs file system 32bit. (REFERENCE: https://access.redhat.com/solutions/2221761)
/dev/oracle_vg1/u01_v                           /u01                    xfs     rw,inode32      0 0
AND
check for jre and jdk 32 bit folders, ./java -version

########################TROUBLESHOOTING 3##############
ERROR: while running ./runInstaller -deinstall option we are getting below error,
The Java RunTime Environment was not found at /tmp/OraInstall/jdk/jre/bin/java
SOLUTION:
oraparam.ini file under the software staged directory path /software/FR11121/Disk1/install/linux64 had an entry JRE_SCRATCH_LOC
 It was set to
JRE_LOC=../../jdk/jre
UPDATE oraparam.ini to use another jre or fix the corrupted jre files in "/u01/erpt/fs1/EBSapps/10.1.2/jdk" folder.

########################TROUBLESHOOTING 4##############
After installing oracle required patches run the below command as root user,
ldconfig -v

check the below
[root@erplx0 lib]# ls -ltr libdb*
-rwxr-xr-x 1 root root 1691492 Sep 11  2014 libdb_cxx-4.7.so
-rwxr-xr-x 1 root root 1575444 Sep 11  2014 libdb-4.7.so
-rwxrwxrwx 1 root root    5825 Aug  3  2016 libdb.so.2
lrwxrwxrwx 1 root root      12 Apr  3 14:33 libdb-4.3.so -> libdb-4.7.so

if link is not there then create it as below,
# ln -s libdb-4.7.so libdb-4.3.so
check if this file exists or not /usr/lib/libdb.so.2

########################TROUBLESHOOTING 5##############
Error that FMW_Home already exists the,
cd /u01/erptest/fs1/FMW_Home/oracle_common/oui/bin
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
>>Then remove all folders except fs1 and re-run adcfgclone.sh script.
[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erpt/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.

########################TROUBLESHOOTING 6##############
R12.2.x ADOP : [ERROR] ETCC not run in the database node DBHOSTNAME

This table TXK_TCC-RESULTS is created in APPLSYS schema, by the latest version of the checkDBpatch.sh script delivered by 17537119.

select owner,table_name from dba_tables where table_name='TXK_TCC_RESULTS'
OWNER   TABLE_NAME
APPLSYS TXK_TCC_RESULTS

Solution:
 1. Login to the database as APPS schema.
 2. Check the table TXK_TCC_RESULTS and verify that the information is same as FND_NODES.
select * from APPLSYS.TXK_TCC_RESULTS;
select * from  FND_NODES;
 3. If the value in APPLSYS.TXK_TCC_RESULTS are not the same (for example: either Production database name after clone), perform the following steps:
 4. Create a backup table
create table APPS.TXK_TCC_RESULTS_backup as select * from APPLSYS.TXK_TCC_RESULTS;
 5. Truncate the APPLSYS.TXK_TCC_RESULTS table:
truncate table  APPLSYS.TXK_TCC_RESULTS;
 6. Download the ETCC Patch - 17537119.
 7.unzip the patch, run the following in Database Node
sh checkDBpatch.sh
It will prompt for database(RDBMS) context file.
 8. Now check the APPLSYS.TXK_TCC_RESULTS again
select * from APPLSYS.TXK_TCC_RESULTS;
If the information in the table is correct, proceed with adop phase.
 9. Also verify the v$parameter for "_system_trig_enabled",
SQL>select value from v$parameter where name='_system_trig_enabled';
VALUE
TRUE

########################TROUBLESHOOTING 7##############
Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone

CAUSE
Patch context file is missing in the FND_OAM_CONTEXT_FILES table. Line 1291 of ADOPValidationUtils.pm line (see error below) is expecting the full path of CONTEXT_FILE name from the patch file system. However, the command returns no results.
SOLUTION
To fix this issue test the following to upload the missing PATCH context file to the database:
1. Get the value of your patch context file.
    1.1. Source your PATCH file system
    1.2. Echo the value of $CONTEXT_FILE and take note of it. You will use this value in step #3.
2. Source the RUN filesystem
3. Execute the following command on the run filesystem to upload the patch context file to the database.
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=<contextfilehere> logfile=/tmp/patchctxupload.log
For Example:
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/data1/oracle/EBSDEV/fs2/inst/apps/EBSDEV_ebsapp01/appl/admin/EBSDEV_ebsapp01.xml logfile=/tmp/patchctxupload.log
Note: the context file parameter should point to the physical location of the patch context file in your system
4. Run the following query and it should return one entry for the recently uploaded context file.
SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';
For any queries related to this article you can email me samiappsdba@gmail.com

Wednesday, May 31, 2017

Oracle EBS R12.2.x LOG FILES

Below are log file locations that Oracle EBS 12.2 Applications DBA need to access quite frequently.
Green color boxes indicates location where log files are.




Log files for Online patching (adop)
The adop log files are located on the non-editioned file system (fs_ne), under:
$NE_BASE/EBSapps/log/adop//__//log

This log directory will contain patch logs, patch worker logs and other patch related log files created for specific purposes
Also, some patch tasks may create separate log files in the same directory.

Online pathcing filesystem cloning process related log files are found under:
$INST_TOP/admin/log


Log files for Autoconfig process
On Applicaion Tier: $INST_TOP/admin/log/
On Database Tier: $ORACLE_HOME/appsutil/log//



Log files for start/stop of services from $ADMIN_SCRIPTS_HOME
In below directory you will find log files related to start/stop process of oacore, forms, apache,opmn, weblogic admin server/node manager:
$LOG_HOME/appl/admin/log



Log/Out files for Concurrent programs/managersThe path is defined by same variables that we used earlier for 11i/R12.1 . Log/Out files are stored in Non-Editioned filesystem.
Log files: $APPLCSF/$APPLLOG (or $NE_BASE/inst//logs/appl/conc/log)
Out files: $APPLCSF/$APPLOUT (or $NE_BASE/inst//logs/appl/conc/out)



Log files for OPMN and OHS processesBelow directory contains log files related OPMN process(opmn.log), OPMN Debug
logs(debug.log), HTTP Transaction logs (access.log), security settings related logs.

$IAS_ORACLE_HOME/instances//diagnostics/logs


Log file for Weblogic Node Manager
Log file is generated by Node Manager and contains data for all domains that are controlled by Node Manager on a given physical machine.
$FMW_HOME/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.log



Log file for Weblogic Oracle Management Service log file
Initial settings AdminServer and Domain level information is written in this log file
$EBS_DOMAIN_HOME/sysman/log



Log files for server processes initiated through Weblogic
Stdout and stderr messages generated by the server instance (server instnaces like forms, oafm, oacore etc) at NOTICE severity level or higher are written by Weblogic Node Manager to below directory.
$EBS_DOMAIN_HOME/servers//logs/.out



I hope you find this article useful. Kindly subscribe to it and if you have any queries send it to samiappsdba@gmail.com


Sunday, April 30, 2017

Granting Privileges On An Object in EBS 12.2.X

In 12.2 When grants are given to "Stub objects" it invalidates all dependent stub objects. "stub objects" are simply pointers to an actual object definition from an ancestor edition that is still being inherited by the given edition.
One of the most important new features of Oracle 11 (11gR2) release 2 is the ability to create object versions, making it easy to fix bugs in applications, allowing you to change procedures, triggers, views and packages without affecting application usage.

SQL> Select sys_context('userenv','current_edition_name') "Current_Edition"  from dual;
Current_Edition
V_20170414_1016


SQL> select object_name, object_type, edition_name from user_objects_ae where object_name ='PER_ALL_PEOPLE_F';
OBJECT_NAME           OBJECT_TYPE        EDITION_NAME
PER_ALL_PEOPLE_F SYNONYM               V_20170213_2045
PER_ALL_PEOPLE_F NON-EXISTENT      ORA$BASE
PER_ALL_PEOPLE_F NON-EXISTENT      V_20170213_0218


ad_zd.grant_privs (created by patch 19674458:R12.AD.C) brings a new API for providing grants to objects with minimal invalidation. Henceforth granting in 12.2.x, should be using only ad_zd.grant_privs instead of 'grant' (DDL) directly

SQL> grant select on FND_CONCURRENT_REQUESTS TO apps;

Syntax:
ad_zd.grant_privs(
X_PERMISSIONS in varchar2, /* comma-seperated permission list */
X_OBJECT_NAME in varchar2, /* name of APPS object or APPS synonym */
X_GRANTEE in varchar2, /* grantee schema or role */
X_OPTIONS in varchar2 default NULL); /* grant options, example: 'WITH GRANT OPTION' */


Example
SQL> exec AD_ZD.grant_privs('SELECT', 'FND_RESPONSIBILITY', 'MSC',X_GRANT_TO_TABLE=>TRUE)PL/SQL procedure successfully completed.

SQL> exec AD_ZD.revoke_privs('SELECT','FND_RESPONSIBILITY','MSC');

The concept here is to execute the GRANT statement for a given editioned object in the edition containing the actual copy of that editioned object.  This is an unfortunate consequence of Edition-based Redefinition. This is fixed in Delta 6 with a new API: AD_ZD.GRANT_PRIVS.  The AD_ZD.GRANT_PRIVS procedure is meant to operate on EBS APPS "logical" objects, meaning the object present or exposed in the APPS schema and identified by its name or synonym in the APPS schema.  Such an object could be:
 1. An editioned code object,  actually present in the APPS schema.
 2. An effectively-editioned table (APPS Synonym -> Editioning View -> Table).
The GRANT_PRIVS API is specialized to the task of granting permissions on EBS objects (exposed in APPS) to other users.  For grants on editioned custom objects, you could make synonym to the custom object in APPS, and then run GRANT_PRIVS on that synonym name.  We could look at extending the support in GRANT_PRIVS to directly handle custom objects not exposed to APPS.

Workaround 1: Move to the Delta 6 code level (or higher) and grant privileges using AD_ZD.GRANT_PRIVS procedure instead of direct DDL

Workaround 2: Start an online patching cycle (adop phase=prepare), connect to the patch edition (source EBSapps.env patch), execute native DB grant statements in the patch edition, then complete the patching cycle (finalize, cutover, cleanup).

Note:
Patch 21841299 R12.AD.C.delta.8 is now available, and recommended to uptake the latest changes,
Patch 19674458:R12.AD.C is available.
This patch creates a new API for providing grants to objects with minimal invalidation.
This patch prereqs R12.AD.C.DELTA5.  In addition, it is recommended that customer also apply 19581770:R12.AD.C, BUNDLE FIXES FOR R12.AD.C.DELTA.5 (18283295), before applying this patch.

References:
Granting Privileges On An Object May Cause Invalidations (Doc ID 1987947.1)
http://nuijten.blogspot.ae/2010/11/edition-based-redefinition-and.html

For any queries, kindly email me at samiappsdba@gmail.com

Thursday, March 30, 2017

Recreate EBS 12.2.x Weblogic Domain

If you are facing errors mentioned in below Knowledge Base documents then the recommended solution from Oracle Support is to recreate the EBS Weblogic Domain.


Script adadminsrvctl.sh Failed To Start AdminServer With "Cannot connect to Node Manager" Error (Doc ID 2086471.1)
After R12.2.4 Upgrade Found OPMNCTL Missing While Starting Services (Doc ID 1953456.1)

After applying AD/TXK Patches on 12.2.0, adop phase=cutover is failing on starting Apache OHS -- timing out (Doc ID 1952008.1)
12.2 E-Business Suite Users Report Error '10.4.5 404 Not Found From RFC 2068 Hypertext Transfer Protocol -- HTTP/1.1' When Attempting To Login After An Upgrade (Doc ID 2240807.1)


Working on EBS 12.2 is not same as how it was with 11i 11.5.10.2 or R12.1.x, especially due to the tight dependency on weblogic. If you are a Fusion middleware DBA, you would know how it works and what is the relevance of the domain. The same things applies for 12.2.x also, all configuration comes under domain directory and if anything messed up there will cause a lot of issues and even you will not be able to start up anything as you have EBS depending completely on weblogic managed servers.
I have faced several issues working with 12.2 and issues with config.xml or passwords or while cloning the production to test, that might get complicated, so in this case you should think about the method of recreating the whole domain cleanly using the context file which is the base for creating all configuration files.
On Database Tier
1. Log into the database tier.
2. Source the environment.
3. Ensure that database and listener are up and running.
On Applications Tier
1. Source the RUN environment
2. $FND_TOP/bin/txkrun.pl -script=ChkEBSDependecies -server=ALL_SERVERS
3. cd $FND_TOP/patch/115/bin
   perl txkEBSDomainConfig.pl

Below prompts are shown by the script:
  Enter the full path of Applications Context File [DEFAULT - ]: Complete path of the RUN context file
  Enter the server start mode for the domain [DEFAULT - prod]: prod
  Enter the APPS schema password :
  Enter weblogic admin server password :

4. Now Start all middle tier services using $ADMIN_SCRIPTS_HOME/adstrtal.sh
5. This will fix the  issue in run filesystem. To propagate the change to patch filesystem run fs_clone.
The above command will create the whole EBS domain.
You can take backup of the existing domain before running above command for safety purpose or to compare files.


For any queries please email me on samiappsdba@gmail.com, and do not forget to subscribe to this blog so I will send you free AppsDBA/CoreDBA/FusionMiddleware Articles/Docs.

Monday, February 27, 2017

Change Weblogic password in EBS 12.2.x

The EBS WebLogic Domain uses Node Manager to control the Administration Server and the managed servers. For the EBS WebLogic Domain, the Node Manager and Oracle WebLogic Server Administration User password must be same, otherwise the AD control scripts will not work properly.

Important: If you need to change the Administration User password,you must change the Node Manager password first. If you do not do this, the WebLogic Server configuration change will not be detected and the next online patching cycle may fail.

The instructions that follow should be performed on the run file system. The password change will be automatically propagated to the patch file system during the next adop prepare phase or fs_clone operation.

1. Shut down all application tier services except the Admin Server.
On the primary node, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh -skipNM -skipAdmin
 On all secondary nodes, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh

2. Change the Oracle WebLogic Server Administration User password by performing the following steps on the run file system of the primary node.
 Source the environment on the run file system.
 Run the commands appropriate for your platform:
 On UNIX, run the command:
 $ perl $FND_TOP/patch/115/bin/txkUpdateEBSDomain.pl -action=updateAdminPassword
The above command is available only if TXK DELTA 7 is applied.
If that is not the case, the password need to changed from Console and few steps need to be performed

3. Start all services on all nodes, using the command:
 $ <ADMIN_SCRIPTS_HOME>/adstrtal.sh

Steps for Changing  the weblogic password pre TXK DELTA 7
1)  Access Administrative console
2) Login using the current weblogic admin username and  password.
3) Select the lock and edit button and click domain
4) Click on Security TAB and then click on advanced tab
5) Provide the new password which you want to configure for weblogic admin user   in the node manager password field
6) Click on “save” and Verify settings updated successfully and then click on “Activate changes”
7) Now click on “Security Realms” and Click on “myrealm”
8)select “users and groups” and select weblogic User and then provide the new password for the weblogic user
9) Make sure the settings has been updated successfully

For any queries please email me on samiappsdba@gmail.com

Tuesday, January 31, 2017

Relocating RAC database and instances across nodes

Presuming you have an RAC database named RACNVT with an instance RACNVT1 running on raclinux1 node and you wish to relocate the instance permanently or temporally to raclinux2 node due to a prolonged maintenance scheduled on the raclunux1 node, or maybe the node resources are heavily consumed and you want to reduce the usage by relocating the instance. We are going to demonstrate here how to relocate an instance from one node to another.
The following output demonstrates where the RAC database and its instances are running:


[oracle@raclinux1 ~]$ ./crs_stat.sh
HA Resource Target                                              State
ora.RACNVT.RACNVT1.inst        ONLINE      ONLINE on raclinux1
ora.RACNVT.db                            ONLINE      ONLINE on raclinux2
ora.raclinux1.ASM1.asm                ONLINE      ONLINE on raclinux1
ora.raclinux1.LISTENER_RACLINUX1.lsnr   ONLINE   ONLINE on raclinux1
ora.raclinux1.LISTENER_RCONFIG_RACLINUX1.lsnr  ONLINE   ONLINE on raclinux1
ora.raclinux1.RACDB_RACLINUX1.lsnr   ONLINE   ONLINE on raclinux1
ora.raclinux1.gsd                            ONLINE        ONLINE on raclinux1
ora.raclinux1.ons                            ONLINE        ONLINE on raclinux1
ora.raclinux1.vip                             ONLINE       ONLINE on raclinux1
ora.raclinux2.ASM2.asm                 ONLINE       ONLINE on raclinux2
ora.raclinux2.LISTENER_RACLINUX2.lsnr    ONLINE   ONLINE on raclinux2
ora.raclinux2.RACDB_RACLINUX2.lsnr         ONLINE   ONLINE on raclinux2
ora.raclinux2.gsd                             ONLINE       ONLINE on raclinux2
ora.raclinux2.ons                             ONLINE       ONLINE on raclinux2
ora.raclinux2.vip                              ONLINE       ONLINE on raclinux2







This output confirms that the RAC database RACNVT is registered on the raclinux2 node and its instance RACNVT1 is running on the raclinux1 node, which is the expected behavior in an RAC environment. Assuming for any valid reasons you wish to relocate the instance from raclinux1 node to raclinux2 node, you will need to complete the following sequence on the raclinux2 node:

1. Prepare an spfile on raclinux2 for the RACNVT2 instance, and make any necessary changes. (We will change the instance name to match the node number.)
2. Create a password file for the RACNVT2 instance on raclinux2 node.
3. Ensure that the disk groups used for RACNVT database are mounted on raclinux2 node, via an ASM instance.
4. Create a Listener on raclinux2 node with the required port.
Relocating the instance
Once we have performed the above steps, let's move on to relocating the instance
using the examples mentioned below.
First, stop the instance on raclinux1 node using the following command:

#srvctl stop instance -d RACNVT -i RACNVT1


After the instance is stopped successfully, remove it from the raclinux1 node using the following command:

#srvctl remove instance -d RACNVT -i RACNVT1
Remove instance RACNVT1 from the database RACNVT? (y/[n]) y
If the RACNVT1 instance is not stopped properly when you attempt to remove it from
the node, the following cluster error will be shown:
PRKP-1023 : The instance {0} is still running.RACNVT


Adding the instance example
After successfully removing the instance from the node, use the following example
to add the instance on the raclinux2 node:
#srvctl add instance –d RACNVT –i RACNVT2 –n raclinux2
As we mentioned earlier, while adding the instance, you have the flexibility to
change its name according to the standards that you follow in your environment.
We changed the instance name from RACNVT1 to RACNVT2. However, ensure that
you have made the changes accordingly to your passwordfile and instance_name
parameter in the spfile to reflect the new name.
Once the instance is added successfully, bring up the instance using the following
cluster command:
#srvctl start instance -d RACNVT –I RACNVT2
Using the following cluster command, you can verify that the database/instance has
been successfully moved to raclinux2 node:
[oracle@raclinux2 ~]$ ./crs_stat.sh

HA Resource Target State
ora.RACNVT.RACNVT2.inst ONLINE ONLINE on raclinux2



For any questions on this topic, please email me on samiappsdba@gmail.com