Tuesday, June 20, 2017

Cloning Oracle EBS 12.2.5 (Delta 7+)

Run the adpreclone.pl script on existing ERP system that will be cloned to new system.
After adpreclone.sh script on primary server, COPY ONLY /u01/erpsource/fs1/EBSapps from Primary to Clone server.

cd $ADMIN_SCRIPTS_HOME
# perl adpreclone.pl appsTier

Now zip the EBSapps directory and copy to the target server where we will do the cloning,
#cd /u03/erpprd/fs1
#tar -cvf EBSapps.tar EBSapps
#cd /u03/erpprd/fs1
#gzip EBSapps.tar

Copy the EBSapps.gz file to the target server
#gunzip EBSapps.gz
#tar -xvf EBSapps.tar

On target server make sure all Pre-requisites are available that are mentioned in this Oracle Document,
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)
High level below are few pre-requisites,

systemctl disable bluetooth.service
systemctl disable libvirtd.service
systemctl disable NetworkManager.service
systemctl disable lvm2-monitor.service
systemctl disable lvm2-lvmetad.service
systemctl disable rdma.service
systemctl disable lvm2-lvmetad.socket
systemctl disable lvm2-lvmpolld.socket
systemctl stop firewalld.service
systemctl diabled firewalld.service
systemctl disabled firewalld.service
systemctl disable firewalld.service

Check pre-requisite RPMs for Database and also install the below rpm for application tier,

• compat-libstdc++-296-2.96-144.0.2.el7.i686
• openmotif21-2.1.30-11.EL7.i6861
• xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
Additionally, the following rpms need to be installed from the Oracle Linux 7 or RHEL 7 distribution media on the application tier:
• libXrender-0.9.8-2.1.el7.i686
• perl-File-CheckTree-4.42-3.el7.noarch
If not using the pre-install rpm, the following packages must be installed from the Oracle Linux 7 or RHEL 7 distribution media on both the application and database tiers:
• binutils-2.23.52.0.1-16.el7.x86_64
• compat-libstdc++-33-3.2.3-71.el7.i686
• gcc-4.8.2-16.el7.x86_64
• gcc-c++-4.8.2-16.el7.x86_64
• gdbm-1.10-8.el7.i686
• gdbm-1.10-8.el7.x86_64
• glibc-2.17-55.el7.i686
• glibc-2.17-55.el7.x86_64
• glibc-common-2.17-55.el7.x86_64
• glibc-devel-2.17-55.el7.i686
• glibc-devel-2.17-55.el7.x86_64
• libgcc-4.8.2-16.el7.i686
• libgcc-4.8.2-16.el7.x86_64
• libstdc++-devel-4.8.2-16.el7.i686
• libstdc++-devel-4.8.2-16.el7.x86_64
• libstdc++-4.8.2-16.el7.i686
• libstdc++-4.8.2-16.el7.x86_64
• libXi-1.7.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.i686
• libXp-1.0.2-2.1.el7.x86_64
• libXtst-1.2.2-2.1.el7.i686
• libaio-0.3.109-12.el7.i686
• libaio-0.3.109-12.el7.x86_64
• libgomp-4.8.2-16.el7.x86_64
• make-3.82-21.el7.x86_64
• redhat-lsb-4.1-24.0.1.el7.x86_64
• sysstat-10.1.5-4.el7.x86_64
• util-linux-2.23.2-16.el7.x86_64
Additionally, the following RPMs are required for the database tier:
• compat-libcap1-1.10-7.el7.x86_64
• compat-libstdc++-33-3.2.3-71.el7.x86_64
• elfutils-libelf-devel-0.158-3.el7.x86_64
• libaio-devel-0.3.109-12.el7.i686
• libaio-devel-0.3.109-12.el7.x86_64
• xorg-x11-utils-7.5-13.1.el7.x86_64

?Install the below listed prerequisites RPM from DVD for Oracle Database
yum install binutils -y
yum install compat-libstdc++-33 -y
yum install compat-libstdc++-33.i686 -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc.i686 -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y
yum install ksh -y
yum install libgcc -y
yum install libgcc.i686 -y
yum install libstdc++ -y
yum install libstdc++.i686 -y
yum install libstdc++-devel -y
yum install libstdc++-devel.i686 -y
yum install libaio -y
yum install libaio.i686 -y
yum install libaio-devel -y
yum install libaio-devel.i686 -y
yum install libXext -y
yum install libXext.i686 -y
yum install libXtst -y
yum install libXtst.i686 -y
yum install libX11 -y
yum install libX11.i686 -y
yum install libXau -y
yum install libXau.i686 -y
yum install libxcb -y
yum install libxcb.i686 -y
yum install libXi -y
yum install libXi.i686 -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel -y
yum install zlib-devel -y
yum install elfutils-libelf-devel -
  • Perform the following command (as root on your system) to create a required link to the Berkeley DB library
[root@erplxtst0 lib]# scp -r -p libdb-4.7.so vebsap2:/usr/lib
# cd /usr/lib                                                                      
# ln -s libdb-4.7.so libdb-4.3.so

  • DBb tier cloning
cd /u01/oracle/11.2.0.4/erphome1/oui/bin
./runInstaller -clone -silent -ignorePreReq ORACLE_HOME="/u01/oracle/11.2.0.4/erphome1" ORACLE_HOME_NAME="oracledb_home1" ORACLE_BASE="/u01/oracle" OSOPER_GROUP=oinstall

  • init.ora file check for these important parameters
service_names    string      erptest, ebs_patch

# perl adbldxml.pl appsuser=APPS appspass=apps
# ./adconfig.sh contextfile=/u01/oracle/11.2.0.4/erphome1/appsutil/erptest_erplx0.xml
  • IF AUTOCONFIG ON DATABASE TIER GIVES BELOW ERROR THEN EXECUTE THE .sh SCRIPTS MANUALLY.
WARNING: [AutoConfig Error Report]
  [PROFILE PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0
      afdbprf.sh              INSTE8_PRF         1
  [APPLY PHASE]
  AutoConfig could not successfully execute the following scripts:
    Directory: /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0
      adcrobj.sh              INSTE8_APPLY       1

SOLUTION
Update TNSNAMES.ORA file and correct the ERPTEST tnsentry and then run the below two .sh scripts manually,
. /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0/afdbprf.sh
. /u01/oracle/11.2.0.4/erphome1/appsutil/install/erptest_erplx0/adcrobj.sh

[oraerp@db02 bin]$ perl adbldxml.pl appsuser=APPS appspass=apps
Starting context file generation for db tier..
Using JVM from /u01/oracle/11.2.0.4/erphome1/appsutil/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/u01/oracle/11.2.0.4/erphome1/appsutil/log/adbldxml_04171416.log
UnsatisfiedLinkError exception loading native library: njni11
java.lang.UnsatisfiedLinkError: no njni11 in java.library.path
Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: db02
Enter Port of Database server: 1522
Enter SID of Database server: erptest
Enter Database Service Name: erptest
Enter the value for Display Variable:
Invalid input, reenter the value for Display Variable: 0.0
The context file has been created at:
/u01/oracle/11.2.0.4/erphome1/appsutil/erptest_db02.xml

On Application Tier for adcfgclone.sh below are the options,
Target System Base Directory : /u01/erptest
Target System Instance Home Directory [/u01/erptest] :
Do you want to preserve the Display [erplx0:0.0] (y/n)  : n
Target System Display [erplx0:0.0] :
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Other Services [disabled] :
Do you want the target system to have the same port values as the source system (y/n) [y] ? : n

[root@erplx0 etc]# cd /u01/erpt/
[root@erplx0 erptest]# mkdir oraInventory
chown -R appltst:oinstall oraInventory

as root user, create new file /etc/oraInst.loc with below content.
[root@erplx0 etc]# vi oraInst.loc
inventory_loc=/u01/erpt/oraInventory
inst_group=oinstall
[root@erplx0 ~]# usermod -g oinstall -G oinstall appltst
[root@erplx0 ~]# id appltst
uid=1001(appltst) gid=54322(oinstall) groups=54322(oinstall)
  • To clean the oracle_home entries from oraInventory run the below
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
  ./runInstaller -detachhome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1

SQL> SELECT ADOP_SESSION_ID,PREPARE_STATUS,APPLY_STATUS,FINALIZE_STATUS,CUTOVER_STATUS,CLEANUP_STATUS,ABORT_STATUS,STATUS,ABANDON_FLAG,NODE_NAME from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

ADOP_SESSION_ID P A F C C A S ABANDON_FLAG NODE_NAME
-----------------------------------------------------
              2 X Y N X Y X C              lxap1
              4 Y Y Y Y Y X C              lxap1
              5 Y Y Y Y Y X C              lxap1
              6 Y Y Y Y Y X C              lxap1
              7 Y Y Y Y Y X C              lxap1
              8 Y Y Y Y Y X C              lxap1
              9 Y Y Y Y Y X C              lxap1
             10 Y Y Y Y Y X C             lxap1
             10 Y Y Y Y Y X C             lxap2
             11 Y Y Y Y Y X C             lxap1
             11 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap2
             12 Y Y Y Y Y X C             lxap1
             12 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             irec
             13 Y Y Y Y Y X C             lxap1
             13 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             lxap2
             14 Y Y Y Y Y X C             irec
             14 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             lxap2
             15 Y Y Y Y Y X C             lxap1
             15 Y Y Y Y Y X C             irec
             16 Y Y Y Y Y X C             lxap1
             16 Y Y Y Y Y X C             lxap2
             16 Y Y Y Y Y X C             irec
             17 Y Y Y Y Y X C             lxap1
             17 Y Y Y Y Y X C             lxap2
             17 Y Y Y Y Y X C             irec
             18 Y Y Y Y Y X C             lxtst0
             19 Y Y Y Y Y X C             lxtst0
             20 Y Y Y Y Y X C             lxtst0
             21 Y Y Y Y Y X C             lxtst0
33 rows selected.

As can be seen above last online patching session (11) was done successfully till cleanup stage (marked as Y)

SQL> SELECT abbreviation,codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in ('txk','ad');
ABBREVIATION                   CODELEVEL
ad                             C.8
txk                            C.8
Above means we are on delta8 patch level, hence we can use the 'dualfs' option while cloning from Delta 7 onwards.

#cd /u01/erptest/fs1/EBSapps/comn/clone/bin
#perl adcfgclone.pl appsTier dualfs

POST CLONE STEPS
1. Change APPS and Weblogic passwords
2. update all context files ----> https to http in both FS1 & FS2
3. fnd_conc_clone.setup_clean;
4. run autoconfig on patch FS (.EBSapps.env patch)--->if patch edition auto config error then disable ebs_logon trigger and later enable it.
5. run autoconfig on run FS (.EBSapps.env run)


  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system
[applerp@erplx0 ~]$ more /u01/erptest/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml | grep https
<EndUserMonitoringURL oa_var="s_endUserMonitoringURL">https://erplx0.ad.councilmolqia.net:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif</EndUserMonitoringURL>
  <externURL oa_var="s_external_url">https://erplx0.ad.councilmolqia.net:8001</externURL>
    <login_page oa_var="s_login_page">https://erplx0.ad.councilmolqia.net:8001/OA_HTML/AppsLogin</login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">https</webentryurlprotocol>

MODIFY THE CONTEXT FILES OF FS1 AND FS2 AS BELOW

<EndUserMonitoringURL oa_var="s_endUserMonitoringURL">http://erplx0.ad.councilmolqia.net:8001/oracle_smp_chronos/oracle_smp_chronos_sdk.gif</EndUserMonitoringURL>
  <externURL oa_var="s_external_url">http://erplx0.ad.councilmolqia.net:8001</externURL>
    <login_page oa_var="s_login_page">http://erplx0.ad.councilmolqia.net:8001/OA_HTML/AppsLogin</login_page>
      <httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
        <webentryurlprotocol oa_var="s_webentryurlprotocol">http</webentryurlprotocol>

vi /u01/erptst/fs1/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
vi /u01/erptest/fs2/inst/apps/erptest_erplx0/appl/admin/erptest_erplx0.xml
:%s/https/http/g -- then change http to http for only this line---><httpslistenparameter oa_var="s_https_listen_parameter">4444</httpslistenparameter>
  • AFTER ABOVE CHANGE FND_CONC_CLONE.SETUP CLEAN AND THEN RUN THE AUTOCONFIG AS BELOW,
THEN RUN THE AUTO CONFIG ON BOTH 'PATCH' AND 'RUN' FILE SYSTEMS.
. /u01/erptst/EBSapps.env patch
cd $ADMIN_SCRIPTS_HOME
sh adautocfg.sh
. /u01/erptst/EBSapps.env run
cd $ADMIN_SCRIPTS_HOME
sh adautocfg.sh
---->IF THIS FAILS THEN SQL> ALTER TRIGGER EBS_LOGON DISABLE; THEN RUN AUTOCONFIG AGAIN and then enable the trigger and then run autoconfig from run file system.
SQL> select owner,trigger_name,trigger_type,status from dba_triggers where trigger_name='EBS_LOGON';
OWNER                          TRIGGER_NAME                   TRIGGER_TYPE STATUS
SYSTEM                         EBS_LOGON                      AFTER EVENT ENABLED

  • update all context files ----> https to http in both FS1 & FS2 and then run autoconfig on patch file system and then on run file system

  • CHANGE WEBLOGIC PASSWORD
1. Shut down all application tier services except the Admin Server.
On the primary node, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh -skipNM -skipAdmin
 On all secondary nodes, run the command:
 $ <ADMIN_SCRIPTS_HOME>/adstpall.sh
2. Change the Oracle WebLogic Server Administration User password by performing the following steps on the run file system of the primary node.
 Source the environment on the run file system.
 Run the commands appropriate for your platform:
 On UNIX, run the command:
 $ perl $FND_TOP/patch/115/bin/txkUpdateEBSDomain.pl -action=updateAdminPassword
The above command is available only if TXK DELTA 7 is applied.
If that is not the case, the password need to changed from Console and few steps need to be performed
3. Start all services on all nodes, using the command:
 $ <ADMIN_SCRIPTS_HOME>/adstrtal.sh
  • Do not do the below on cloned instance as it is done only on production. On clone instance it is not required to enable the workflow mailer after cloning for test environments. 
Setup this profile for mailer:
WF: Workflow Mailer Framework Web Agent:  http://erp.ad.councilmolqia.net:8011
Please change the value to appropriate server.
Also run the below script to setup the Workflow Admin role
SQL> update wf_resources
set text='FND_RESP|FND|COUNCILMOLQIA__WF_ADMIN|STANDARD'
where name = 'WF_ADMIN_ROLE';
SQL> COMMIT;

########################TROUBLESHOOTING 1##############
  • 12.2 patch edition auto config error
txkGenADOPWrapper.pl  INSTE8_APPLY 1
Solution:
To run autoconfig from patch file system you must disable trigger ebs_login prior to running autoconfig
1. alter trigger ebs_logon disable;
2. source patch file system and run autoconfig successfully.
3. alter trigger ebs_logon enable;
4. source patch file system and then connect as SYSTEM user using sqlplus

########################TROUBLESHOOTING 2##############
ERROR:
Enter the APPS password :
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object
chmod: cannot access /u01/erptest/fs1/EBSapps/comn/clone/bin/../FMW/tempinfoApps.txt: No such file or directory
SOLUTION:
In /etc/fstab file make xfs file system 32bit. (REFERENCE: https://access.redhat.com/solutions/2221761)
/dev/oracle_vg1/u01_v                           /u01                    xfs     rw,inode32      0 0
AND
check for jre and jdk 32 bit folders, ./java -version

########################TROUBLESHOOTING 3##############
ERROR: while running ./runInstaller -deinstall option we are getting below error,
The Java RunTime Environment was not found at /tmp/OraInstall/jdk/jre/bin/java
SOLUTION:
oraparam.ini file under the software staged directory path /software/FR11121/Disk1/install/linux64 had an entry JRE_SCRATCH_LOC
 It was set to
JRE_LOC=../../jdk/jre
UPDATE oraparam.ini to use another jre or fix the corrupted jre files in "/u01/erpt/fs1/EBSapps/10.1.2/jdk" folder.

########################TROUBLESHOOTING 4##############
After installing oracle required patches run the below command as root user,
ldconfig -v

check the below
[root@erplx0 lib]# ls -ltr libdb*
-rwxr-xr-x 1 root root 1691492 Sep 11  2014 libdb_cxx-4.7.so
-rwxr-xr-x 1 root root 1575444 Sep 11  2014 libdb-4.7.so
-rwxrwxrwx 1 root root    5825 Aug  3  2016 libdb.so.2
lrwxrwxrwx 1 root root      12 Apr  3 14:33 libdb-4.3.so -> libdb-4.7.so

if link is not there then create it as below,
# ln -s libdb-4.7.so libdb-4.3.so
check if this file exists or not /usr/lib/libdb.so.2

########################TROUBLESHOOTING 5##############
Error that FMW_Home already exists the,
cd /u01/erptest/fs1/FMW_Home/oracle_common/oui/bin
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
>>Then remove all folders except fs1 and re-run adcfgclone.sh script.
[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/webtier
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erpt/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/oracle_common
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.

[appltst@erplx0 bin]$ ./runInstaller -detachHome ORACLE_HOME=/u01/erptest/fs1/FMW_Home/Oracle_EBS-app1
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16383 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/erptest/oraInventory
'DetachHome' was successful.
For any queries on this article please send me email to samiappsdba@gmail.com