Friday, October 22, 2021

Sync AD tables when ADOP fails in EBS 12.2

Here we will see how to Synchronize below AD tables when adop fails on EBS 12.2 on premises or on Cloud,
FND_NODES
ADOP_VALID_NODES
FND_OAM_CONTEXT_FILES


CASE 1 ERROR:
# adop phase=prepare
*******FATAL ERROR*******
PROGRAM : (/appl/ad/12.0.0/bin/adzdoptl.pl)
TIME : Fri 10 Oct 03:00:45 2021
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ]
ERRORMSG: adop is not able to detect any valid application tier nodes in ADOP_VALID_NODES table.
Ensure autoconfig is run on all nodes.
[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 255 (Fail)

CASE 1 CAUSE: 
# adop phase=prepare
In this case, The fnd_nodes table has NULL for domain entry of this valid server.
Example Data Supporting the Conclusion

[table.applsys.fnd_nodes]
NODE     PLAT    D C A F W    NODE_NAME    SERVER_ADDRESS     DOMAIN    WEBHOST                              VIRTUAL_IP        S
12,121     226    N Y Y Y Y    NEW122APP1   ###.###.###.103     NULL        new122app1.<host.domain>       NULL               Y



CASE 2 ERROR:
# adop phase=apply / actualize_all
[APPLY PHASE]
*******FATAL ERROR*******
AutoConfig could not successfully execute the following scripts:
Directory: <EBS_BASE>/fs#/FMW_Home/webtier/perl/bin/perl -I <EBS_BASE>/fs#/FMW_Home/webtier/perl/lib/5.10.0 -I <EBS_BASE>/fs#/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I <EBS_BASE>/fs#/EBSapps/appl/au/12.0.0/perl -I <EBS_BASE>/fs#/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /<EBS_BASE>/fs#/inst/apps/<SID_hostname>/admin/install
txkGenADOPWrapper.pl INSTE8_APPLY 1
SQL> SQL> Connected.
SQL> SELECT ad_zd_adop.get_node_type('<hostname>') FROM DUAL
*
ERROR at line 1:
ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.AD_ZD_ADOP", line 3049


CASE 2 CAUSE: 
# adop phase=apply / actualize_all
[table.applsys.fnd_oam_context_files]
NAME S_CONTEXTFILE S_CONTEXTTYPE S_CONTEXTNAME Version Number SERIAL APPL_TOP_NAME HOST SHARED_FILE_SYSTEM EDITION
<SID_hostname> ./<SID_hostname>.xml APPL_TOP Context <SID_hostname> 120.401.12020000.85 6999 AT_<hostname> <hostname> false run
<SID_hostname> ./<SID_hostname>.xml APPL_TOP Context <SID_hostname> 120.401.12020000.85 6999 AT_<hostname> <hostname> false run
<SID_hostname> ./<SID_hostname>.xml APPL_TOP Context <SID_hostname> 120.401.12020000.85 7001 AT_<hostname> <hostname> false patch
<SID_hostname> ./<SID_hostname>.xml Database Context <SID_hostname> 120.52.12020000.11 0 NULL <hostname> NULL database 


SOLUTION:
Test the following steps in a cloned instance first, and then migrate accordingly once the desired result is confirmed:

Note(s): 1. Due to the method required for "cleaning out" / "re-synchronizing" the following tables, it is EXPECTED / REQUIRED that the Applications have been shutdown. The only thing running should be the Database Tier.

              2. A full backup should be taken before any testing begins.

              3. The following query can be used to find out one's AD/TXK levels.

               SQL>col release_name format a14

        col abbreviation format a14
        col codelevel format a10

         select f.release_name, t.abbreviation, t.codelevel from apps.fnd_product_groups f,ad_trackable_entities t where abbreviation in ('txk','ad');


1. Backup the fnd_oam_context_files, fnd_nodes, and adop_valid_nodes tables, and if on AD/TXK 8 or higher; ad_nodes_config_status in the EBS env nodes:

      Linux#> sqlplus applsys/<pwd>

      SQLPlus#> create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;

      SQLPlus#> create table fnd_nodes_bk as select * from fnd_nodes;

        SQLPlus#> create table adop_valid_nodes_bk as select * from adop_valid_nodes;

      If on AD/TXK 8 or higher:

      SQLPlus#> create table ad_nodes_config_status_bk as select * from ad_nodes_config_status;

2. Truncate the following tables (Continue from step 1):

      SQLPlus#> truncate table fnd_oam_context_files;

      SQLPlus#> truncate table fnd_nodes;

      SQLPlus#> truncate table adop_valid_nodes;

      If on AD/TXK 8 or higher:

      SQLPlus#> truncate table ad_nodes_config_status;

 3.  Run AutoConfig on the DB tier:

Source the <RDBMS_ORACLE_HOME> home.

Linux#> cd <RDBMS_ORACLE_HOME>/appsutil/scripts/<SID>_<HOSTNAME>/

Linux#> ./adautocfg.sh

              ... Confirm Autoconfig completes successfully.

              ... If RAC, Repeat step 3 on all RAC nodes.

4.  Run Autoconfig on the run file system.

Linux#> source <EBS_BASE>/EBSapps.env run

Linux#> cd $ADMIN_SCRIPTS_HOME

Linux#> ./adautocfg.sh

      ... Confirm Autoconfig completes successfully.

      ... If more than one EBS node, repeat step 4 on all EBS nodes.

5.  Run Autoconfig on the patch file system

Before running Autoconfig on the patch file system the ebs_login trigger MUST be disabled
... After the successful completion of Autoconfig the ebs_login trigger MUST be re-enabled.


a. Disable the ebs_login trigger using the following SQL.

Linux#> sqlplus system/<pwd>

SQLPlus#> alter trigger ebs_logon disable;

 

b. At this time Run autoconfig with the patch env sourced .       

Linux#> source <EBS_BASE>/EBSapps.env patch

Linux#> cd $ADMIN_SCRIPTS_HOME

Linux#> ./adautocfg.sh

... Confirm Autoconfig completes successfully

... If more than one EBS node, repeat step 5.b on all EBS nodes.


c. Enable the ebs_login trigger using the following SQL.

Linux#> sqlplus system/<pwd>

SQLPlus#> alter trigger ebs_logon enable;

 

 6.  After Autoconfig has been run successfully on all nodes, run the following two (2) queries in order to verify the tables have been correctly populated:

SQL>    set pagesize 5
        set linesize 132
        col node_name format a15
        col server_id format a8
        col server_address format a15
        col platform_code format a4
        col webhost format a12
        col domain format a20
        col virtual_ip format a12

        select node_id, platform_code, support_db D, support_cp C, support_admin A, support_forms F, support_web W, node_name, server_id,
               server_address, domain, webhost, virtual_ip, status
          from fnd_nodes order by node_id;
 

SQL>    set pagesize 5
        set linesize 132
        col NAME format A20
        col VERSION format A12
        col PATH format A110
        col STATUS format A10

        select NAME,VERSION,PATH, STATUS from FND_OAM_CONTEXT_FILES;