Oracle Autonomous Health Framework AHF

Autonomous Health Framework (AHF) provides a number of diagnostic tools in a single bundle, making it easy to gather diagnostic information about the Oracle database and clusterware, which in turn helps with problem resolution when dealing with Oracle Support. AHF) combines the existing Trace File Analyzer (TFA) and ORAchk/EXAchk, along with a bunch of other stuff, into a single download and setup. For any scenario where you would have installed one of these individual tools, you should now install AHF.

Autonomous Health Framework (AHF) will be shipped with new releases of the database, but you should always download the latest version. Single installer for Trace File Analyzer (TFA) and ORAchk/EXAchk

Autonomous Health Framework (AHF) - Including TFA and ORAchk/EXAChk (Doc ID 2550798.1)

If possible you should install as root. This will give you the richest capabilities.. Unzip the software and run the ahf_setup command. Answer the questions when prompted. 

Download the appropriate installer from above, copy to a target machine and unzip

Run the TFA & ORAchk/EXAchk install command 

./ahf_setup [-ahf_loc install_dir] [-data_dir data_dir]

# cd /u01/software
# unzip -oq AHF-LINUX_v20.4.0.zip
#
# ./ahf_setup

AHF Installation Log : /tmp/ahf_install_7860_2020_12_31-11_11_11.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 204000 Build Date: 202012317860

Default AHF Location : /opt/oracle.ahf

Do you want to change AHF Location (/opt/oracle.ahf) ? Y|[N] :

AHF Location : /opt/oracle.ahf

AHF Data Directory stores diagnostic collections and metadata.
AHF Data Directory requires at least 5GB (Recommended 10GB) of free space.

Choose Data Directory from below options :

1. /opt/oracle.ahf [Free Space : 1007860 MB]
2. Enter a different Location

Choose Option [1 - 2] : 1

AHF Data Directory : /opt/oracle.ahf/data

Do you want to add AHF Notification Email IDs ? [Y]|N :

Enter Email IDs separated by space : sami.malik@oracle.com

Extracting AHF to /opt/oracle.ahf

Configuring TFA Services

Discovering Nodes and Oracle Resources

Starting TFA Services
Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.

.------------------------------------------------------------------------------.
| Host      | Status of TFA | PID  | Port  | Version    | Build ID             |
+-----------+---------------+------+-------+------------+----------------------+
| localhost | RUNNING       | 7860 | 24035 | 20.4.0.0.0 | 204000202012311007860|
'-----------+---------------+------+-------+------------+----------------------'

Running TFA Inventory...

Adding default users to TFA Access list...

.-------------------------------------------------------.
|              Summary of AHF Configuration             |
+-----------------+-------------------------------------+
| Parameter       | Value                               |
+-----------------+-------------------------------------+
| AHF Location    | /opt/oracle.ahf                     |
| TFA Location    | /opt/oracle.ahf/tfa                 |
| Orachk Location | /opt/oracle.ahf/orachk              |
| Data Directory  | /opt/oracle.ahf/data                |
| Repository      | /opt/oracle.ahf/data/repository     |
| Diag Directory  | /opt/oracle.ahf/data/localhost/diag |
'-----------------+-------------------------------------'

AHF binaries are available in /opt/oracle.ahf/bin

AHF is successfully installed

Moving /tmp/ahf_install_7860_2020_12_31-11_11_11.log to /opt/oracle.ahf/data/localhost/diag/ahf/

As shown above, the tfactl and orachk commands are available from the following directory.

/opt/oracle.ahf/bin

[root@ed-ol-rac-lin01::~]$ exachk -v

EXACHK  VERSION: 20.4.0_20201214

As part of installation 'oracle-tfa.service' will be enabled and started.

# systemctl status oracle-tfa.service

● oracle-tfa.service - Oracle Trace File Analyzer

   Loaded: loaded (/etc/systemd/system/oracle-tfa.service; enabled; vendor preset: disabled)

   Active: active (running) since Wed 2020-12-22 11:11:11 NZDT; 1 weeks 2 days ago

 Main PID: 1007860 (init.tfa)

   CGroup: /system.slice/oracle-tfa.service

           ├─ 1007860 /bin/sh /etc/init.d/init.tfa run >/dev/null 2>&1 </dev/null

           ├─ 12345 /opt/oracle.ahf/jre/bin/java -server -Xms256m -Xmx512m -Djava.awt.headless=true -Ddisable.checkForUpdate=true -XX:HeapDump...

           └─284747 /bin/sleep 30

Dec 31 09:39:54 ed-ol-raclin01 su[195700]: (to oracle) root on none

Dec 31 09:39:59 ed-ol-raclin01 su[196070]: (to root) root on none

Dec 31 09:39:59 ed-ol-raclin01 su[196075]: (to root) root on none

Dec 31 09:39:59 ed-ol-raclin01 su[196079]: (to oracle) root on none

Dec 31 09:39:59 ed-ol-raclin01 su[196086]: (to oracle) root on none

Dec 31 09:39:59 ed-ol-raclin01 su[196091]: (to oracle) root on none

Dec 31 09:40:00 ed-ol-raclin01 su[196265]: (to oracle) root on none

Dec 31 09:40:00 ed-ol-raclin01 su[196268]: (to oracle) root on none

Dec 31 09:40:00 ed-ol-raclin01 su[196271]: (to oracle) root on none

Dec 31 09:40:00 ed-ol-raclin01 su[196276]: (to oracle) root on none

Warning: oracle-tfa.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Use 'systemctl daemon-reload' command to reload units

#systemctl daemon-reload

• tfactl and orachk commands are available from the following directory.

$ORACLE_HOME/ahf/oracle.ahf/bin

With the installation complete we can use the orachk command to check the services running on the server. This can be done on-demand, or run as a background task, which will send an email to the notification email addresses.

# as root

cd /opt/oracle.ahf/bin

# as non-root

cd $ORACLE_HOME/ahf/oracle.ahf/bin

# on-demand

./orachk

# background

./orachk -autostart

Once this is complete the oracle-orachkscheduler.service service will be enabled and running.

# systemctl status oracle-orachkscheduler.service

Run a TFA Collection

With the installation complete we can use the tfactl command to perform a number of collections, including TFA Service Request Data Collections (SRDC). There are a large number of SRDC collection types, with each gathering different information, as described here.

Here are a few examples.

# root
cd /opt/oracle.ahf/bin
# non-root
cd $ORACLE_HOME/ahf/oracle.ahf/bin


# Gather information about errors. You are prompted to select a specific incident.
./tfactl diagcollect -srdc ORA-00600
./tfactl diagcollect -srdc ORA-07445

# Collect data for all components for a specific time period.
./tfactl diagcollect -from "2020-12-15 11:11:11" -to "2020-12-31 11:11:11"

# Collect data for all components for the last 12 hours.
./tfactl diagcollect

Each TFA collection produces a single zip file that can be uploaded to My Oracle Support (MOS), as described below.

Upload Files to My Oracle Support (MOS)

You can manually upload a zip file produced by TFA or ORAchk by attaching it to your SR through the MOS website in the normal way. An alternative is to get TFA to upload it and attach it to your Service Request (SR). To do this you must provide your MOS credentials. This can be done as part of the command line, or they can be stored in a secure wallet by running the following command as the "root" user.

# root
cd /opt/oracle.ahf/bin
# non-root
cd $ORACLE_HOME/ahf/oracle.ahf/bin

# ./tfactl setupmos
Enter User Id: sami.malik@oracle.com
Enter Password:
SUCCESS - CERTIMPORT - Successfully imported certificate
#

The result of a TFA collection can be uploaded directly to MOS by including the "-sr" option in the command to specify the target SR number.

# MOS credentials supplied by wallet.
./tfactl diagcollect -srdc ORA-00600 -sr 1-12345678910

# MOS credentials supplied on command line. You are prompted for the password.
./tfactl diagcollect -srdc ORA-00600 -sr 1-12345678910 -user sami.malik@oracle.com

You can upload one or more files (both TFA generated and other files) as a separate action using the following commands.

# MOS credentials supplied by wallet.
./tfactl upload -sr 1-12345678910 -wallet file1.zip file2.zip file3.zip

# MOS credentials supplied on command line. You are prompted for the password.
./tfactl upload -sr 1-12345678910 -user sami.malik@oracle.com file1.zip file2.zip file3.zip

The Autonomous Health Framework (AHF) installer includes:

• TFA is the Primary diagnostic collection tool, with Database Support Tools Bundle

Memory can be limited at either the system level using:

tfactl setresourcelimit -resource kmem

or combined system and swap memory, using:

tfactl setresourcelimit -resource swmem

For example:

To limit the memory usage to only 1GB of system memory run:

tfactl setresourcelimit -resource kmem -value 1024

Alternatively to limit the combined total of system memory and the swap memory to 2GB use:

tfactl setresourcelimit -resource swmem -value 2048

• ORAchk / EXAchk

Oracle Stack Compliance Checks

• procwatcher

Automate & capture database performance diagnostics & session level hangs

See 459694.1 for more details.

• events

Reports warnings and errors seen in the logs

• managelogs

Shows disk space usage and purges ADR log and trace files

• alertsummary

Provides summary of events for one or more database or ASM alert files from all nodes

• ls / dir

Lists all files TFA knows about for a given file name pattern across all nodes

• summary

High level summary of the configuration

• vi / notepad

Open alert or trace files for viewing a given database and file name pattern in the vi editor

• tail

Run a tail on an alert or trace files for a given database and file name pattern

• param

Show all database and OS parameters that match a specified pattern

• oswatcher

Collect and archive OS metrics, useful for instance / node evictions & performance Issues.

See 301137.1 for more details.

• oratop

Near real-time database monitoring

See 1500864.1 for more details.

• pstack

Generate process stack for specified processes across all nodes

• grep / findstr

Search alert or trace files with a given database and file name pattern, for a search string

• dbglevel

Set and unset multiple CRS trace levels with one command

• history

Show the shell history for the tfactl shell

• changes

Report any noted changes in the system setup over a given time period. This includes database a parameters, OS parameters, patches applied etc

• calog

Reports major events from the Cluster Event log

• ps / tasklist

Finds processes

• triage

Summarize oswatcher/exawatcher data

References: 

Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)

Autonomous Health Framework (AHF) - Including TFA Trace File Analyser and ORAchk/EXAChk (Doc ID 2550798.1)