My Blog

TFA – Trace File Analyzer

by admin on 8th March 2015 No comments

One of the highlights of the recent UKOUG RAC SIG for me was a talk from Jamie Wallis from Oracle on the TFA. It silently gets installed during an 11.2.0.4 upgrade, but was available long before that as a standalone download. It is an answer to those all too common requests from Oracle Support to gather a selection of logs from all nodes in a cluster, but is also much more than that.

A few highlights before diving down into a little more detail –

– Collect all relevant logs from all nodes of a cluster, and collate on a single node ready for sending to Oracle
– Cluster aware (many commands including installation run across cluster)
– On Exadata will recognise storage cells and collect logs from them too
– Installed by default on ODA
– Can be configured to perform automatic collection upon incident
– Self manages repository (default size 10GB, won’t run if filesystem has less than 1GB free)
– Run as root, oracle or any other user configured within tool to have access
– Part of the support tools roadmap. Other support tools being integrated into TFA
– Can be patched manually but also patched by PSUs
– “Zero configuration”. Many options available, but by default will run discovery for all relevant log locations

Now a little more detail.

Command line utility

Very similar in syntax design to other Oracle tools. Get help with “help”. Get help with certain commands with “help “. This feels very familiar.

Status / Start / Stop

TFA runs a lightweight deamon process which is automatically started by init. Manual control is also possible. Note that starting and stopping happens on the local node only.

Configuration

Lots of configuration possible.

Directories

Take a look at some of the directories picked up by default on my cluster. As you can see, the changes of Oracle Support having to assign an SR back to you in order to provide a new log file are extremely remote!

/etc/oracle
/u01/app/11.2.0.4/grid/OPatch/crs/log
/u01/app/11.2.0.4/grid/cfgtoollogs
/u01/app/11.2.0.4/grid/cfgtoollogs/cfgfw
/u01/app/11.2.0.4/grid/cfgtoollogs/crsconfig
/u01/app/11.2.0.4/grid/cfgtoollogs/opatch
/u01/app/11.2.0.4/grid/cfgtoollogs/oui
/u01/app/11.2.0.4/grid/crs/log
/u01/app/11.2.0.4/grid/css/log
/u01/app/11.2.0.4/grid/cv/log
/u01/app/11.2.0.4/grid/evm/admin/log
/u01/app/11.2.0.4/grid/evm/admin/logger
/u01/app/11.2.0.4/grid/evm/log
/u01/app/11.2.0.4/grid/install
/u01/app/11.2.0.4/grid/log/diag/clients
/u01/app/11.2.0.4/grid/log/diag/clients/user_root/host_313715646_80/trace
/u01/app/11.2.0.4/grid/log/diag/tnslsnr/uf1/listener_scan1/trace
/u01/app/11.2.0.4/grid/log/diag/tnslsnr/uf1/listener_scan2/trace
/u01/app/11.2.0.4/grid/log/diag/tnslsnr/uf1/listener_scan3/trace
/u01/app/11.2.0.4/grid/log/uf1
/u01/app/11.2.0.4/grid/log/uf1/acfs
/u01/app/11.2.0.4/grid/log/uf1/acfslog
/u01/app/11.2.0.4/grid/log/uf1/acfsrepl
/u01/app/11.2.0.4/grid/log/uf1/acfsreplroot
/u01/app/11.2.0.4/grid/log/uf1/acfssec
/u01/app/11.2.0.4/grid/log/uf1/admin
/u01/app/11.2.0.4/grid/log/uf1/agent/crsd/oraagent_oracle
/u01/app/11.2.0.4/grid/log/uf1/agent/crsd/orarootagent_root
/u01/app/11.2.0.4/grid/log/uf1/agent/crsd/scriptagent_oracle
/u01/app/11.2.0.4/grid/log/uf1/agent/ohasd/oraagent_oracle
/u01/app/11.2.0.4/grid/log/uf1/agent/ohasd/oracssdagent_root
/u01/app/11.2.0.4/grid/log/uf1/agent/ohasd/oracssdmonitor_root
/u01/app/11.2.0.4/grid/log/uf1/agent/ohasd/orarootagent_root
/u01/app/11.2.0.4/grid/log/uf1/client
/u01/app/11.2.0.4/grid/log/uf1/crflogd
/u01/app/11.2.0.4/grid/log/uf1/crfmond
/u01/app/11.2.0.4/grid/log/uf1/crsd
/u01/app/11.2.0.4/grid/log/uf1/cssd
/u01/app/11.2.0.4/grid/log/uf1/ctssd
/u01/app/11.2.0.4/grid/log/uf1/cvu
/u01/app/11.2.0.4/grid/log/uf1/cvu/cvulog
/u01/app/11.2.0.4/grid/log/uf1/cvu/cvutrc
/u01/app/11.2.0.4/grid/log/uf1/diskmon
/u01/app/11.2.0.4/grid/log/uf1/evmd
/u01/app/11.2.0.4/grid/log/uf1/gipcd
/u01/app/11.2.0.4/grid/log/uf1/gnsd
/u01/app/11.2.0.4/grid/log/uf1/gpnpd
/u01/app/11.2.0.4/grid/log/uf1/mdnsd
/u01/app/11.2.0.4/grid/log/uf1/ohasd
/u01/app/11.2.0.4/grid/log/uf1/racg
/u01/app/11.2.0.4/grid/log/uf1/srvm
/u01/app/11.2.0.4/grid/network/log
/u01/app/11.2.0.4/grid/oc4j/j2ee/home/log
/u01/app/11.2.0.4/grid/opmn/logs
/u01/app/11.2.0.4/grid/racg/log
/u01/app/11.2.0.4/grid/rdbms/log
/u01/app/11.2.0.4/grid/scheduler/log
/u01/app/11.2.0.4/grid/srvm/log
/u01/app/oraInventory/ContentsXML
/u01/app/oraInventory/logs
/u01/app/oracle/cfgtoollogs
/u01/app/oracle/cfgtoollogs/asmca
/u01/app/oracle/cfgtoollogs/catbundle
/u01/app/oracle/cfgtoollogs/dbca/test
/u01/app/oracle/cfgtoollogs/emca
/u01/app/oracle/cfgtoollogs/netca
/u01/app/oracle/diag/asm/+asm/+ASM1/trace
/u01/app/oracle/diag/rdbms/test/test1/trace
/u01/app/oracle/diag/tnslsnr/uf1/listener/trace
/u01/app/oracle/oradiag_root/diag/clients/user_root/host_313715646_80/trace
/u01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs
/u01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/cfgfw
/u01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch
/u01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/oui
/u01/app/oracle/product/11.2.0.4/db_1/install
/var/log

Adding hosts

Added a host to your cluster? No problem.

Manual Collections

We can trigger a manual collection. As you can imagine, there are a lot of command line options for this but by default it’ll collect 4 hours of logs from all locations.

We actually a lot of feedback about what is going on while this collection is taking place. Click on the image for a closer look.

As you can see, it has collected everything to one node and even reminds us where the repository is located.

We can query the collections which have taken place –

Automatic Collections

TFA can also be configured to trigger a collection upon certain events (such as cluster events, database ORA-600’s and other configured errors).

Let’s switch it on across the cluster –

And then raise an ORA-600 in our ASM instance –

5 minutes later we’ll see the collection took place –

Different collections take place depending on the event, and these are clearly documented. However, in this case an ORA-600 on the ASM instance will trigger a collection for the ASM instance only. In this case it will also only include log data for 10 minutes either side of the event. It has produced a ZIP file 84KB in size, so you can see that there is plenty of space in my 10GB repository for many collections!


TFA on Exadata

TFA will automatically detect Exadata storage servers, and offers collection from those too!

We can see it is configured and working –

A really great tool in my opinion, and definitely something that will come in handy next time we have a Sev 1 running and Oracle Support give us their log file shopping list!

adminTFA – Trace File Analyzer

Related Posts

Take a look at these posts

Join the conversation