Tracing

From DocWiki

Revision as of 19:10, 15 November 2011 by Ginod (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Contents

Pod Tracing

All pods have had the following tracing pre-configured.

Component Tracelevel
EIM/WIM processes INFO
MR PIM procmon <cust inst> PGxx pimx

trace mr* /on

trace mr_function_call /off

trace mr_heartbeat_messages /off

CTI Server 0xF8
OPC opctest /cust <cust_inst> /node PGxx

debug /agent /closedcalls /ctimsg /routing /cstacer /inrcmsg /tpmsg /calls /realtime /task

EMSTraceMask 0x40

Router \icm\bin\rtrtrace.exe

RouteRequests, Translation Route, NetworkVRUTracing, CallTypeRealtime, Agent Changes

CMB Verbose Level 10

Unified System CLI

Starting in 8.x, we can avoid the registry and use the CLI to set tracing for UCCE components with a simple "debug level 1 ...".

Cli.png


These same trace levels can be set from the web-based Portico Diagnostic Framework.

Portico.png


Since UCCE does not use OAMP like CVP, there is some minor configuration required. Reference the UCCE Serviceability Best Practices Guide for details on how to configure your devices.csv file.

EIM/WIM Trace Levels

Starting with EIM/WIM 4.3(2), there are 8 different levels of logging.

  1. Fatal
  2. Error
  3. Warn
  4. Info
  5. Perf
  6. DBquery
  7. Debug
  8. Trace

How the trace levels correlate to the log files

Each line of EIM/WIM process logs tell you which level it comes from. Below, we see messages from 3 different levels: WARN, ERROR, and INFO. Higher levels will include messages from lower levels.

2011-05-01 19:02:23.758 GMT-0400 <@> WARN <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId:  <@> com.egain.platform.framework.dataaccess.DataAccess <@> executeSingleValueResultQuery() <@> Throwing EmptyResultSetException : No result returned by the query : user::get_user_state <@> 
2011-05-01 19:02:23.774 GMT-0400 <@> ERROR <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId:  <@> com.egain.platform.user.User <@> getUserState() <@> I18N_EGPL_USER-USER_NOT_FOUND   -1 <@> 
2011-05-01 19:02:23.774 GMT-0400 <@> INFO <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId:  <@> com.cisco.ipcc.mr.request.NewTaskProcessor <@> processNewTaskMsg <@> Error while selecting sticky Agent. User not logged-in. Activity: 1139.

Trace Level Recommendations

The default trace level for EIM/WIM components is ERROR. In most cases, ERROR level logs are not sufficient to find root cause.

DEBUG gives a significant bit more detail (i.e. SQL queries being run) that can be useful for low-level investigations, but is in many ways redundant and not absolutely necessary for troubleshooting. Starting with 4.3(2), DEBUG level logging will automatically return to ERROR level after a user-specified period of time. INFO level logging will not do this.

This entire class will use INFO level tracing for EIM/WIM components. It is a safe trace level for production usage as long as Virtual Memory is set correctly per the 4.3 Deployments Guide: Virtual memory setting should be set to 1.5 times the physical memory. To ensure that adequate space is available during run time, distribute the virtual memory across disk volumes. If multiple processes are set to INFO level without adequate virtual memory allocated, the system will start to crawl and eventually halt altogether.

Disk Space Concerns

Note that log file size is constant at 5MB. It is not possible to change this from the System Console > Logger folder in the UI.

MaxFileSize.png


If the logging level of a process is increased past ERROR, a total of 100x 5MB log files will be written for the process. EIM will automatically overwrite the oldest files once this limit is reached.

  • The size of each file is set in C:\CIM\eService\config\egpl_log4j.xml.
  • This also tells processes what to name their logfiles, and where to save them (File Server).
<appender class="com.egain.platform.util.logging.appender.RollingFileAppender" name="eimServices:rules-cache-process">
    <param name="Threshold" value="info#com.egain.platform.util.logging.Level"/>
    <param name="File" value="eg_log_eimServices_rules-cache-process.log"/>
    <param name="MaxBackupIndex" value="-1"/>
    <param name="MaxFileSize" value="5MB"/>

egpl_log4j.xml Corruption

The egpl_log4j.xml file can sometimes become corrupted, causing process startup and logging to fail. If only one or two java/javaw processes are starting, this file is likely corrupted.

  • A clean version of this file is included in the same directory as "egpl_log4j.pristine"
  • In the event of file corruption, the pristine file can be copied and overwrite the current egpl_log4j.xml.
  • Start Services with the fresh file, and EIM/WIM will "finish" the file's contents much like it does with a JBoss directory when you first start a newly-created Application Server.

Diving in

In ICM/UCCE logs, a common first step is to grep for strings such as "Exception" or "Error". With EIM/WIM logging, this is not useful due to the sheer amount of results returned.

  • All ERROR-level log lines will be returned, even though they may not actually be errors.
  • There are a large number of exceptions occurring throughout the system even in steady-state behavior. Many of these are benign or unrelated to whatever issue you may be troubleshooting. The best option is to start with something specific to your issue - an activity ID, case ID, user ID, queue ID, etc.
  • Note that Activity IDs will belong to a Case ID. This will typically be a many-to-one relationship.

Setting Trace Levels

Tracing is set from the System partition: http://<webserver>/system

View a video example of setting trace levels.

Rating: 0.0/5 (0 votes cast)

Personal tools