All pods have had the following tracing pre-configured.
|MR PIM|| procmon <cust inst> PGxx pimx
trace mr* /on
trace mr_function_call /off
trace mr_heartbeat_messages /off
|OPC|| opctest /cust <cust_inst> /node PGxx
debug /agent /closedcalls /ctimsg /routing /cstacer /inrcmsg /tpmsg /calls /realtime /task
RouteRequests, Translation Route, NetworkVRUTracing, CallTypeRealtime, Agent Changes
|CMB||Verbose Level 10|
Unified System CLI
Starting in 8.x, we can avoid the registry and use the CLI to set tracing for UCCE components with a simple "debug level 1 ...".
These same trace levels can be set from the web-based Portico Diagnostic Framework.
Since UCCE does not use OAMP like CVP, there is some minor configuration required. Reference the UCCE Serviceability Best Practices Guide for details on how to configure your devices.csv file.
EIM/WIM Trace Levels
Starting with EIM/WIM 4.3(2), there are 8 different levels of logging.
How the trace levels correlate to the log files
Each line of EIM/WIM process logs tell you which level it comes from. Below, we see messages from 3 different levels: WARN, ERROR, and INFO. Higher levels will include messages from lower levels.
2011-05-01 19:02:23.758 GMT-0400 <@> WARN <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId: <@> com.egain.platform.framework.dataaccess.DataAccess <@> executeSingleValueResultQuery() <@> Throwing EmptyResultSetException : No result returned by the query : user::get_user_state <@> 2011-05-01 19:02:23.774 GMT-0400 <@> ERROR <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId: <@> com.egain.platform.user.User <@> getUserState() <@> I18N_EGPL_USER-USER_NOT_FOUND -1 <@> 2011-05-01 19:02:23.774 GMT-0400 <@> INFO <@> [32:Thread-6] <@> ProcessId:1740 <@> PID:1 <@> UID:12 <@> HttpSessionId: <@> com.cisco.ipcc.mr.request.NewTaskProcessor <@> processNewTaskMsg <@> Error while selecting sticky Agent. User not logged-in. Activity: 1139.
Trace Level Recommendations
The default trace level for EIM/WIM components is ERROR. In most cases, ERROR level logs are not sufficient to find root cause.
DEBUG gives a significant bit more detail (i.e. SQL queries being run) that can be useful for low-level investigations, but is in many ways redundant and not absolutely necessary for troubleshooting. Starting with 4.3(2), DEBUG level logging will automatically return to ERROR level after a user-specified period of time. INFO level logging will not do this.
This entire class will use INFO level tracing for EIM/WIM components. It is a safe trace level for production usage as long as Virtual Memory is set correctly per the 4.3 Deployments Guide: Virtual memory setting should be set to 1.5 times the physical memory. To ensure that adequate space is available during run time, distribute the virtual memory across disk volumes. If multiple processes are set to INFO level without adequate virtual memory allocated, the system will start to crawl and eventually halt altogether.
Disk Space Concerns
Note that log file size is constant at 5MB. It is not possible to change this from the System Console > Logger folder in the UI.
If the logging level of a process is increased past ERROR, a total of 100x 5MB log files will be written for the process. EIM will automatically overwrite the oldest files once this limit is reached.
- The size of each file is set in C:\CIM\eService\config\egpl_log4j.xml.
- This also tells processes what to name their logfiles, and where to save them (File Server).
<appender class="com.egain.platform.util.logging.appender.RollingFileAppender" name="eimServices:rules-cache-process"> <param name="Threshold" value="info#com.egain.platform.util.logging.Level"/> <param name="File" value="eg_log_eimServices_rules-cache-process.log"/> <param name="MaxBackupIndex" value="-1"/> <param name="MaxFileSize" value="5MB"/>
The egpl_log4j.xml file can sometimes become corrupted, causing process startup and logging to fail. If only one or two java/javaw processes are starting, this file is likely corrupted.
- A clean version of this file is included in the same directory as "egpl_log4j.pristine"
- In the event of file corruption, the pristine file can be copied and overwrite the current egpl_log4j.xml.
- Start Services with the fresh file, and EIM/WIM will "finish" the file's contents much like it does with a JBoss directory when you first start a newly-created Application Server.
In ICM/UCCE logs, a common first step is to grep for strings such as "Exception" or "Error". With EIM/WIM logging, this is not useful due to the sheer amount of results returned.
- All ERROR-level log lines will be returned, even though they may not actually be errors.
- There are a large number of exceptions occurring throughout the system even in steady-state behavior. Many of these are benign or unrelated to whatever issue you may be troubleshooting. The best option is to start with something specific to your issue - an activity ID, case ID, user ID, queue ID, etc.
- Note that Activity IDs will belong to a Case ID. This will typically be a many-to-one relationship.
Setting Trace Levels
Tracing is set from the System partition: http://<webserver>/system
View a video example of setting trace levels.