Additional troubleshooting information for Unified Expert Advisor 7.6.1
From DocWiki
Back: Troubleshooting Cisco Unified Expert Advisor
View/Add tips for Release 7.6(1): Category:Unified Expert Advisor, Release 7.6(1)
Contents
|
Problem Solving Process
The following steps provide some guidelines to assist in isolating a particular issue.
- Analyze the problem and create a clear problem statement. Define symptoms and potential causes.
- Gather the facts that you need to help isolate possible causes.
- Consider the Guidelines to Assist in Isolating Problems (below) based on the facts that you gathered.
- Create an action plan. Begin with the most likely problem and devise a plan in which you manipulate only one variable at a time.
- Implement the action plan, performing each step while testing to see whether the symptom disappears.
- Analyze the results to determine whether the problem has been resolved. If it has, the process is complete.
- If the problem has not been resolved, create an action plan based on the next most probable cause on your list, or contact the Cisco Technical Assistance Center (TAC), or your Cisco Partner.
 | Note: | Only change one variable at a time. If that does not resolve the issue, undo that change and move on to the next step of your plan. |
Guidelines to Assist in Isolating Problems
- Was anything recently added, removed, or modified?
- Is it a reproducible event?
- Does it occur at a particular time of day, or day of week?
- Have there been any changes made to the domain, network, or security policies?
Troubleshooting Checklist
Complete this checklist to assist in isolating the issue, or to provide information to your support partner or Cisco Technical Support.
- What is the version of Unified Expert Advisor that is currently running? Include any patch or upgrade information.
- Is this a new installation or an upgrade?
- If this is an upgrade, what version was previously installed?
- When did the problem occur?
- What are the observed symptoms, and the conditions under which these symptoms occur?
- Was anything changed or updated in hardware, software, or network components prior to the first occurrence of the observed symptoms?
- Describe the related call flow. Some examples include: Public Switched Telephone Network(PSTN) originated or IP Phone originated.
- Is the problem reproducible?
- What is the call transfer method used?
- Are you able to capture a screen shot of the error or failure? If Yes, save it to a file and attach to a case.
Network Topology
Complete this checklist to assist in isolating the issue, or to provide information to Cisco Technical Support.
- Has auto-negotiate been disabled on all PCs, routers, and switch ports?
Note: Duplex/speed mismatch between a device and its corresponding port on the switch is the single most common problem for network latency. - Is a network topology diagram available?
- Which type of IP Gateway is being used in this Unified Expert Advisor solution?
- On which server are the recorded media files located, and what is the path to those files?
- Collect and provide versions of IOS, applications, and Engineering Special (ES)/patch levels in the environment.
- Has auto-negotiate been disabled on all PCs, routers, and switch ports?
Obtaining Log Files
Log files are obtained through the Real Time Monitoring Tool (RTMT) Plugin. You obtain the plugin from the operations console.
Downloading and Installing the RTMT Plugin
The RTMT plugin can be downloaded from the operations console. Versions of the plugin are available for both Windows and Linux platforms. To download and install the RTMT plugin:
- Log in to the operations console for Unified Expert Advisor.
- Select Tools > RTMT Plugin Downloads.
- Select the platform (Windows or Linux) then click Download. The plugin is downloaded to your local computer. The file size of the executable is around 35 MB.
- Double-click the plugin executable on your local computer to install it. Details on the installation can be found in Chapter 2 of the Real Time Monitoring Tool Administration Guide for Cisco Unified Expert Advisor.
Obtaining Log Files Using the Real Time Monitoring Tool
Use the RTMT tool to browse, view, and download log files. This task explain show to obtain the logs for the runtime server(s).
- Start the RTMT tool. From Windows, select Start > Programs > Cisco > CallManager Serviceability > Real-time Monitoring Tool.
- Log in with the username and password you created for Unified Expert Advisor while installing Unified Expert Advisor.
- Select the Default Profile.
- In the left pane, labeled System, Click Trace & Log Central.
- In Trace & Log Central, double-click Remote Browse. A dialog appears.
Note: More details about Remote Browse can be found in the Real Time Monitoring Tool Administration Guide for Cisco Unified Expert Advisor. - In the dialog, select the radio button for Trace Files. Click Next.
- Select the checkbox next to Expert Advisor Runtime Service. You can check either the box for All Servers to view the logs for all runtime servers in this cluster, or you can check the box for individual servers. Click Next.
- You can optionally select System Services/Applications to obtain logs for additional system services, or just click Finish to obtain just the runtime server logs.
- A new pane appears in Trace and Log Central with a folder called Nodes in it. Double click the Nodes folder and drill down to the runtime servers. Double-click the runtime folder todisplay the list of log files in the right pane. There are several log files that appear in the right page, including: MMCA-runtime.[TIMESTAMP].log, MMCA-runtime.[TIMESTAMP].startup.log , MMCA.[TIMESTAMP].out, Error-runtime.[TIMESTAMP].startup.log, and Various zipped version of the above files.
- Double-click the latest version of MMCA-runtime.[TIMESTAMP].log to open it in the default viewer. This file contains a majority of the call-logging for the runtime server. You can optionally select the file and click Download at the bottom of the page.
Note: Zip files must be downloaded and unzipped locally with a zip program to be viewed, theycannot be viewed from within the RTMT tool.
Trace Definitions
This section lists the trace definitions for the infrastructure, runtime server, reporting server, ORM server, OAMP server, and common subsystems.
Infrastructure Trace Definitions
This section lists the infrastructure trace definitions for subsystems.
| Trace | Description |
|---|---|
| TRACE_HANDLED_EXCEPTION | Description of the exception and how it was handled |
| TRACE_JMX | JMX and management interface related traces |
| TRACE_JMS | JMS and message bus related traces |
| TRACE_HEARTBEAT | Related to heartbeats, heartbeat thread, or heartbeat send/received |
| TRACE_PARAM | For any parameters (not just method arguments) |
| TRACE_CALL | For traces related to a call / call processing |
| TRACE_MESSAGE | For general debug details of incoming/outgoing messages |
| TRACE_NOTIFICATION | Trace for notification API |
| TRACE_GENERAL_CFG | General traces for config API |
| TRACE_OOOQUEUE | Set this bit to enable OoOQueue tracing |
| TRACE_METHOD | When entering/exiting a method |
| TRACE_LOW_LEVEL | Bits and Bytes, etc |
Runtime Server Subsystem Trace Definitions
This section lists the Infrastructure Trace Definitions for Subsystems.
Agent State Monitoring (ASM)
| Trace | Description |
|---|---|
| TRACE_ASM | Details on Agent State Monitoring |
Reporting Adaptor (RA)
| Trace | Description |
|---|---|
| CONFIG | Trace configuration |
| MSG_PUBLISHING | Trace message publishing |
| RAI_MESSAGE | Trace MPI messages |
| CEI_MESSAGE | Trace CEI Messages |
| SYSTEM | Trace global activity |
| REI_MESSAGE | Trace REI messages |
| REPORTING_ADAPTER | Trace reporting adapter |
Intelligent Call Manager Gateway (ICMGW)
| Trace | Description |
|---|---|
| CONNECTION | ACMI connection management tracing, including OPEN_REQ/OPEN_CONF, and CLOSE_REQ/CLOSE_CONF |
| AGENT_STATE_UPDATE | Agent State Update Trace Mask |
| AUTO_CONFIG | Auto-config Trace Mask |
| ROUTING | Routing Trace Mask |
| ILLEGAL | Illegal Message Trace Mask |
| REPORTING | Reporting Trace Mask |
| OAMP | OAMP Trace Mask |
| HEARTBEAT | Reporting Trace Mask |
Work Assigner (WA)
| Trace | Description |
|---|---|
| TRACE_CONTACT | Trace everything associated with a contact |
| TRACE_MATCHING | Trace all match processing |
| TRACE_RESOURCE | Trace everything associated with a resource |
| TRACE_SENDMSGS | Trace all messages sent |
| TRACE_RECVMSGS | Trace all messages received |
| TRACE_DEBUG | Trace all debug events |
| TRACE_STATE | Trace all state processing |
| TRACE_QUEUING | Trace all queuing activity |
| TRACE_TRACKINGOBJECT | Trace everything associated with a tracking object |
| TRACE_CONFIG | Trace all configuration related activity |
| TRACE_TIMER | Trace all timer related events |
| TRACE_ADMIN | Trace all admin events and responses |
Resource Manager (RM)
| Trace | Description |
|---|---|
| ENDPOINT | For the Endpoint API tracing |
| MPI_METHOD_ENTRY/EXIT | When entering/exiting a method in the MPI channel |
| MPI_MESSAGE | For general debug details of incoming/outgoing messages for MPI channel |
| RDI_METHOD_ENTRY/EXIT | When entering/exiting a method in the RDI channel |
| ASSIGNMENT_QUEUE_ENTRY/EXIT | For the Assignment Queue API method entry/exit tracing |
| CMI_MESSAGE | For general debug details of incoming/outgoing messages for CMI channel |
| RESOURCE_METHOD_ENTRY/EXIT | For the Resource/Agent API method entry/exit tracing |
| REI_MESSAGE | For general debug details of incoming/outgoing messagesfor REI channel |
| TASK_METHOD_ENTRY/EXIT | For the Task API method entry/exit tracing |
| CMI_METHOD_ENTRY/EXIT | When entering/exiting a method in the CMI channel |
| TASK | Traces for the Tasks in the core API. |
| CONFIG_METHOD_ENTRY/EXIT | When entering/exiting a method in the RDI channel |
| RESOURCE | Traces for Resources in the core API |
| RDI_MESSAGE | For general debug details of incoming/outgoing messagesfor RDI channel |
| INTERACTION_METHOD_ENTRY/EXIT | For the Interaction API method entry/exit tracing |
| INTERACTION | For the Interaction API tracing |
| MPI_CALL | For traces related to a call / call processing inside the MPIchannel |
| MPI_EVENTS | For internal events processed inside the MPI channel |
| REI_METHOD_ENTRY/EXIT | When entering/exiting a method in the REI channel |
| WRI_EVENTS | For internal events processed inside the WRI channel |
| WRI_MESSAGE | For general debug details of incoming/outgoing messagesfor WRI channel |
| RESOURCE_EVENTS | For internal events posted or dispatched by the ResourceAPI |
| MESSAGE | For general debug details of incoming/outgoing messages |
| REI_EVENTS | For internal events processed inside the RDI channel |
| ASSIGNMENT_QUEUE | Traces for Assignment Queues in the core API |
| ENDPOINT_METHOD_ENTRY/EXIT | For the Endpoint API method entry/exit tracing |
| WRI_METHOD_ENTRY/EXIT | When entering/exiting a method in the WRI channel |
Contact Manager (CM)
| Trace | Description |
|---|---|
| CONFIG | Trace config activity |
| BRE_MESSAGE | Trace BRE messages |
| CONTACT_MANAGER | Trace contact activity |
| MPI_MESSAGE | Trace MPI messages |
| SYSTEM | Trace global activity |
| CONTACT | Trace contact activity |
| KB_MESSAGE | Trace KB messages |
| PARTICIPANT | Trace participant activity |
| WA_MESSAGE | Trace Work Assigner messages |
| ICM_MESSAGE | Trace ICM Gateway messages |
| CONTACT_DETAIL | Trace CONTACT_DETAIL activity |
Media Platform Interface (MPI)
Only set the traces in this section with the assistance of Cisco.
| Trace |
|---|
| SIP_STACK_DSUTIL_THREAD |
| SIP_STACK_AUTHENTICATION |
| SIP_STACK_DSSIPLLAPI_RESOLVER |
| SIP_STACK_DSSIPMLAPI_CALLSTATE |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT_ACK |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT_PRACK |
| SIP_STACK_DSSIPMLAPI_CALLMANAGEMENT |
| SIP_STACK_CONFIG |
| SIP_STACK_DSSIPLLAPI_CONNECTION |
| SIP_STACK_DSSIPLLAPI_LlSM |
| SIP_STACK_DSSIPOBJECT_HEADER |
| SIP_STACK_DSSIPMLAPI_REGISTRATION |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT |
| SIP_STACK_DSSIPREFER_REFER |
| SIP_STACK_DUMP |
| SIP_STACK_DSSIPLLAPI_LlSM_CLIENT |
| SIP_STACK_DSSIPDIALOG_OFFERANSWER |
| SIP_STACK_DSSIPMIME_MIME |
| MPI_CALL_TRACE |
| MPI_METHOD_TRACE |
| SIP_STACK_DSSIPLLAPI_LlSM_SERVER_SWITCHSTATE |
| SIP_STACK_DSUTIL_DSMESSAGESTATISTICS |
| SIP_STACK_DSSIPHLAPI_HLCALL |
| SIP_STACK_DSSIPLLAPI_CONNECTIONMANAGEMENT |
| SIP_STACK_DSSIPLLAPI_LlSM_CLIENT_SWITCHSTATE |
| SIP_STACK_DSSIPLLAPI_LlSM_CLIENT_TIMERS |
| SIP_STACK_DSSIPLLAPI_PERF |
| SIP_STACK_DSSIPEVENTS_EVENTS |
| SIP_STACK_DSSIPLLAPI_LlSM_CLIENT_USERCB |
| SIP_STACK_DSSIPLLAPI_WIRE |
| MPI_HANDLED_EXCEPTION_TRACE |
| SIP_STACK_DSSIPHLAPI_HLCALLMANAGEMENT |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT_CANCE |
| MPI_PARAM_TRACE |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT_REQUEST |
| MPI_LOW_LEVEL_TRACE |
| SIP_STACK_DSUTIL_SOCKET |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONKEY |
| SIP_STACK_DSSIPLLAPI_TRANSACTIONMANAGEMENT_RESPONSE |
| SIP_STACK_EXCEPTION |
| SIP_STACK_DSSIPLLAPI_LlSM_SERVER_USERCB |
| SIP_STACK_DSSIPLLAPI_LlSM_SERVER_TIMERS |
| SIP_STACK |
| SIP_STACK_DSSIPOBJECT_MESSAGE |
| SIP_STACK_DSSIPLLAPI_LlSM_SERVER |
Resource Desktop Adaptor (RDA)
| Trace | Description |
|---|---|
| CONFIG | Config trace bit is used to trace the RDA configuration issue with the database including loading, updating, deleting the message set, initial setup configuration properties, expert advisor configuration properties. |
| IM_ACTIVITY | IM activity is a place holder for future IM activity. Currently, no trace is used by this trace bit. |
| MPI_MESSAGE | MPI message trace bit is used to trace the RDA JMSmessage exchange between the MPI layer.
MPI message trace bit is used to trace the RDA JMSmessage exchange between the MPI layer. MPI layer including the agent presence subscription and notification, system user publication, registration to the presence service.It also indicates the condition of the IM message exchange between the expert advisor and the system. |
| SYSTEM | System trace bit is used to trace the RDA subsystem status such as whether the system is in partial service, out of service or in service It also trace the error condition like topic issue. |
| REI_MESSAGE | RDI message trace bit is used to trace the RDA JMSmessage exchange between the REI protocol. |
| PRESENCE_ACTIVITY | PRESENCE_ACTIVITY trace bit is a place holder for future PRESENCE activity. Currently, no trace is used by this trace bit. |
| PARSER | PARSER trace bit is place holder for any PARSER activity.Currently, no trace is used by this trace bit. |
| RDI_MESSAGE | RDI message trace bit is used to trace the RDA JMSmessage exchange between the RDI protocol.
RDI message trace bit is used to trace the RDA JMSmessage exchange between the RDI protocol. It mainly indicates the condition of the IM message exchange between the expert advisor with the RM subsystem including offerTaskRequest, re-prompting, taskAssignCmd and resource state change request. |
Reporting Server Subsystem
Reporting Subsystem (RS)
| Trace | Description |
|---|---|
| EXTRA_DEBUG | Only set this trace with the assistance of Cisco |
| DETAILED_DEBUG | Only set this trace with the assistance of Cisco |
| DEBUG | Only set this trace with the assistance of Cisco |
ORM Server Subsystem Trace Definitions
| Trace | Description |
|---|---|
| ORM (OAMP Resource Manager) | Only set these traces with the assistance of Cisco. |
OAMP Server Subsystem Trace Definitions
| Trace | Description |
|---|---|
| TRACE_BULK | To control logging for looping/bulk operations |
| TRACE_GENERAL_UI | For tracing the general OAMP UI |
| TRACE_EXCEPTION | For tracing Exceptions |
| TRACE_PARAM | For tracing Parameters |
| TRACE_DBACCESS | Trace DB Access for db fetch and modify such as Save,Update, Delete |
| TRACE_METHOD | For tracing of Entry/Exit of Methods |
Common Subsystem Trace Definitions
Infrastructure
| Trace | Description |
|---|---|
| TRACE_STATS | Operations of the Stats Manager |
| TRACE_SERVICEABILITY | Traces to do serviceability, the act of logging and tracing |
| TRACE_THREAD | All Infrastructure thread operations |
| TRACE_SNMP | TRACE_SNMP SNMP Forwarder/logging/stats |
| TRACE_SHUTDOWN | Log detailed shutdown info |
| TRACE_LICENSING | Log any/all licensing operations |
| TRACE_STARTUP | Log detailed startup info |
| LOAD_SUBSYSTEM | When loading subsystems |
| TRACE_TIMER | Logs when a Timer expires |
OAMP_BO
| Trace | Description |
|---|---|
| TRACE_BULK | To control logging for looping/bulk operations |
| TRACE_EXCEPTION | For tracing Exceptions |
| TRACE_GENERAL_BO | General Traces for OAMP back-end |
| TRACE_PARAM | For tracing Parameters |
| TRACE_DBACCESS | Trace DB Access for db fetch and modify such as Save,Update, Delete |
| TRACE_METHOD | For tracing of Entry/Exit of Methods |
