General Trouble shooting Tips:
1) When "CallType RealTime" Tracing is enabled in router logs, Similar message would be seen for the Call_Type being initialized, when the call was answered and also handling the TCD from OPC if it is enabled. Logs with CallType RealTime tracing. 21:45:37 Trace: CT_SL_Timer(40)(Init): CT(5000,CT_1), Threshold(15 + 2). …. 21:45:43 Trace: CT_SL_Timer(40), CT(5000,CT_1), DeleteSelf(0). 21:45:43 Trace: CountCallsAnswered: AnswerWaitTime = 3. …… 21:45:47 Trace: Handling ClosedCallRecord_V11: RCK(149494-223), CallType(5000,CT_1). 21:45:47 Trace: Handling TCD: CallType(5000,CT_1), SLTimer ID(40), disposition (13) 21:45:47 Trace: CT_SL_Timer(40)(delete), CT(5000,CT_1).
2) When multiple processes' logs and tables are collected, make sure all of them are available to the DE for the same time frame when the issue occurs. Make sure to turn off the tracing mentioned after the logs are collected.
3) Check the ICM version and if any ES installed.
4) Refer to the following link:
|Symptom/Observation / Issues||Initial Analysis Checklist||Resolution Steps/Data Collection Inputs/Things to do by TAC|
|Router assert/crash/restarted||1) Collect the router.exe, .pdb ,minidump file and router logs for that time frame.
2) Analyze the logs to see if router has hit a EMSFailAssert point before the crash/restart. Search for already existing defects or cases using the CallStack printed on the router logs as the search string. 3) collect the output of the dumpcfg command, this might help the analysis if the crash if due to a configuration change done. refer to the link for dumpcfg http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00806919d8.shtml
|Routing issues||1) Call flow of the customer.|
|Problems with Label (Agent) selection in IPCC?VRU and Label related problems?||1) Check if the label is configured for the correct Routing Client. In case of routing to an Agent, check the labels configured under Devicetarget Explorer / Agent targeting Rule.
In case of Routing to VRU, check the label configured under Network VRU Explorer. Depending on the which node is used (Send to VRU /TransRouteToVRU) check the call flow along with the Type of VRU that is configured. Note: When Configuring a Type 10 VRU, do not select the NewtworkVRU on the Advance tab of the PG explorer for the VRU PG. 2) On any issue with the customer's configuration, take a back up of the Logger DB configuration through ICMDBA. Refer to the link for more info
|Router becomes unresponsive and appears to not processing any messages for ~ seconds (Determined from Router Logs)||Look for any RTS restarts that happened during the timeframe of the pause. If so, enable the Registry for partial Real Time Update Fix(7.2.6 upwards) HKLM\Software\Cisco System,Inc.\<inst>\Router<side>\Router\CurrentVersion\RealTimeData\PartialUpdateInterval (In milliseconds)
Threshold: If threshold is set to 0, router will send all of the real time data at once. PartialUpdateInterval(in milliseconds): the default value is 500ms. If it is set to 0, router will send all of the real time data at once.
|Scripteditor crash/problems.||Script/IS Editor crash?||1) Event viewer logs and edt logs(tracing ffff)|
|Scripting problems? Script problems due to config issues? Script validation problems?||1) Check if the issue is specific to Script Editor or is it applicable to ISE also.
2) Does it happen on Client AW also, if yes, are the versions different in both distributor and client AW.
|VRUPIM Problems||VRU PIM crash/Restart?||Collect the event viewer logs, vrupim logs Collect the router.exe, .pdb ,minidump file along with the exact version info(any ES installed)|
|Translation route to VRU failures?||Check for "Consider if" condition on the TransRouteToVRU node (if used).Enable watch for the same and analyze one set of logs to see if the trunks status is reported to the Router.|
|Connectivity between the VRU and the VRU PIM||Enable the EMSUserData of the VRUPIM to hex 7F F7 E0
EMSTraceMask is not used and should be set to 0. Check for the Heartbeat messages between them. Collect the VRUCAP files which would capture the messages between the PIM and the VRU. Refer to the link for collect the VRUCAP files. http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00800949b1.shtml
|VRU-VRUPIM related issue||Refer to the below document for any VRUPIM/VRU related issue:|
|Reporting issues||Discrepancy In Reporting||Watch commands can be enable on router logs using the rttest utility to watch a particular field to see if its incremented or decremented.
1) For example to see the variables associated with a Skill Group, run rttest utility from the command prompt, and run the command dump_vars /group <Skill group's Skill target ID given in the logger DB> Output of this will list the variables for which the watch command can be enabled. Use "dump_vars /?" to know on what other categories the command can be used. 2) Here is an Example of a Watch on a Skill group's one of the variable that would appear on the above list (RouterCallsOfferedHalf) watch SkillGroup.<Skill group name>.RouterCallsOfferedHalf For more info on rttest utility refer to http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00800ac69b.shtml 3) For Half Hour reporting issues, collect table from the Logger/HDS. 4) For Real Time reporting issues, collect the table output from AWDB. 5) The below information on CallType and Skill group reporting, would aid to zero down the issue.
|Skill Group Reporting||1) Skill Group Reporting is done by the PG except for the Service Level Calculation is done by router.
2) Collect the TCD, RCD, and the corresponding Skill Group table Report(half Hour)
|Call Type Reporting||1) For CallType "Offered" field is incremented by router.
2) Fields like handled and abandoned (their respective service level calculation) depends on the TCD(Call disposition flag) cut by the PG. Refer to the DB Schema for Call disposition. Call values. 3) Answered fields depends on the real time input from the PG.
|Tracing||CallType: From RtrTrace enable- Route Request, Translation Route, Call Queuing, CallType RealTime, Network VRU Tracing.
From rttest, enable the appropriate watch commands on the fields that have an issue.
|DBlookup issues.||DBWorker not connecting to DB||1) Refer to the following link on how to setup Login Credentials for external databases accessed by the DBWorker Process.
2) A common mistake which is made is to give the External Database Server's IP Address in the ICM Configuration (i.e. DBLookUp Explorer Tool through Configuration Manager) and/or the Registry under the Router key - "HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\ICM\<instance name>\Router<side>\Router\CurrentVersion\Configuration\Database\SQLLogin".
Please make sure that this should be given as Servername and Servername alone. Giving this as IP Address at either place would cause the DBWorker to fail connecting to the External DB.
|DBLookup Node failing||1) Check the DbWorker logs for any errors.
2) Make sure to have the primary key column specified in the DBLookUp Explorer.
|For all the issues, Collect the DBWorker logs with ffff tracing enabled and corresponding router logs along with Event Viewer logs.|
|Application Gateway Lookup issues.||AppGateway Process not connecting to the Application.||1) Collect the AppGW (AGI) logs with ffff tracing enabled and router logs with CICR Request tracing enabled from the ICM side. From the Application side, collect the AGI Host logs with ffff tracing enabled. Check the logs if the heartbeat message request/response is received.
2) In Application Gateway List on the ICM Configuration manager make sure to select the Type as "custom gateway" for Appgateway Lookup against Remote ICM which is used in cases of NAM/CICM and ICM-ICM deployments. 3) The port number given with the IP Address on the Connection Tab on the Application Gateway List should match the Application Gateway port number on the Host machine given under Cisco ICM GatewayHost configurator.