Cisco Unified Contact Center Serviceability Guide for release 7.x

From DocWiki

Jump to: navigation, search

CONTACT CENTER SERVICEABILITY GUIDE


This document is supplemental guide to our existing network management documentation. It provides detailed (vs. abstract) information and “How To” instructions for customers to better manage their Contact Center installations.

<

Contents

Introduction

This document is supplemental guide to our existing network management documentation. It provides a detailed (vs. abstract) information and “How To” instructions for to customers to better manage their Contact Center installations. It includes, but is not limited to, the following topics

  • platform management
  • event management
  • event correlation
  • troubleshooting techniques.

This is a living document that is expected to be continually updated to meet the needs of Contact Center customers.

References

SNMP Health Monitoring

Contact Center Applications MIB

The Cisco Contact Center Application MIB contains tables of objects and for the following ICM/IPCC components.

  • Routers
  • Loggers
  • Peripheral Gateways (PGs)
  • Distributor Admin Workstations (AWs)
  • CTI Gateways (CGs)
  • CTI OS Servers

Contact Center Application MIB SNMP Agent provides access to component inventory, component status, performance metrics, and links to IETF standard host-based MIBs. An example of the data provided by an actual ICM installation is shown in appendix A.

Host Resources MIB

The Host Resources MIB SNMP Agent is a complete implementation of the Host Resources MIB, proposed standard RFC 1514. The Host Resources MIB is also compliant with Host Resources MIB, draft standard RFC 2790. The agent provides SNMP access to useful host information, such as the storage resources, process table, device information, and the installed software base.

Each cccaComponentElmtEntry in the cccaComponentElmtTable corresponds to an ICM/IPCC managed process. ccaComponentElmtName contains the process executable name without the .exe. ccaComponentElmtRunID contains the process id which can be used as an index to the Host Resources MIB to obtain current values from the table's hrSWRunTable and hrSWRunPerfTable. This relationship is shown in the following example for cccaComponentElmtRunID.0.1.5 = 5384 using the results in appendix Error: Reference source not found and a subset of the results provided by the Host Resources MIB SNMP agent on the same system.

cccaComponentElmtName.0.1.5 = router
cccaComponentElmtRunID.0.1.5 = 4040
cccaComponentElmtStatus.0.1.5 = active(5)
hrSWRunIndex.4040= 4040
hrSWRunName.4040= router.exe
hrSWRunPath.4040= C:/icm/bin/router.exe
hrSWRunType.4040= application(4)
hrSWRunStatus.4040= notRunnable(3)
hrSWRunPerfCPU.4040= 20
hrSWRunPerfMem.4040= 6428

System Applications MIB

The SYSAPPL MIB SNMP Agent is an implementation of the System-level Managed Objects for Applications (SYSAPPL) MIB, RFC 2287. The SYSAPPL MIB supports configuration, fault detection, performance monitoring, and control of application software. It provides for tables that define an application as a series of processes and services. This includes objects for applications installed on the system, elements and processes that are included in an application, and current and previously run applications.

SNMP Notifications

ICM/IPCC SNMP notifications are stateful events. Each event correlates to a managed object which can either be dual or single state.

Dual State Objects

Most objects are defined as dual state having either a raised state which means that the object has a problem or fault associated with it, and a cleared state which indicates that the object is operating normally. Dual state ICM/IPCC SNMP notifications contain either a value of raise(4) or clear(0)in the cccaEventState field. In some cases, multiple raise notifications can correlate to the same object. For example, an object can go offline for a variety of reasons: process crash, network failure, software fault, etc. A SNMP notification's cccaEventComponentId field specifies a unique identifier that can be used to correlate common raise and clear notifications to a single managed object. The following example shows a pair of raise and clear notifications received at a management station with the same cccaEventComponentId.

snmpTrapOID.0 = cccaIcmEvent
cccaEventComponentId = 4_1_CCBU-CSA-RGR1A_ICM\acme\RouterA
cccaEventState = raise(4)
cccaEventMessageId = 2701295877
cccaEventOriginatingNode = CCBU-CSA-RGR1A\acme
cccaEventOriginatingNodeType = router(1)
cccaEventOriginatingProcessName = nm
cccaEventOriginatingSide = sideA(1)
cccaEventDmpId = 0
cccaEventSeverity = warning(2)
cccaEventTimestamp = 2006-03-31,14:19:42.0
cccaEventText = The operator/administrator has shutdown the ICM software on ICM\acme\RouterA.
snmpTrapOID.0 = cccaIcmEvent
cccaEventComponentId = 4_1_CCBU-CSA-RGR1A_ICM\acme\RouterA
cccaEventState = clear(0)
cccaEventMessageId = 1627554051
cccaEventOriginatingNode = CCBU-CSA-RGR1A\acme
cccaEventOriginatingNodeType = router(1)
cccaEventOriginatingProcessName = nm
cccaEventOriginatingSide = sideA(1)
cccaEventDmpId = 0
cccaEventSeverity = informational(1)
cccaEventTimestamp = 2006-03-31,13:54:12.0
cccaEventText = ICM\acme\RouterA Node Manager started.  Last shutdown was by operator request.

The CCCA-Notifications.txt file, installed in the icm\snmp directory as part of ICM/IPCC installation, contains the complete set of SNMP notifications including and can be used to identify grouped events. The Correlation ID is the data used to generate the cccaEventComponentId which is determined at run time. The following entries correspond to the SNMP notifications in the above example.


NOTIFICATION 1028105
cccaEventMessageId 2701295877 (0xA1028105)
DESCRIPTION Node Manager on the ICM node has been given the command to stop ICM services. This occurs when an operator/administrator stops ICM services using ICM Service Control, 'nmstop', 'net stop', Control Panel Services, or shuts down the node.
cccaEventState Raise
SUBSTITUTION STRING The operator/administrator has shutdown the ICM software on %1.
ACTION Contact the operator/administrator to determine the reason for the shutdown.
cccaEventComponentId { cccaEventOriginatingNode %1 }
CorrelationId { CLASS_NM_INITIALIZING cccaEventOrginatingNode %1 }
NOTIFICATION 1028103
cccaEventMessageId 1627554051 (0x61028103)
DESCRIPTION The Node Manager successfully started. The last reason the Node Manager stopped was because a clean shutdown of the ICM code was requested by the operator.
cccaEventState Clear
SUBSTITUTION STRING  %1 Node Manager started. Last shutdown was by operator request.
ACTION No action is required.
cccaEventComponentId { cccaEventOriginatingNode %1 }
CorrelationId { CLASS_NM_INITIALIZING cccaEventOrginatingNode %1 }

Single State Objects

An object can also be defined as single state having only a raised state Since there is no corresponding Clear event. it is necessary for a TAC engineer to manually clear the object. Single state objects are typically used where a corresponding Clear event cannot be tracked, for example the database being corrupted. Single state ICM/IPCC SNMP notifications contain either a value of raise(9)in the cccaEventState field. They are identified in the CCCA-Notifications.txt file with a value of Single‑state Raise for cccaEventState as shown in the following example.


NOTIFICATION 105023C
cccaEventMessageId 3775201852 (0xE105023C)
DESCRIPTION The router has detected that it is no longer synchronized with its partner. One result of this is that the router might be routing some calls incorrectly.
cccaEventState Single-state Raise
SUBSTITUTION STRING The router has detected that it is no longer synchronized with its partner.
ACTION Recommended action: Stop the router on both sides. After both sides are completely stopped, restart both routers.

Alternate Action: Restart the router on one side. After doing this, the routers might still route some calls incorrectly, but they will be in sync. Other actions: Collect all rtr, mds, ccag process logs from both routers from the entire day. Collect all sync*.sod files (where * is some number) that exist in the icr\<instance>\ra directory of router A and in the icr\<instance>\rb directory of router B.

Contact the Support Center.

cccaEventComponentId {cccaEventOriginatingNode cccaEventOriginatingProcessName cccaEventOriginatingSide }
CorrelationId {CLASS_RTR_SYNC_CHECK cccaEventOriginatingNode cccaEventOriginatingProcessName cccaEventOriginatingSide }

Monitoring SNMP Notifications

Using the contents of the following ICM/IPCC SNMP notification fields, a SNMP Monitoring tool can group ICM/IPCC SNMP notifications in an organized, hierarchical manner.


cccaEventOriginatingNode = CCBU-CSA-RGR1A\acme

cccaEventOriginatingNodeType = router(1)
cccaEventOriginatingSide = sideA(1)
0100090000030202000002008a01000000008a01000026060f000a03574d46430100000000000100bf7b0000000001000000e802000000000000e8020000010000006c00000000000000000000002c0000007100000000000000000000007d400000bd10000020454d4600000100e80200000e000000020000000000000000000000000000009913000064190000d400000013010000000000000000000000000000943d0300de320400160000000c000000180000000a00000010000000000000000000000009000000100000003c0f0000f3030000250000000c0000000e000080120000000c000000010000005200000070010000010000009cffffff000000000000000000000000900100000000000004400012540069006d006500730020004e0065007700200052006f006d0061006e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002331093000000000040000000000ae30603109300000000047169001000002020603050405020304877a0020000000800800000000000000ff01000000000000540069006d00650073002000000065007700200052006f006d0061006e00000000000000f22f093070cbae3000331400010000000000000004481100f2b40230044811006c4eaf301c4811006476000800000000250000000c00000001000000180000000c00000000000002540000005400000000000000000000002c00000071000000010000006238874076628740000000005a000000010000004c0000000400000000000000000000003c0f0000f50300005000000020002c002d00000046000000280000001c0000004744494302000000ffffffffffffffff3d0f0000f4030000000000004600000014000000080000004744494303000000250000000c0000000e0000800e000000140000000000000010000000140000000400000003010800050000000b0200000000050000000c029d005c02040000002e0118001c000000fb020200010000000000bc02000000000102022253797374656d0000000000000000000000000000000000000000000000000000040000002d01000004000000020101001c000000fb02f1ff0000000000009001000000000440001254696d6573204e657720526f6d616e0000000000000000000000000000000000040000002d010100050000000902000000020d000000320a0e00000001000400000000005b029d0020cf0700040000002d010000030000000000

where:

  • ICM Node Name = left side of cccaEventOriginatingNode (CCBU-CSA-RGR1A)
  • Instance Name = right side of cccaEventOriginatingNode (acme)
  • Component name = cccaEventOriginatingNodeType + cccaEventOriginatingSide letter (routerA)

For example:

0100090000030202000002008a01000000008a01000026060f000a03574d46430100000000000100bf7b0000000001000000e802000000000000e8020000010000006c00000000000000000000002c0000007100000000000000000000007d400000bd10000020454d4600000100e80200000e000000020000000000000000000000000000009913000064190000d400000013010000000000000000000000000000943d0300de320400160000000c000000180000000a00000010000000000000000000000009000000100000003c0f0000f3030000250000000c0000000e000080120000000c000000010000005200000070010000010000009cffffff000000000000000000000000900100000000000004400012540069006d006500730020004e0065007700200052006f006d0061006e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002331093000000000040000000000ae30603109300000000047169001000002020603050405020304877a0020000000800800000000000000ff01000000000000540069006d00650073002000000065007700200052006f006d0061006e00000000000000f22f093070cbae3000331400010000000000000004481100f2b40230044811006c4eaf301c4811006476000800000000250000000c00000001000000180000000c00000000000002540000005400000000000000000000002c00000071000000010000006238874076628740000000005a000000010000004c0000000400000000000000000000003c0f0000f50300005000000020002c002d00000046000000280000001c0000004744494302000000ffffffffffffffff3d0f0000f4030000000000004600000014000000080000004744494303000000250000000c0000000e0000800e000000140000000000000010000000140000000400000003010800050000000b0200000000050000000c029d005c02040000002e0118001c000000fb020200010000000000bc02000000000102022253797374656d0000000000000000000000000000000000000000000000000000040000002d01000004000000020101001c000000fb02f1ff0000000000009001000000000440001254696d6573204e657720526f6d616e0000000000000000000000000000000000040000002d010100050000000902000000020d000000320a0e00000001000400000000005b029d0020cf0700040000002d010000030000000000

Within this node, raise and clear events with the same cccaEventComponentId can be grouped as a single object.

Logger Heartbeat Notification

The csfs heartbeat notification is arguably the most critical SNMP notification that should be monitored.


cccaEventMessageId
cccaEventState
SUBSTITUTION STRING
1630142467 (0x612A0003)
Clear
HeartBeat Event for %1

The Message ID for this event is defined incorrectly in the CCCA-Notifications.txt file as 19529731 (0x12A0003).

The notification is sent periodically by the Logger csfs process to indicate that there is a healthy connection between the Router and the Logger and that the Logger's SNMP notification feed is active. The interval is set to 720 minutes (12 hours) by default, most likely that high for using a modem to communicate notifications, and is configurable via the Windows Registry value

heartbeatIntervalMinutes

in

HKLM\SOFTWARE\Cisco Systems, Inc.\ICM\<instance>\Logger<A or B>\CSFS\

CurrentVersion

The actual interval can be as much as 1 minute longer than the configured interval so the logic that reacts to these events should employ a certain "deadband".

Critical SNMP Notifications

The following table contains a subset of SNMP Notifications that should be considered immediately actionable. However, the entire list of events should be considered important and be monitored. Event details can be found in the CCCA-Notifications.txt file.


cccaEventMessageId
cccaEventState
SUBSTITUTION STRING
3775053825 (0xE102C001)
Raise
Critical process %1 died. Rebooting node.
2701312003 (0xA102C003)
Clear
Restarting process %1.
2701312009 (0xA102C009)
Raise
Process %4 exited after %1 seconds. Minimum required uptime for %4 process is %2 seconds. Delaying process restart for %3 seconds.
2701312010 (0xA102C00A)
Clear
Restarting process %2 after having delayed restart for %1 seconds.
3775053835 (0xE102C00B)
Raise
Terminating process %1.
3775053836 (0xE102C00C)
Raise
Process %1 exited after having detected a software failure.
2701312013 (0xA102C00D)
Raise
Process %1 detected failure and requested that it be restarted by the Node Manager.
3775053838 (0xE102C00E)
Raise
Process %1 exited with unexpected exit code %2.
2701312015 (0xA102C00F)
Raise
Process %3 exited after %1 seconds. Process restart will be delayed for a minimum of %2 seconds.
2701312016 (0xA102C010)
Clear
Process %1 successfully reinitialized after restart.
1627570193 (0x6102C011)
Clear
Process %1 successfully started.
2701312018 (0xA102C012)
Raise
Process %1 exited cleanly and requested that it be restarted by the Node Manager.
3775053844 (0xE102C014)
Raise
Process %1 exited and requested that the Node Manager reboot the system.
3775054081 (0xE102C101)
Raise
 %1 node critical process %2 died. Rebooting node.
2701312259 (0xA102C103)
Clear
 %1 node restarting process %2.
1627570439 (0x6102C107)
Clear
 %1 Node Manager started. Last shutdown was for reboot after failure of critical process.
3775054088 (0xE102C108)
Clear
 %1 Node Manager started. Last shutdown was for unknown reasons. Possible causes include a power failure, a system crash or a Node Manager crash.
2701312265 (0xA102C109)
Raise
 %4 node process %5 exited after %1 seconds. Minimum required uptime for %5 process is %2 seconds. Delaying process restart for %3 seconds.
2701312266 (0xA102C10A)
Clear
 %2 node restarting process %3 after having delayed restart for %1 seconds.
3775054091 (0xE102C10B)
Raise
Terminating process %2.
3775054092 (0xE102C10C)
Raise
 %1 node process %2 exited after having detected a software failure.
2701312269 (0xA102C10D)
Raise
Process %2 on %1 has detected a failure. Node Manager is restarting the process.
3775054094 (0xE102C10E)
Raise
Process %2 on %1 went down for unknown reason. Exit code %3. It will be automatically restarted.
2701312271 (0xA102C10F)
Raise
Process %4 on %3 is down after running for %1 seconds. It will restart after delaying %2 seconds for related operations to complete.
2701312272 (0xA102C110)
Clear
 %1 node process %2 successfully reinitialized after restart.
1627570449 (0x6102C111)
Clear
 %1 node process %2 successfully started.
2701312274 (0xA102C112)
Raise
 %1 node process %2 exited cleanly and requested that it be restarted by the Node Manager.
3775054100 (0xE102C114)
Raise
 %1 node process %2 exited and requested that the Node Manager reboot the system.
3775057921 (0xE102D001)
Raise
Node Manager crashed after having been up for %1 seconds. Scheduling system reboot in %2 seconds.
3775057922 (0xE102D002)
Raise
Node Manager crashed after having been up for %1 seconds. Auto-reboot is disabled. Will attempt service restart.
3775057923 (0xE102D003)
Raise
Node Manager requested reboot after having been up for %1 seconds. Scheduling system reboot in %2 seconds.
3775057924 (0xE102D004)
Raise
Node Manager requested reboot after having been up for %1 seconds. Auto-reboot is disabled. Will attempt service restart.
3775058177 (0xE102D101)
Raise
 %3 Node Manager crashed after having been up for %1 seconds. Scheduling system reboot in %2 seconds.
3775058178 (0xE102D102)
Raise
 %2 Node Manager crashed after having been up for %1 seconds. Auto-reboot is disabled. Will attempt service restart.
3775058179 (0xE102D103)
Raise
 %3 Node Manager requested reboot after having been up for %1 seconds. Scheduling system reboot in %2 seconds.
3775058180 (0xE102D104)
Raise
 %2 Node Manager requested reboot after having been up for %1 seconds. Auto-reboot is disabled. Will attempt service restart.
3775058181 (0xE102D105)
Raise
 %2 A Critical Process has requested a reboot after the service has been up for %1 seconds. Auto-reboot on Process Request is disabled. Will attempt service restart.
3775058182 (0xE102D106)
Raise
 %3 A Critical Process has requested a reboot after having been up for %1 seconds. Scheduling system reboot in %2 seconds.
2701393936 (0xA1040010)
Raise
Synchronizer timed out trying to establish connection to peer.
3775135778 (0xE1040022)
Raise
Connectivity with duplexed partner has been lost due a failure of the private network, or duplexed partner is out of service.
1627652131 (0x61040023)
Clear
Communication with peer Synchronizer established.
1627717757 (0x6105007D)
Clear
Peripheral %2 (ID %1) is on-line.
3775201406 (0xE105007E)
Raise
ACD/IVR %2 (ID %1) is off-line and not visible to the Peripheral Gateway. Routing to this site is impacted.
1627717840 (0x610500D0)
Clear
Physical controller %2 (ID %1) is on-line.
3775201489 (0xE10500D1)
Raise
Peripheral Gateway %2 (ID %1) is not connected to the Central Controller or is out of service. Routing to this site is impacted.
1627717842 (0x610500D2)
Clear
PG has reported that peripheral %2 (ID %1) is operational.
3775201491 (0xE10500D3)
Raise
PG has reported that peripheral %2 (ID %1) is not operational.
1627717887 (0x610500FF)
Clear
Side %1 %2 process is OK.
3775201536 (0xE1050100)
Raise
Process %2 at the Central Site side %1 is down.
1627718129 (0x610501F1)
Clear
ICM Node %2 (ID %1) is on-line.
3775201778 (0xE10501F2)
Raise
ICM Node %2 (ID %1) is off-line.
1627718136 (0x610501F8)
Clear
ICM Node %2 (ID %1) on system %3 is on-line.

ICM/IPCC SNMP Agent Configuration

The ICM/IPCC SNMP component is installed and enabled automatically as part of ICM/IPCC installation replacing the Windows SNMP service. In addition to providing SNMP agents for the Contact Center Applications MIB, Systems Applications MIB, and Host Resources MIB, ICM/IPCC SNMP also provides support for all installed SNMP extension agents supported by the Windows SNMP service.

However, no Windows SNMP configuration, specifically security and trap destinations, is retained and must be recreated using the SNMP Agent Configuration Tool. The tool provides the following configuration menus.

  • General Agent Properties
  • SNMP v1/v2c Community Strings
  • SNMP v3 User Names, Authentication/Privacy Protocols
  • SNMP Trap Destinations
  • SYSLOG Feed

These are described in the SNMP Guide for Cisco ICM/IPCC Enterprise and Hosted Editions.

IMPORTANT: Configuration MUST be performed before any SNMP operations or traps will work.

Monitoring ICM/IPCC Processes

Each ICM/IPCC component consists of one or more processes which are managed by NodeManager. Each component has a separate NodeManager which is installed as a Windows services. All NodeManager services have the same process name, nodeman.exe. Each component has a core set of processes that are enabled and managed by NodeManager. Additional processes may also be enabled depending on setup and/or configuration options.

  • Router

ccagent.exe

dbagent.exe

mdsproc.exe

router.exe

rtsvr.exe

testsync.exe

A separate process will be enabled for each NIC enabled during setup. In our example, the crspnic.exe process has been enabled.

  • Logger

configlogger.exe

csfs.exe

cw2kfeed.exe

histlogger.exe

recovery.exe

  • Peripheral Gateway (PGs)

mdsproc.exe

opc.exe

pgagent.exe

testsync.exe

A separate process will be enabled for each PIM enabled during setup. Multiple PIMs of the same type can be enabled for a PG. In our example, the crspnic.exe process has been enabled, two instances of the mrpim.exe process have been enabled.

  • Distributor Admin Workstation (AW)

configlogger.exe

rtclient.exe

updateaw.exe

rtdist.exe

From the Local Desktop

Using ICM/IPCC and Windows Tools

ICM Service Control displays the NodeManager service for each component using the format

Cisco ICM <instance> <component>

The following screen-shot displays all the component NodeManager services and their current state on our target machine. The Router component's NodeManager service is identified as "Cisco ICM acme RouterA" and its status is "Running".


[[Image:]]

Each running process has an associated window on the desktop. The window title bar uniquely identifies each process as follows. Some processes may display additional status information.

<instance>-<component> <process>

The ICM/IPCC window title bars are displayed on the Windows Task Manager Application window. The following screen shot shows all the running processes for the Router component.


[[Image:]]


Right clicking on a process brings up a menu with a "Go To Process" entry. Selecting this entry will bring the user to the corresponding process entry in the Task Manager Process window. The entry for the router.exe process highlighted above is shown in the following screen shot.


[[Image:]]


Using the Local Registry

The ICM Windows registry hive contains the set of all installed components and their processes. However, to determine which processes are being managed, the user will have to traverse the NodeManager registry key for each component. The following screen-shot shows the set of processes included with the acme‑RouterA router component. The key name is typically not the same as the process name. The process name is contained in the ImageName value without the .exe. In our example, the key name for the router process is rtr. The process name is contained in the ImageName value shown in the right pane. If the ProcDisabled value is set to 0, as is the case for the router process, the process will be started and managed by the RouterA NodeManager process.

[[Image:]]


From a SNMP Management Station

In addition to the information available using the local desktop tools and registry, the Contact Center SNMP agent returns information about all ICM/IPCC enabled processes whether or not they are running. This information is available from the cccaInstanceTable, cccaComponentTable, and cccaComponentElmtTable. The instance number and component index correlate a process to a specific instance and component. The cccaComponentElmtRunID value, which the process id, is valid if the cccaComponentElmtStatus is not stopped(0).

Here are the entries for acme-RouterA router process which is running.

cccaInstanceName.0 = acme

cccaComponentType.0.1 = router(1)
cccaComponentName.0.1 = RouterA
cccaComponentStatus.0.1 = started(4)
cccaComponentElmtName.0.1.5 = router
cccaComponentElmtRunID.0.1.5 = 4040
cccaComponentElmtStatus.0.1.5 = active(5)

Here are the entries for acme-LoggerA configlogger process which is stopped and therefore contains a cccaComponentElmtRunID value of 0.

cccaInstanceName.0 = acme
cccaComponentType.0.2 = logger(2)
cccaComponentName.0.2 = LoggerA
cccaComponentStatus.0.2 = stopped(3)
cccaComponentElmtName.0.2.8 = configlogger
cccaComponentElmtRunID.0.2.8 = 0
cccaComponentElmtStatus.0.2.5 = stopped(3)

Syslog Event Feed - CiscoLog Format

At ICM 7.2(1), the Syslog event feed has been altered to format all events in CiscoLog format. CiscoLog specification provides the following key benefits:

  • Precisely documented message format for wide interoperability.
  • Compatible with IOS message format.
  • Support for logging directly to a log file.
  • Support for local and remote logging through the syslog protocol.
  • Support for logging directly to file in the same format as via the syslog protocol.
  • Precise message source identification with host, IP address, application, process, etc.
  • Message ordering with sequence numbers and time stamp with millisecond precision.
  • Support for tagging of messages for correlation or external filtering.
  • Support for internationalization of host, tags and message text.
  • In a supplement document, defines an initial set of standard tags.

The CiscoLog event format is:

<PRI>SEQNUM: HOST: MONTH DAY YEAR HOUR:MINUTES:SECONDS.MILLISECONDS TIMEZONE: 

%APPNAME-SEVERITY-MSGID: %TAGS: MESSAGE

where
  • PRI - syslog PRI Part (RFC-3164) which is set to LOCAL0(16 << 3)|SEVERITY
  • SEQNUM - used to order messages. Starts at 0
  • HOST - hostname of originating system
  • MONTH - current month name in Mmm format ("Jan", "Feb", ..." Dec")
  • DAY - current day in dd format (01 to 31)
  • YEAR - current year in yyyy format (e.g. 2007)
  • HOUR - hour of the timestamp in 24-hour hh format, (00 to 23)
  • MINUTE - minute of the timestamp in mm format (00 to 59)
  • SECONDS - second of the timestamp in ss format (00 to 59)
  • MILLISECONDS - millisecond of the timestamp in mmm format (000 to 999)
  • TIMZONE - abbreviated Timezone offset, set to +0000 (UTC)
  • APPNAME - name of the application generating the event. Consists of the following fields

PRODUCT_COMPONENT_SUBCOMPONENT

PRODUCT - e.g., ICM

COMPONENT - e.g., Router, Logger, PG, AW, etc.

SUBCOMPONENT - e.g. CallRouter, NodeManager, INRCEngine

  • SEVERITY - supported values are 3 (Error), 4 (Warning), 6 (Informational), 7 (Debug)
  • MSGID - hexadecimal message id that uniquely identifies the event (e.g. 10500FF)
  • TAGS - supported tags are [comp=%s][pname=%s][iid=%s][mid=%s][sev=%s]

[comp=%s] - component name including side, e.g., Router-A

[pname=%s] - process name, e.g., rtr

[iid=%s] - instance name, e.g. ipcc1

[mid=%d] - message id, e.g. 10500FF

[sev=%s] - severity, one of "error", "warning", "info", or "debug"

  • MESSAGE - The unique event message to be displayed.

Example (an actual entry would be displayed on a single line).

<134>25: host-w3k: Feb 13 2007 18:23:21.408 +0000: %ICM_Router_CallRouter-6-10500FF: [comp=Router A][pname=rtr][iid=ipcc1][mid=10500FF][sev=info]: Side A rtr process is OK.

Dumplog Utility - optional CiscoLog Format

The Dumplog utility coverts binary log files written by ICM processes into readable text format. The document, How to Use the Dumplog Utility, provides all the details needed to understand and use the utility.

An enhancement has been added to Dumplog with ICM 7.2(1) to optionally display the binary log files in CiscoLog format. See the reference to the CiscoLog specification in section Error: Reference source not found for details about CiscoLog format.

Header

CiscoLog formatted log entries include a more comprehensive header compared to dumplog standard format.

Dumplog standard format

<TIMESTAMP> <COMPONENT-PROCESS> <MESSAGE>

where the timestamp is a 24 hour value hh:mm:ss. It does not include the date which is displayed on a separate line at the beginning of the file and when a new day starts

Events from February 8, 2007:

Example.

00:37:44 ra-rtr MDS is in service.

CiscoLog format

CiscoLog formatted dumplog entries are displayed using the following fields.

<SEQNUM>: <HOST>: <TIMESTAMP> <TIMEZONE>: %APPNAME: %<TAGS>: <MESSAGE>

See section Error: Reference source not found for a detailed description of the CiscoLog fields. The contents of the following fields are different from those in section Error: Reference source not found.

  • APPNAME

PRODUCT_COMPONENT_MESSAGECATEGORY

PRODUCT - always ICM

COMPONENT - e.g. ROUTER, LOGGER, PG, AW, etc.

MESSAGECATEGORY - e.g. Diagnostic, ProcessSynchronization, MessageDelivery, etc.

  • TAGS - supported tags are [comp=%s][pname=%s][iid=%s][sev=%s] and optionally [part=%1.%2/%3]
[part=1.%2/%3] - is used only for multi-line entries described in section Error: Reference source not found.

Example (an actual entry would be displayed on a single line).

10: CICMRGRA: Feb  8 2007 05:37:44.658 +0000: %ICM_Router_ProcessSynchronization: [comp=Router‑A][pname=rtr][iid=ipcc][sev=info]: MDS is in service.

Timestamp

Timestamps displayed in dumplog standard format are in local time relative to the machine on which dumplog is run.

Timestamps displayed in CiscoLog format are displayed in UTC time independent of the machine on which dumplog is run.

IMPORTANT: Date/Time options specified on the command line are entered in local time, even when the CiscoLog option is selected. Therefore, timestamps displayed as part of the CiscoLog formatted entry may appear to be outside the date/time range selected.

Multi-line Entries

The message portion of some dumplog entries may contain one or more embedded new line characters, '\n', causing the message to be displayed on multiple lines which may also include blank lines. This is especially true for entries containing statistics. For a dumplog standard formatted message, only the first line will contain the header field as follows:

00:36:09 ra-nm ICM\ipcc\RouterA node reporting process statistics for process ccag. 
  Process name:                      ccag
  Process status                     ACTIVE
  Process ID                         6c0
  Number of times process started    1
  Last start time:                   00:35:31  2/8/2007
  Pings completed in zero time:      0
  Pings completed in first third:    0
  Total first third milliseconds:    0
  Pings completed in second third:   0
  Total second third milliseconds:   0
  Pings completed in third third     0
  Total third third milliseconds:    0
  Longest Ping Time                  0

For a CiscoLog formatted message, each line will contain a separate header as follows

19: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.1/14]: ICM\ipcc\RouterA node reporting process statistics for process ccag. 
20: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.2/14]:   Process name:                      ccag
21: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.3/14]:   Process status                     ACTIVE
22: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.4/14]:   Process ID                         6c0
23: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.5/14]:   Number of times process started    1
24: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.6/14]:   Last start time:                   00:35:31  2/8/2007
25: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.7/14]:   Pings completed in zero time:      0
26: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.8/14]:   Pings completed in first third:    0
27: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.9/14]:   Total first third milliseconds:    0
28: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.10/14]:   Pings completed in second third:   0
29: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.11/14]:   Total second third milliseconds:   0
30: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.12/14]:   Pings completed in third third     0
31: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.13/14]:   Total third third milliseconds:    0
32: CICMRGRA: Feb  8 2007 05:36:09.890 +0000: %ICM_Router_unknown: [comp=Router-A][pname=nm][iid=ipcc][sev=info][part=19.14/14]:   Longest Ping Time                  0

To differentiate each line in the entry, the part tag, [part=%1.%2/%3], is added to each header where:

%1 = the sequence number of the first line. This is the same for all lines in the entry.

%2 = the part number of the specific line.

%3 = the total number of parts in the entry.

In the example at the line beginning with sequence number 30:

%1 = 19.

%2 = 12.

%3 = 14.


Cisco Contact Center Application MIB Results Example

The following example displays the data provided by the Contact Center Application MIB SNMP agent on the target ICM installation icm70 in response to a series of SNMP getnext requests beginning at node ciscoCcaMIB, OID 1.3.6.1.4.1.9.9.473.

A single icm instance, cccaInstanceName.2 = acme, has been installed with instance number 0.

The following icm components have been installed

  • Router
    cccaComponentName.instanceNumber(0).componentIndex(1) = RouterA
  • Logger
    cccaComponentName.instanceNumber(0).componentIndex(2) = LoggerA
  • PeripheralGateway
    cccaComponentName.instanceNumber(0).componentIndex(3) = PG1A
  • Distributed AW
    cccaComponentName.instanceNumber(0).componentIndex(4) = Distributor

A single CRSP NIC has been installed as part RouterA.

cccaNicType..instanceNumber(0).componentIndex(1).nicIndex(1) = crsp

A single IPCC Express PIM (acmiCRS) has been installed as part of PG1A.

    cccaPimPeripheralName.instanceNumber(0).componentIndex(3).
        cccaPimNumber(1) = ACD 1
cccaName.0 = ccbu-csa-rgr1a
cccaDescription.0 = Cisco Intelligent Contact Management / IP Contact Center
cccaVersion.0 = 7.1(1)
cccaTimeZoneName.0 = Eastern Standard Time
cccaTimeZoneOffsetHours.0 = 5
cccaTimeZoneOffsetMinutes.0 = 0
cccaSupportToolsURL.0 = 
cccaInstanceName.0 = acme
cccaComponentType.0.1 = router(1)
cccaComponentType.0.2 = logger(2)
cccaComponentType.0.3 = pg(4)
cccaComponentType.0.4 = distAW(3)
cccaComponentName.0.1 = RouterA
cccaComponentName.0.2 = LoggerA
cccaComponentName.0.3 = PG1A
cccaComponentName.0.4 = Distributor
cccaComponentStatus.0.1 = started(4)
cccaComponentStatus.0.2 = started(4)
cccaComponentStatus.0.3 = started(4)
cccaComponentStatus.0.4 = started(4)
cccaComponentElmtName.0.1.1 = ccagent
cccaComponentElmtName.0.1.2 = crspnic
cccaComponentElmtName.0.1.3 = dbagent
cccaComponentElmtName.0.1.4 = mdsproc
cccaComponentElmtName.0.1.5 = router
cccaComponentElmtName.0.1.6 = rtsvr
cccaComponentElmtName.0.1.7 = testsync
cccaComponentElmtName.0.2.8 = configlogger
cccaComponentElmtName.0.2.9 = csfs
cccaComponentElmtName.0.2.10 = histlogger
cccaComponentElmtName.0.2.11 = recovery
cccaComponentElmtName.0.3.12 = mdsproc
cccaComponentElmtName.0.3.13 = opc
cccaComponentElmtName.0.3.14 = pgagent
cccaComponentElmtName.0.3.15 = acmipim
cccaComponentElmtName.0.3.16 = testsync
cccaComponentElmtName.0.4.17 = configlogger
cccaComponentElmtName.0.4.18 = rtclient
cccaComponentElmtName.0.4.19 = rtdist
cccaComponentElmtName.0.4.20 = updateaw
cccaComponentElmtRunID.0.1.1 = 3336
cccaComponentElmtRunID.0.1.2 = 2992
cccaComponentElmtRunID.0.1.3 = 3600
cccaComponentElmtRunID.0.1.4 = 3920
cccaComponentElmtRunID.0.1.5 = 4040
cccaComponentElmtRunID.0.1.6 = 3532
cccaComponentElmtRunID.0.1.7 = 4100
cccaComponentElmtRunID.0.2.8 = 948
cccaComponentElmtRunID.0.2.9 = 3248
cccaComponentElmtRunID.0.2.10 = 1248
cccaComponentElmtRunID.0.2.11 = 3272
cccaComponentElmtRunID.0.3.12 = 4724
cccaComponentElmtRunID.0.3.13 = 4864
cccaComponentElmtRunID.0.3.14 = 4964
cccaComponentElmtRunID.0.3.15 = 5236
cccaComponentElmtRunID.0.3.16 = 5228
cccaComponentElmtRunID.0.4.17 = 5460
cccaComponentElmtRunID.0.4.18 = 5488
cccaComponentElmtRunID.0.4.19 = 5504
cccaComponentElmtRunID.0.4.20 = 5536
cccaComponentElmtStatus.0.1.1 = active(5)
cccaComponentElmtStatus.0.1.2 = started(4)
cccaComponentElmtStatus.0.1.3 = active(5)
cccaComponentElmtStatus.0.1.4 = active(5)
cccaComponentElmtStatus.0.1.5 = active(5)
cccaComponentElmtStatus.0.1.6 = active(5)
cccaComponentElmtStatus.0.1.7 = active(5)
cccaComponentElmtStatus.0.2.8 = active(5)
cccaComponentElmtStatus.0.2.9 = active(5)
cccaComponentElmtStatus.0.2.10 = active(5)
cccaComponentElmtStatus.0.2.11 = active(5)
cccaComponentElmtStatus.0.3.12 = active(5)
cccaComponentElmtStatus.0.3.13 = active(5)
cccaComponentElmtStatus.0.3.14 = active(5)
cccaComponentElmtStatus.0.3.15 = standby(6)
cccaComponentElmtStatus.0.3.16 = active(5)
cccaComponentElmtStatus.0.4.17 = active(5)
cccaComponentElmtStatus.0.4.18 = active(5)
cccaComponentElmtStatus.0.4.19 = active(5)
cccaComponentElmtStatus.0.4.20 = active(5)
cccaRouterSide.0.1 = sideA(1)
cccaRouterCallsPerSec.0.1 = 0
cccaRouterAgentsLoggedOn.0.1 = 0
cccaRouterCallsInProgress.0.1 = 0
cccaRouterDuplexPairName.0.1 = ccbu-csa-rgr1a
cccaRouterNicCount.0.1 = 1
cccaNicType.0.1.1 = crsp(5)
cccaNicStatus.0.1.1 = started(4)
cccaLoggerSide.0.2 = sideA(1)
cccaLoggerType.0.2 = standard(1)
cccaLoggerRouterSideAName.0.2 = ccbu-csa-rgr1a
cccaLoggerRouterSideBName.0.2 = ccbu-csa-rgr1a
cccaLoggerDuplexPairName.0.2 = ccbu-csa-rgr1a
cccaLoggerHDSReplication.0.2 = 0
cccaDistAwSide.0.4 = sideA(1)
cccaDistAwType.0.4 = standard(0)
cccaDistAwAdminSiteName.0.4 = ccbu-csa-rgr1a
cccaDistAwRouterSideAName.0.4 = ccbu-csa-rgr1a
cccaDistAwRouterSideBName.0.4 = ccbu-csa-rgr1a
cccaDistAwLoggerSideAName.0.4 = ccbu-csa-rgr1a
cccaDistAwLoggerSideBName.0.4 = ccbu-csa-rgr1a
cccaDistAwDuplexPairName.0.4 = ccbu-csa-rgr1a
cccaDistAwHDSEnabled.0.4 = 0
cccaDistAwWebViewEnabled.0.4 = false(2)
cccaDistAwWebViewServerName.0.4 = 
cccaPgNumber.0.3 = 1
cccaPgSide.0.3 = sideA(1)
cccaPgRouterSideAName.0.3 = ccbu-csa-rgr1a
cccaPgRouterSideBName.0.3 = ccbu-csa-rgr1a
cccaPgDuplexPairName.0.3 = ccbu-csa-rgr1a
cccaPgPimCount.0.3 = 1
cccaPimPeripheralName.0.3.1 = ACD 1
cccaPimPeripheralType.0.3.1 = acmiCRS(19)
cccaPimStatus.0.3.1 = started(4)
cccaPimPeripheralHostName.0.3.1 = LabHost

Rating: 0.0/5 (0 votes cast)

Personal tools