Cisco IOS Voice Troubleshooting and Monitoring -- Voice Performance Statistics on Cisco Gateways
The Voice Performance Statistics on Cisco Gateways feature enables the collection of voice call signaling statistics and VoIP AAA accounting statistics based on user-configured time ranges. The statistics can be displayed on your console or formatted and archived to an FTP or syslog server. This feature can assist you in diagnosing performance problems on the network and identifying impaired voice equipment.
Prerequisites for Voice Performance Statistics on Cisco Gateways
- Your gateway must be configured to support VoIP and must be functioning properly.
Restrictions for Voice Performance Statistics on Cisco Gateways
- This feature does not support Media Gateway Control Protocol (MGCP).
- This feature does not support parsing, presentation, or analysis of syslog files.
- The integrity of statistical information in the syslog files is not guaranteed because of unreliable User Datagram Protocol (UDP) transport.
- If the gateway clock needs to be synchronized after the gateway is reset or rebooted or during Network Time Protocol (NTP) client synchronization, there may be a problem for collection during the collection intervals.
- The call-statistics data is dependent on the start and end time of the interval; that is, the collection is time driven, not event driven. The following two situations will result in erroneous call-statistics data:
- Clock reset during a collection interval.
- Clock set to a time during an already specified interval.
- To avoid any call-statistics data from being collected within an incomplete interval, this feature reports only call-statistics data that is collected during a complete interval. This includes call-statistics data that is pushed to an FTP server or stored on a gateway.
- This feature cannot assure accuracy or consistency of the reports generated when large clock updates occur during a batch reporting period.
- Call statistics cannot be accessed using RADIUS protocols.
- The signaling behavior of two-stage non-Direct-Inward-Dialing (non-DID) ISDN calls using a default session application is not supported.
- Digital Signal 0 (DS0) is not supported.
Information About Voice Performance Statistics on Cisco Gateways
To configure the Voice Performance Statistics on Cisco Gateways feature, you should understand the following:
- Basic Terminology and Feature Design
- Management of the Statistical Collection
- Management of the Archive Process
- Display of Records and Time Ranges
- Voice Interface Changes During Call-Statistics Collection Periods
The benefits of this feature are listed in the Benefits of Voice Performance Statistics on Cisco Gateways.
Basic Terminology and Feature Design
The Voice Performance Statistics on Cisco Gateways feature is designed to include many options for collecting, archiving, and displaying call statistics records, which include PSTN interface/port statistics, IP interface statistics, and counts of RADIUS accounting messages. This feature also enables counts of Cisco IOS generated internal error codes (IECs). The basic terminology that describes the functionality of the feature is explained in the subsections that follow.
The following are feature options:
- Counting accounting records (messages to and from RADIUS servers).
- Collecting various signal-layer (IP and PSTN interfaces) statistics from individual gateway ports.
- Displaying the signaling statistics at different aggregation levels.
- Collecting IECs.
- Collecting the statistics at user-configured time intervals.
- Archiving the statistics to an FTP or syslog server and formatting the output.
- Displaying the statistics on a console.
- Displaying the available memory and the memory that has been used for the collection of records.
- Specifying thresholds for packet jitter, lost packets, and packet latency.
- Specifying the length of time to be used as the maximum call duration.
- Specifying a maximum time for which to store the statistics in system memory.
This section has the following subsections:
- What Are the Types of Accounting Statistics?
- What Are the Types of Signaling Statistics and Aggregation Levels?
- What Are IECs?
What Are the Types of Accounting Statistics?
Accounting record statistics are counts of RADIUS messages that are transmitted to and received from a RADIUS server. They can be collected by method list, type of RADIUS message (for example, starts, interim updates, and/or stops), and call-leg interface association (PSTN or IP). This feature can be configured to count all of these message types or only a subset of them.
A method list is a sequential list used by the RADIUS client on the gateway that defines the authentication methods used to authenticate a user. For the purposes of the Voice Performance Statistics on Cisco Gateways feature, you are required to specify only the name of the method list on the gateway.
Once enabled and configured, the feature counts RADIUS messages on both inbound and outbound call legs. Each time a RADIUS accounting message is received by the gateway, it is counted as successful if it is accepted and processed by the RADIUS agent on the gateway; each time a RADIUS accounting message is transmitted by the gateway, it is counted as passed if an ACK comes back from the RADIUS server.
You can also specify that accounting messages be collected from a broadcast method list, in which case you can set all the server groups that are in a method list to monitor the server group acknowledgements.
What Are the Types of Signaling Statistics and Aggregation Levels?
The signaling statistics are collected at the port level, but can be displayed at various aggregation levels.
The aggregation levels are hierarchical. The highest level is a summary of total statistics for all aggregation levels on the gateway, whereas the lowest level provides statistics for each voice port. Statistics can also be collected for the following aggregation levels:
- Gateway level
- VoIP level
- PSTN level
- Trunk group level
- Voice-port level
An example of collected statistics at the different aggregation levels for a PSTN statistic labeled "X" is as follows:
- When the aggregation level is gateway: X = 4
- When the aggregation level is trunk group:
- Trunk group A (configured ports 1 and 3): X = 3
- Trunk group B (configured ports 2 and 4): X = 1
- When the aggregation level is port:
- Port 1: X = 1
- Port 2: X = 0
- Port 3: X = 2
- Port 4: X = 1
The following are supported call-statistics fields that can be collected on Cisco gateways:
- Incoming calls-All incoming call attempts, whether successful or not.
- Incoming calls answered by the gateway-Incoming calls that were answered.
- Incoming calls rejected by the gateway-Incoming calls that, for whatever reason, failed.
- Outgoing calls attempted-Outgoing calls regardless of whether they were successful.
- Outgoing calls that receive answers-Calls that were answered.
- Outgoing calls fail-Calls that failed.
- Total duration of all incoming and outgoing calls-Total duration from outgoing seizure to disconnect.
- Total duration of incoming and outgoing answered calls-Total connected time: from answer to disconnect.
- Originating side disconnected before outgoing calls connected.
- Number of incoming and outgoing calls whose connected time is less than the configured minimum call duration (MCD).
- Number of answered incoming and outgoing calls terminated with any cause codes other than "normal."
- Total duration (after the dial delay) on incoming calls-Defined as "alert sent time-setup in time."
- Total duration (after the dial delay) on outgoing calls-Defined as "alert received time-setup out time."
- Total setup delay duration-Defined as "setup out time-setup in time."
- IP-specific statistic fields (exist only in the IP-level statistics):
- Number of calls losing more than the configured number of packets-The default is 1000.
- Number of calls encountering more than the configured amount of latency-The default is 250 milliseconds.
- Number of calls encountering more than the configured amount of jitter-The default is 250 milliseconds.
- Number of incoming and outgoing calls disconnected with each cause code-The cause codes are defined in the Call Control Application Programming Interface (CCAPI) and in the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) standard Q.850.
What Are IECs?
Cisco IOS generated internal error codes (IECs) are gateway-detected errors that cause the gateway to release or refuse a call. IECs enhance troubleshooting for VoIP networks by helping to determine the source and reason for call termination.
The collection of Cisco IOS generated IECs is described in Cisco VoIP Internal Error Codes. To decode internal error codes, see the Voice Gateway Error Decoder for Cisco IOS at the following URL:
Management of the Statistical Collection
The management of statistical collection involves time ranges, thresholds, storage capacities, and memory usage. This section has the following subsections:
- What Are the Allowable Time Ranges?
- What Are Thresholds?
- What Are the Allowable Storage Capacities?
- How Is Memory Used?
What Are the Allowable Time Ranges?
The Voice Performance Statistics on Cisco Gateways feature enables you to configure time ranges to capture statistics. The time ranges are as follows:
- From the last reset time to the present. You can examine the statistics using the show voice statistics command and reset the statistics using the clear voice statistics csr command.
- By specific start and end time. That is, a set amount of minutes after the configuration time for preparation of the resource allocation of the gateway.
- By periodic intervals, with an optional total duration. Allowable intervals are 5 minutes, 15 minutes, 30 minutes, 1 hour, and 1 day. The optional total duration is unlimited but must be a multiple of the specified interval. For example, the interval could be 15 minutes, and the total duration could be 3 hours.
What Are Thresholds?
The following three thresholds can be configured to customize how you measure the voice signaling statistics for your network configuration:
- Packet jitter
- Packet latency
- Lost packets
These thresholds are all pre-configured with default settings. The jitter, latency, and lost packets thresholds only apply to IP statistics. In addition, you can configure the minimum call duration (MCD) value for determining which calls are measured during the statistics collection.
Jitter is a variation in the delay of received packets. At the sending side, packets are sent in a continuous stream, spaced evenly apart. Because of network congestion, improper queueing, or configuration errors, this steady stream can be interrupted by delays between packets.
You can specify the threshold at which a record will not be collected. For example, if you have set a threshold of 250 milliseconds and a delay exceeds that threshold, the message is not collected.
Packet latency is the amount of time that it takes a packet to go from its source to its destination. You can specify the threshold at which a record will not be collected. For example, the gateway can be configured to drop messages that take more than 250 milliseconds to reach the destination.
Lost packets are a result of jitter that is so great that it causes packets to be out of the range of the jitter buffer. These packets are discarded. You can specify the threshold for lost packets in milliseconds.
Minimum Call Duration
Using the minimum call duration (MCD) value, you can configure the gateway to collect statistics for calls that last a minimum amount of time. For example, if you configure the MCD value to 2 milliseconds, the gateway counts the number of incoming or outgoing calls with a connect time less than 2 milliseconds.
What Are the Allowable Storage Capacities?
The Voice Performance Statistics on Cisco Gateways feature allows you to specify how long the gateway will store the statistics. You can specify a number of days, hours, or minutes.
How Is Memory Used?
You can display how much system memory is being used when statistics are being collected, how much system memory is available, and how much system memory is estimated for future use by using the show voice statistics memory-usage command. The command displays all memory usage for accounting and signaling by fixed interval and since a system reset or reboot. The output includes the number of call statistics records per interval, each record size, the approximate memory used, and the allocated memory for future use.
Management of the Archive Process
The FTP or syslog server archive process is used to download statistics to a file on the server for further processing, presentation, and analysis. The download to the server is configured using the command-line interface (CLI). When the specified end time of collection is reached, the gateway downloads the statistics data to the provisioned server. Using FTP, the gateway formats the statistics in an ASCII file and transfers the file to a Cisco Networking Services Performance Engine (CNS-PE). Using the syslog server, gateways send the information in syslog messages, and you can specify the message size.
|Note:||Because of unreliable UDP transport, the integrity and completeness of the call statistics in the syslog files are not guaranteed.|
Figure: Syslog and FTP Servers and the CNS-PE shows the components as used by this feature.
Figure: Syslog and FTP Servers and the CNS-PE
You can format the output to display with specified record separators. The separator can be a space, tab, new line, or ASCII character.
Display of Records and Time Ranges
You can display statistics by accounting method lists, by aggregation levels, by numbered intervals (time ranges), and/or since the last system reset or reboot. This section has the following subsections:
What Records Are Displayed Since System Reset or Reboot?
You can display the collected records since a system reset or reboot including accounting and/or signaling (aggregation level), and intervals (time ranges). There are show commands to display all signaling statistics that were collected since the last reset or reboot of the system. You can specify a concise or verbose display, and you can configure the gateway to push the statistics to the FTP or syslog server.
Displaying Accounting Statistics
You can use the show voice statistics csr since-reset accounting command to display the method list by RADIUS server, PSTN incoming and outgoing records that have passed and failed, and the IP incoming and outgoing records that have passed and failed.
Displaying Aggregation-Level Statistics
You can use the show voice statistics csr since-reset aggregation-level command to display all the collected statistics for every aggregation level or just the statistics for a specific level (gateway, IP, PSTN, trunk group, or voice port).
What Time Ranges Are Displayed?
You can use the show voice statistics interval-tag command to display the start and end times by a numbered interval. To display the time ranges by voice port, you can use the show voice statistics csr interval aggregation command.
Voice Interface Changes During Call-Statistics Collection Periods
It is recommended that you do not change the voice interface configuration during the period when call statistics are being collected. It is also recommended that during the period when call statistics are being collected, the following configuration changes not take place:
- Adding or removing a PRI or DS1 group
- Adding or removing a trunk group
- Adding or removing a trunk in a trunk group
|Note:||It is recommended that any existing call-statistics collection be stopped and set to zero before any configuration modification is made to any of the voice interfaces.|
This section has the following subsections:
Addition or Removal of a Voice Port
The following scenarios apply to a voice port whether or not it is in any trunk group:
- The addition of a PRI or DS1 group, Foreign Exchange Station (FXS), Foreign Exchange Office (FXO), or ear and mouth (E&M) device.
- The removal of a PRI or DS1 group, FXS, FXO, or E&M device.
If a new voice interface is added during any collection period, new entries that correspond to the new voice interface are added in the statistics collected for that collection period.
If an existing voice interface is removed during any collection period, the statistics that correspond to that voice interface are still kept in the set. The statistics are frozen (that is, nothing more is added) after the time that the voice interface is removed. Gateways still send the statistics for the removed interface to the CNS-PE or syslog server.
In either of the above scenarios, the statistics for PSTN ports include all the collected data at the DS1 or channel associated signaling (CAS) level, except the statistics for interfaces added or removed during the collection period.
Configuration Change of Any Trunk Group
The following scenarios apply when there is a configuration change of any trunk group:
- The addition or removal of a DS1 or CAS group into or from a trunk group.
- If you are adding a DS1 or CAS group to a trunk group during any collection period, the collection process moves the associated statistics from the upper aggregation level (PSTN) to the trunk-group level. The call statistics before the configuration change time are also totalled to the statistics of the trunk-group level at the end of the collection period. In this case, the statistics of the trunk-group level can exceed the limit. However, the PSTN-level statistic is still accurate.
- If you are removing a DS1 or CAS group from a trunk group during any collection period, the collection process removes the trunk-group level of the PSTN aggregation. In contrast to the scenario of adding a DS1 or CAS group, the statistic of the trunk-group level is under the limit.
- The addition or removal of a trunk group.
- If you are adding a new trunk group (one that contains a trunk or trunks) to a gateway during any collection period, the existing statistics of all member trunks aggregate to the trunk-group level statistic. That is, the statistic of the trunk-group level is over its limit. The PSTN-level call statistics are still accurate.
- If you are removing an existing trunk group from a gateway during any collection period, the existing statistics of all member trunks are totaled at the configuration change time. The statistics of the trunk-group level and of the PSTN level are accurate.
|Note:||The inaccurate call statistics that can result from the four scenarios that are listed above are acceptable because the transient information during the configuration change is often unusable.|
Benefits of Voice Performance Statistics on Cisco Gateways
- Gateway call statistics can be audited or compared against the statistics of other network devices for improved monitoring.
- Malfunctioning DSPs (DS1 only) can be discovered.
- Discrepancies between RADIUS records sent by the gateway and received and reported on the server can be uncovered.
- Potential lost revenue can be highlighted.
- Call-success rates and accuracy of reports can be determined.