Cisco Unified Communications -- Poor Voice Quality
This information applies to all Cisco Unified Communications System releases.
Nearly all voice quality problems can be attributed to some kind of degradation on the IP network that the voice traffic traverses. Network problems that might not be noticeable for normal data traffic are very apparent in a voice conversation because of the need to minimize packet loss and variable delay in an IP telephony network. A variety of issues can result in poor voice quality:
- Packet drops
- Queuing problems
In addition to the information in this article, refer to Troubleshooting QOS Choppy Voice Issues (requires Cisco.com login) for additional techniques on resolving voice quality issues.
IP telephony demands that voice packets reach their destination within a predictable amount of time and without being dropped somewhere along the path from the source to the destination. In a properly designed network with appropriate QoS provisioning in place, packet loss should be near zero. All voice codecs can tolerate some degree of packet loss without dramatically affecting voice quality. Upon detecting a missing packet, the codec decoder on the receiving device makes a best guess as to what the waveform during the missing period of time should have been. Most codecs can tolerate up to five percent random packet loss without noticeable voice quality degradation. This assumes that the five percent of packets being lost are not being lost at the same time, but rather are randomly dropped in groups of one or two packets. Losing multiple simultaneous packets, even as a low percentage of total packets, can cause noticeable voice quality problems.
- Note: You should design your network for zero packet loss for packets that are tagged as voice packets. A converged voice/data network should be engineered to ensure that only a specific number of calls are allowed over a limited-bandwidth link. You should guarantee the bandwidth for those calls by giving priority treatment to voice traffic over all other traffic. For more information on prioritizing voice over data, refer to Voice Quality information available on Cisco.com.
There are various tools that you can use to determine whether you are experiencing packet loss in your network and where in the network the packets are getting dropped. The starting point to look for lost packets is the call statistics screen on Cisco Unified IP Phones.
- 1. Do one of the following:
- If you are troubleshooting at the phone experiencing the problem, access these statistics by pressing the help (i or ?) button on the IP phone twice in quick succession during an active call.
- If you are working with a remote user, open a web browser on your computer and enter the IP address of the user's phone. During an active call, choose the Streaming Statistics > Stream 1 options from the display.
- 2. Examine the counters RxDisc and RxLost shown on the IP phone (or Rcvr Lost Packets if you are viewing the statistics remotely using a web browser).
- RxLost measures the number of packets that were never received because they were dropped in the network somewhere. By detecting a missing RTP sequence number, the IP phone can determine that a packet has been lost.
- RxDisc corresponds to packets that were received but were discarded because they could not be used at the time they arrived. RxDisc can come from an out-of-order packet or a packet that arrived too late.
- 3. If either of these two counters increments, you should investigate to learn why packets are being lost or discarded.
Regardless of how low your packet loss is, if it is not zero, you should investigate the root cause because it might be a sign of a bigger problem that will get worse with higher call volume. Also, although small packet loss might not be perceptible in a conversation between two people, it can be detrimental to fax and modem transmissions. The packet loss can be occurring at any layer of the OSI model, so be sure to check for all possibilities for each hop. For example, if there is a Frame Relay connection over a T1 between two sites, you should:
- Make certain that there are no errors at the physical layer on the T1.
- Determine if you are exceeding your committed information rate (CIR) on the Frame Relay connection.
- Verify that you are not dropping the packets at the IP layer because you are exceeding your buffer sizes.
- Check that you have your QoS improperly configured.
- Ensure that your service provider not only guarantees packet delivery but also guarantees a low-jitter link. Some service providers may tell you that they do not provide a CIR but guarantee that they will not drop any packets. In a voice environment, delay is as important as packet loss. Many service providers' switches can buffer a large amount of data, thereby causing a large amount of jitter.
One common cause of drops in an Ethernet environment is a duplex mismatch, when one side of a connection is set to full duplex and the other side is set to t half duplex. To determine if this is the case, perform the following steps:
- Check all the switch ports through which a given call must travel and ensure that there are no alignment or frame check sequence (FCS) errors. Poor cabling or connectors can also contribute to such errors; however, duplex mismatches are a far more common cause of this kind of problem.
- Examine each link between the two endpoints that are experiencing packet loss and verify that the speed and duplex settings match on either side.
Although duplex mismatches are responsible for a large number of packet loss problems, there are many other opportunities for packet loss in other places in the network as well. When voice traffic must traverse a WAN, there are several places to look. First, check each interface between the two endpoints, and look for packet loss. On all Cisco IOS software platforms, you can find this information using the show interface command. If you are seeing dropped packets on any interface, there is a good chance that you are oversubscribing the link. This could also be indicative of some other traffic that you are not expecting on your network. The best solution in this case is to take a sniffer trace to examine which traffic is congesting the link.
Sniffers are invaluable in troubleshooting voice quality problems. With a sniffer, you can examine each packet in an RTP stream to see if packets are really being lost and where in the network they are being lost. To troubleshoot using a sniffer, perform the following steps:
- Start at the endpoint that is experiencing the poor-quality audio where you suspect packet loss.
- Take a sniffer trace of a poor-quality call and filter it so that it shows you only packets from the far end to the endpoint that is hearing the problem. The packets should be equally spaced, and the sequence numbers should be consecutive with no gaps.
- If you are seeing all the packets in the sniffer trace, continue taking traces after each hop until you get a trace where packets are missing.
- When you have isolated the point in the network where the packet loss is occurring, look for any counters on that device that might indicate where the packets are being lost.
Queuing delay can be a significant contributor to variable delay (jitter). When you have too much jitter end-to-end, you encounter voice quality problems. A voice sample that is delayed over the size of the receiving device's jitter buffer is no better than a packet that is dropped in the network because the delay still causes a noticeable break in the audio stream. In fact, high jitter is actually worse than a small amount of packet loss because most codecs can compensate for small amounts of packet loss. The only way to compensate for high jitter is to make the jitter buffer larger, but as the jitter buffer gets larger, the voice stream is delayed longer in the jitter buffer. If the jitter buffer gets large enough such that the end-to-end delay is more than 200 ms, the two parties on the conference feel like the conversation is not interactive and start talking over each other.
Remember that every network device between the two endpoints involved in a call (switches, routers, firewalls, and so on) is a potential source of queuing or buffering delays. The ideal way to troubleshoot a problem in which the symptoms point to delayed or jittered packets is to use a sniffer trace at each network hop to see where the delay or jitter is being introduced.
For more information on jitter, refer to Understanding Jitter in Packet Voice Networks (requires Cisco.com login).
Have you had experience troubleshooting Cisco Unified Communications equipment? Can you expand on this topic? Please contribute to this wiki and share the tools and techniques you used by sending E-mail to firstname.lastname@example.org. Guidelines for submitting content are available on the About DocWiki pages.