Cisco IOS Voice Troubleshooting and Monitoring -- Common QoS Problems

From DocWiki

Latest revision as of 17:09, 16 October 2012

Quality on Voice Ports

Table: Troubleshooting Voice Quality on Voice Ports lists some QoS problems you might encounter after configuring voice ports and suggests some remedies. To configure QoS features on voice ports, see the "Fine-Tuning Analog and Digital Voice Ports" section in the Cisco IOS Voice Port Configuration Guide.

Table: Troubleshooting Voice Quality on Voice Ports

Symptom	Suggested Action
Telephony device buzzes or does not ring.	Use the show voice port command to confirm that ring frequency is configured correctly. It must match the connected telephony equipment and might be country-dependent.
Distorted speech.	Use the show voice port command to confirm that the cptone keyword setting (also called region tone) is set to us for the United States. Note: Having a wrong cptone setting can cause faulty voice reproduction during analog-to-digital or digital-to-analog conversions.
Music on Hold (MOH) is not heard.	Reduce the music-threshold level to make VAD less sensitive with the music-threshold voice-port configuration command. Valid entries are from -70 to -30.
Background noise is not heard.	Enable the comfort-noise command.
Long pauses occur in conversation, as if you were speaking on a walkie-talkie.	Overall delay is probably excessive; the standard for adequate voice quality is 150 ms one-way transit delay. Measure delay by using ping tests at various times of the day with different network traffic loads. If delay must be reduced, examine propagation delay of signals between the sending and receiving endpoints, voice encoding delay, and the voice packetization time for various VoIP codecs.
Jerky or choppy speech.	Variable delay, or jitter, is being introduced by congestion in the packet network. There are two possible remedies: Reduce the amount of congestion in your packet network. Pings between VoIP endpoints provide information about the round-trip delay of a link, which should never exceed 300 ms. Network queuing and dropped packets should also be examined. Increase the size of the jitter buffer with the playout-delay command. (See the Jitter Adjustment for more information.)
Clipped or fuzzy speech.	Reduce input gain. (See the Voice Level Adjustment for more information.) Change the VAD level. Sometimes VAD cuts the sound too early and the speaker's voice is clipped. You can also change the time period that VAD waits for silence.
Clipped speech.	Reduce the input level at the listener's router. (See the Voice Level Adjustment for more information)
Volume too low or missed DTMF.	Increase speaker's output level or listener's input level. (See the Voice Level Adjustment for more information.)
Echo interval is greater than 25 ms (sounds like a separate voice).	Configure the echo-cancel enable command and increase the value for the echo-cancel coverage keyword. (See the Echo Adjustment for more information.)
Too much echo.	Reduce the output level at the speaker's voice port. (See the Voice Level Adjustment for more information.)

Delay in Voice Networks

Delay is the time it takes for voice packets to travel between two endpoints. Excessive delay can cause quality problems with real-time traffic such as voice. However, because of the speed of network links and the processing power of intermediate devices, some delay is expected.

When listening to speech, the human ear normally accepts up to about 150 ms of delay without noticing it. The ITU G.114 standard recommends no more than 150 ms of one-way delay for a normal voice conversation. Once the delay exceeds 150 ms, a conversation is more like a "walkie-talkie" conversation in which one person must wait for the other to stop speaking before beginning to talk.

Several different types of delay combine to make up the total end-to-end delay associated with voice calls:

• Coder delay-Amount of time used by the digital signal processor (DSP) to compress samples.
• Handling delay-Amount of time it takes to process data (adding headers, taking samples, forming packets, and so forth).
• Serialization delay-Fixed delay related to the clock rate on the interface. The amount of delay needed for different frame sizes is shown in Table: Serialization Delay.

Table: Serialization Delay

• Propagation delay-Amount of time it takes the data to physically travel over the media.
• Queuing delay-Amount of time lost due to congestion. Every network device between two endpoints involved in a call is a potential source of queuing or buffering delays. Use a sniffer trace at each hop to see where the delay or jitter is being introduced.

Output drops can occur because of incorrectly configured priority queuing. For information about troubleshooting output drops with priority queueing, refer to Troubleshooting Output Drops with Priority Queueing , document ID 10105.

• Variable delay or jitter-Amount of time that causes the conversation to break and become unintelligible. Jitter is described in the Jitter Adjustment.

You can measure delay by using ping tests at various times of the day with different network traffic loads. Sniffer traces can also be used to find where delay is being introduced in the network. For more information about measuring QoS, see the Measuring QoS.

The recommended ranges for delay are shown in Table: Recommended Delay Ranges. These ranges are for connections in which echo is controlled. Echo cancellers are required when one-way delay exceeds 25milliseconds. For more information about echo cancellation, see the Echo Adjustment. These ranges are for total end-to-end delay, not just the delay on the WAN side of a connection. For more information about developing a delay budget, refer to the "Building the Delay Budget" section of the Understanding Delay in Packet Voice Networks white paper.

Table: Recommended Delay Ranges

Packet Loss

Packet loss is a very common cause of voice quality problems on an IP network. In a properly designed network, packet loss should be near zero. Voice codecs can tolerate some degree of packet loss without a dramatic effect on voice quality. Voice packets should be tagged for zero packet loss. On converged voice/data networks, the quantity of the voice calls and link bandwidth should be limited. The bandwidth should be guarenteed for the voice calls by giving priority to voice traffic. Packet loss can have an impact voice quality in several ways:

• Packet loss can grow as call volume increases
• Fax and modem transmissons are greatly affected by packet loss

A common cause of packet loss is duplex mismatching. This occurs when one side of the connection is set up for full duplex and the other is set up for half duplex. Look at the link between the two endpoints experiencing the packet loss and check the speed and duplex settings for each side.

Although duplex mismatch is common, there are many other opportunities for packet loss in the network. To find the cause of these packet drops, check each interface between two endpoints by using the show interface command. If dropped packets are showing up on any interface, the link might be oversubscribed or there might be other traffic on this link that you are not expecting. In this case, use a network analyzer (or "sniffer") to check what traffic might be congesting the link. For more information about network analyzers, see the Network Analyzers.

Jitter Adjustment

Delay can cause unnatural starting and stopping of conversations, but variable-length delays (also known as jitter) can cause a conversation to break and become unintelligible. Jitter is not usually a problem with PSTN calls because the bandwidth of calls is fixed and each call has a dedicated circuit for the duration of the call. However, in VoIP networks, data traffic might be bursty, and jitter from the packet network can become an issue. Packets from the same conversation can arrive at different interpacket intervals, especially during times of network congestion, which can disrupt the steady, even delivery needed for voice calls. Cisco voice gateways have built-in jitter buffering to compensate for a certain amount of jitter; the playout-delay command can be used to adjust the jitter buffer.

Normally, the defaults in effect are sufficient for most networks. However, a small playout delay from the jitter buffer can cause lost packets and choppy audio, and a large playout delay can cause unacceptably high overall end-to-end delay.

Note:

For Cisco IOS Release 12.1(5)T and later, in most cases playout delay should be configured in dial-peer configuration mode on the VoIP dial peer that is on the receiving end of the voice traffic that is to be buffered. This dial peer senses network conditions and relays them to the DSPs, which adjust the jitter buffer as necessary. When multiple applications are configured on the gateway, playout delay should be configured in dial-peer configuration mode. When there are numerous dial peers to configure, it might be simpler to configure playout delay on a voice port. If there are conflicting playout delay configurations on a voice port and also on a dial peer, the dial-peer configuration takes precedence.

Jitter is more likely to occur on low-speed links because even a single packet in the queue on a low-speed link can dramatically affect the amount of time a voice packet needs to wait in the queue before being transmitted. Jitter can be an even bigger problem if you do not have priority queuing, such as low latency queuing (LLQ), enabled or configured correctly on your WAN connections. For more information about LLQ, see the Cisco IOS Quality of Service Solutions Configuration Guide. For information about troubleshooting VoIP over Point-to-Point Protocol (PPP) links with QoS for low latency queueing, refer to VoIP over PPP Links with Quality of Service (LLQ / IP RTP Priority, LFI, cRTP), document ID 7111.

Low-speed links also require special considerations when data traffic is also present to ensure that a large data packet does not cause excessive jitter. Generally on WAN links that are 768 kbps or slower, you should use some form of fragmentation and interleaving to ensure that large data packets do not starve smaller voice packets. Even with LLQ enabled, the voice packet must wait if it arrives when a data packet is in the process of being transmitted.

To configure the playout delay jitter buffer, perform the following steps.

SUMMARY STEPS

1. enable
2. configure terminal
3. 'voice-port slot/port:ds0-group-number
4. playout-delay mode {adaptive | fixed}
5. playout-delay {nominal value | maximum value | minimum {default | low | high}}
6. exit

	Command	Purpose
1.	enable Example: Router> enable	Enables higher privilege levels, such as privileged EXEC mode. Enter your password if prompted.
2.	configure {terminal | memory | network} Example: Router# configure terminal	Enters global configuration mode.
3.	voice-port slot/port:ds0-group-number Example: Router(config)# voice-port 1/0:0	Enters voice-port configuration mode on the selected slot, port, and DS-0 group. Each defined DS-0 group number is represented on a separate voice port. This allows you to define individual DS-0s on the digital T1/E1 card.
4.	playout-delay mode {adaptive | fixed} Example: Router(config-voiceport)# playout-delay mode adaptive	Determines the mode in which the jitter buffer operates for calls on this voice port. adaptive-Adjusts the jitter buffer size and amount of playout delay during a call based on current network conditions. This is the default. Note: Cisco recommends using the adaptive mode playout buffer as a place to begin checking QoS issues. Changing the playout buffers is an advanced technique that should be done last after all other QOS methods have been explored. fixed-Defines the jitter buffer size as fixed so that the playout delay is not adjusted during a call. A constant playout delay is added.
5.	playout-delay {nominal value | maximum value | minimum {default | low | high}} Example: Router(config-voiceport)# playout-delay nominal 0	Tunes the playout buffer to accommodate packet jitter caused by switches in the WAN. The keywords and arguments are as follows: nominal-Defines the amount of playout delay applied at the beginning of a call by the jitter buffer in the gateway. In fixed mode, this is also the maximum size of the jitter buffer throughout the call. value-Specifies the range which depends on the type of DSP and configured codec complexity. For medium codec complexity, the range is from 0 to 150 ms. For high codec complexity and DSPs that do not support codec complexity, the range is from 0 to 250 ms. maximum (adaptive mode only)-Specifies the jitter buffer's upper limit (80 ms), or the highest value to which the adaptive delay is set. minimum (adaptive mode only)-Specifies the jitter buffer's lower limit (10 ms), or the lowest value to which the adaptive delay is set.* default-Specifies 40 ms.
6.	exit Example: Router(config-voiceport)# exit	Exits voice-port configuration mode.

For more information about jitter, refer to Understanding Jitter in Packet Voice Networks (Cisco IOS Platforms), document ID 18902.

Echo Adjustment

Echo is a common problem, but can sometimes be difficult to troubleshoot. Echo exists in almost any telephone network but problems are more apparent in packet networks. Echo is perceived as a problem when all of the following conditions are true:

• Signal leakage between analog transmit (Tx) and receive (Rx) paths. This leakage is in one of these forms of echo:
  • Acoustic echo-Acoustic echo is caused by poor acoustic isolation between the earpiece and the microphone in handsets and hands-free devices.
  • Hybrid echo-Hybrid echo is caused by an impedance mismatch in the hybrid circuit, such as a two-wire to four-wire interface. This mismatch causes the Tx signal to appear on the Rx signal.
• Perceptible delay between when the originating signal is transmitted and when the echo returns. In most telephones, sidetone helps mask some of the echo. Echos must be delayed by at least 20 milliseconds to be perceived.
• Sufficient echo amplitude. If the amplitude of the echo is low, it can go unnoticed.

The packet segment of the voice connection introduces a significant delay, typically 30 ms in each direction. The introduction of delay causes echoes generated on analog tail circuits that were normally indistinguishable from the side tone to be perceived by the user. The delay introduced by packet voice is unavoidable, therefore, the voice gateways must prevent the echo.

For a description about echo cancellation features and how echo cancellation is implemented on different Cisco platforms and Cisco IOS software releases, refer to the Cisco IOS Voice Port Configuration Guide.

To configure echo cancellation, refer to the "How to Configure the Extended G.168 Echo Canceller" section in the Cisco IOS Voice Port Configuration Guide.

When troubleshooting echo problems, see the following:

• Determine Where Echo Is Occurring
• Troubleshooting Echo in Cisco IOS Gateways
• Measuring Echo in Cisco IOS Gateways

Determine Where Echo Is Occurring

When troubleshooting echo, first determine who is being affected by the echo problems.

PSTN Phone Users Hears Echo

In this case, a PSTN phone user hears echo, which is caused by acoustic coupling between the earpiece and the microphone in the IP phone handset. The solution is to use a load ID on the IP phone, which includes echo suppression on the handset and headset. Available load IDs only include echo cancellation on the speaker phone.

IP Phone User Hears Echo

In this case, an IP phone user hears echo caused by hybrids in a PSTN network. The solution is to configure and verify echo cancellation operation on a Cisco IOS gateway. The echo canceller in the voice gateway cancels the echo heard by the IP phone user.

Troubleshooting Echo in Cisco IOS Gateways

It is important to make sure that the echo canceller has enough information to distinguish between echo and voice conversation. The following parameters control the distinction:

• Input level-Signal input gain is performed before the echo canceller has detected the echo.
• Output level-Signal output attenuation is performed after the echo canceller has seen the original output signal.
• Echo canceller coverage-The amount of time the echo canceller retains a signal that has been output. This parameter must be set to a value greater than the time the echo needs to return to the gateway.

To eliminate echo, perform the following steps.

SUMMARY STEPS

1. Verify echo cancellation.
2. Configure echo canceller coverage.
3. Measure the echo and adjust the echo signal level.
4. Verify the impedance configured in the voice port.

1. Verify that echo cancellation is enabled on the voice port. Echo cancellation is enabled by default on most platforms in most Cisco IOS releases.

Gateway(config-voiceport)# echo-cancel 
  coverage   Echo Cancel Coverage  
  enable     Echo Cancel Enable 

Note:

You must enter the shut, then the no shut commands on the voice port for the changes to take effect.

2. Configure the echo canceller coverage to a value greater than the time the echo needs to return to the gateway. It must be long enough to cover the worst case for your environment, but not longer.

Gateway(config-voiceport)# echo-cancel coverage 
  16  16 milliseconds echo canceller coverage  
  24  24 milliseconds echo canceller coverage  
  32  32 milliseconds echo canceller coverage  
  8   8 milliseconds echo canceller coverage 

Note:

You must enter the shut and no shut commands on the voice port for the changes to take effect.

The default coverage is set to 8 ms, but it can be increased up to 64 ms, depending on the Cisco IOS software release. If the PSTN delay (tail length) is more than 32 ms, echo cancellers in Cisco IOS gateways cannot cancel the echo.

3. Measure the echo and adjust the echo signal level as required.

Insufficient echo return loss (ERL) to handle the echo might cause the following problems:

• Echo canceller is cancelling but not enough to make echo inaudible. If the ERL value is too low, the total ERL seen by the IP network Acombined (ACOM) might be insufficient to suppress the echo. ERL should be approximately 20 dB (at least 15 dB).

ACOM is the total ERL seen across the incoming and outgoing terminals of the echo canceller. The incoming terminal is equal to the signal sent into the ECAN toward the PSTN (voice), and the outgoing terminal is equal to the signal coming out of the ECAN toward the IP network (echo). ACOM is the sum of the ERL, plus ERLE, or the total ERL seen by the network. ACOM (Total loss) is equal to the ERL (tail loss) plus ERLE (ECAN loss).

• Echo canceller is not canceling. If the ERL value is too low, the echo signal returning to the gateway might be too loud (within 6 dB of the talker signal), causing the echo canceller to consider it as voice (double-talk) instead of echo. As a consequence, the echo canceller does not cancel it. ERL should be approximately 6 dB or higher for the echo canceller to engage. In 12.2(13)T, you can configure this ERL level.

To prevent these problems, you need to measure the ERL and signal levels, and adjust the signal levels on the Cisco IOS gateway based on the results. See theMeasuring QoS for more information about measuring QoS. See the Measuring Echo in Cisco IOS Gateways for more information about measuring ERL. You can adjust these levels by configuring positive values for output attenuation and negative values for input gain. Input gain is performed before the echo canceller has seen the echo signal, and output attenuation is performed after the echo canceller has seen the original output signal.

voice-port 1/1:15  
  input gain -3  
  output attenuation 3 

Note:

You must enter the shut and no shut command on the voice port for the changes to take effect.

Note:

In Cisco IOS Release 12.2(1) and later, output attenuation can be set to a negative value, which amplifies the output signal.

4. An impedance mismatch can cause echo if both sides are not configured identically. Verify the impedance configured in the voice port and modify it if needed. A default of 600 ohms is consistent with most lines on the PSTN and PBXs.

Gateway(config-voiceport)# impedance 
  600c 600 Ohms complex 
  600r 600 Ohms real 
  900c 900 Ohms complex 
  complex1 complex 1 
  complex2 complex 2

Measuring Echo in Cisco IOS Gateways

If you have a phone number to which echo is reproducible, using a Cisco IP Phone 7960 or Cisco IP Phone 7940, perform the following steps to generate test tones and measure the echo return loss (ERL).

SUMMARY STEPS

1. Enable the tone generator.
2. Place a call to the source of echo.
3. Press the i button twice.
4. Set the input and output levels to 0 dB gain/attenuation.
5. View the input and output levels for your call.
6. Configure 1 dB of attenuation in each direction.
7. Configure 3 dB of attenuation in each direction.
8. Change the coverage to 32 ms.

1. To enable the tone generator, type **3 on the Cisco IP Phone 7960 or 7940 keypad while the phone is not on a call. This enables the Tone softkey for as long as this phone is registered to Cisco CallManager.

Note:

From Load #P0030301ESP5, you need to unlock the phone first, then press **3. If you press **# **3 consecutively you reset the phone (because of the **#** sequence). Therefore, you need to press **#, make a call, then press **3.

2. Place a call to the source of echo.

3. After the call is established, press the i button twice. This brings up the statistics for the call.

Note:

If pressing **3 worked, you should have a Tone softkey available. Press the Tone softkey and the phone begins to generate a 1004 Hz tone at -15 dB. The only way to stop the tone is to end the call.

4. Make sure you start with a cleared configuration by setting the input and output levels to 0 dB gain/attenuation. After the tone is being generated, keep the call up, then perform the remaining steps to measure the decibel levels.

5. Use the show call active voice command to view the input and output levels for your call. The following is a call to a loopback number that simply echoes back whatever is sent with no attenuation.

Gateway# show call active voice  
   
  ! snip 
  OutSignalLevel=-15  
  InSignalLevel=-15  
  ERLLevel=25  
  ! snip 

The test tone in this example is output as -15 and is looped back with 0 dB loss. Therefore, the tone is coming back at -15 dB. The ERL value in the example has no significance because the echo canceller does not consider the input signal to be echo.

Note:

The OutSignalLevel shows the value of the level after the output attenuation has been applied to the signal. The InSignalLevel shows the value of the level after the input gain has been applied.

6. If you configure 1 dB of attenuation in each direction, the resulting levels decrease, as shown in the following configuration examples:

voice-port 1/1:15 
  input gain -1 
  output attenuation 1 
Gateway#show call active voice  
  ! snip 
  OutSignalLevel=-16 
  InSignalLevel=-17 
  ERLLevel=11 
  ! snip 

Note:

Notice that the OutSignalLevel is -16 because the -15 dB signal is attenuated by 1 dB. The InSignalLevel is -17 dB, the result of the -1 input gain. The ERL is 11 dB in the example, but it is actually 2 dB; the echo canceller still does not acknowledge the input signal as echo.

7. If you configure 3 dB of attenuation in each direction the resulting levels are shown in the following examples:

voice-port 1/1:15 
  input gain -3 
  output attenuation 3 
Gateway#show call active voice  
  ! snip 
  OutSignalLevel=-18 
  InSignalLevel=-21 
  ERLLevel=6 
  ! snip 

Note:

Notice that the OutSignalLevel is -18 because the -15 dB signal is attenuated by 3 dB. The InSignalLevel is -21 dB as a result of the input gain of -3. The expected ERL of 6 dB is now correct.

8. If you keep the same 3 dB in each direction, but change the coverage to 32 ms, the resulting levels are shown in the following examples.

voice-port 1/1:15  
  input gain -3  
  output attenuation 3  
  echo-cancel coverage 32 
Gateway#show call active voice  
                                     
  ! snip 
  OutSignalLevel=-18 
  InSignalLevel=-21 
  ERLLevel=6 
                                     
  ! snip 

The levels look the same as the previous result, however the echo canceller takes a few more seconds longer to converge. Do not set the echo cancel coverage higher than necessary.

For more information about troubleshooting echo problems, refer to the following documents:

• Troubleshooting Echo Problems between IP Phones and Cisco IOS Gateways, document ID 19640
• Echo Analysis for Voice over IP white paper, at the following website:

http://www.cisco.com/en/US/tech/tk652/tk701/technologies_white_paper09186a00800d6b68.shtml

Voice Level Adjustment

Voice levels can require adjustment for a variety of reasons. The voice on a call might be too loud or quiet, or callers might have an issue with comfort noise generated when using voice activity detection (VAD). This section contains informaton about the following:

• Input and Output Levels
• Voice Activity Detection Level

Input and Output Levels

As much as possible, it is desirable to achieve a uniform input decibel level to the packet voice network. Doing so eliminates or at least reduces any voice distortion because of incorrect input and output decibel levels. Adjustments to levels might be required by the type of equipment connected to the network or by local country-specific conditions.

Incorrect input or output levels can cause echo, as can an impedance mismatch. Too much input gain can cause clipped or fuzzy voice quality. If the output level is too high at the remote router's voice port, the local caller hears an echo. If the local router's voice port input decibel level is too high, the remote side hears clipping. If the local router's voice port input decibel level is too low or the remote router's output level is too low, the remote side voice can be distorted at a very low volume, and DTMF signaling might be missed.

Use the input gain and output attenuation commands to adjust voice levels, and the impedance command to set the impedance value to match that of the voice circuit to which the voice port connects.

To change parameters related to voice levels, enter the following commands.

SUMMARY STEPS

1. enable
2. configure terminal
3. voice-port port:ds0-group-number
4. input gain value
5. output attenuation value
6. impedance {600c | 600r | 900c | complex1 | complex2}
7. exit

Router> enable  

Router# configure terminal  

Router(config)# voice-port 1/0:0  

Router(config-voiceport)# input gain 0  

Router(config-voiceport)# output attenuation -6  

Router(config-voiceport)# impedance 600r  

Router(config-voiceport)# exit  

Voice Activity Detection Level

Another way to reduce clipped or fuzzy speech is to change the VAD level. Sometimes VAD cuts the sound too early and the speaker's voice is clipped. Use the vad and no vad commands in dial-peer configuration mode to enable and disable VAD.

With VAD, voice data packets fall into three categories: speech, silence, and unknown. Speech and unknown packets are sent over the network; silence packets are discarded. The sound quality is slightly degraded with VAD, but the connection monopolizes much less bandwidth. If you use the no form of this command, VAD is disabled and voice data is continuously sent to the IP backbone.

When the aggressive keyword is used, the VAD noise threshold is reduced from -78 to -62 dBm. Noise that falls below the -62 dBm threshold is considered to be silence and is not sent over the network. Additionally, unknown packets are considered to be silence and are discarded.

You can also change the amount of time that VAD waits for silence. Use the voice vad-time command in global configuration mode to change the minimum silence detection time. For example, setting the VAD time to 3000 allows 30 seconds of silence before VAD is initiated.

On gateways that use MGCP, the call agent controls the amount of VAD. For H.323 gateways, VAD must be disabled on the gateway itself. VAD is enabled by default on H.323 gateways. To disable VAD, you must match a VoIP dial peer that has the no vad command configured.

Some codecs, such as G.729b and G.729ab, have VAD built into them and cannot be diabled. If you do not want to use VAD, use the standard G.729 or G.729a codecs instead.

Hissing or Static While You Are Using Voice Activity Detector

The VAD detects silence periods in the voice signal and temporarily discontinues transmission of the signal during these periods. This saves bandwidth and allows the far-end to adjust its jitter buffer. The downside is that the far-end phone has to generate its own signal to play to the listener during silence periods. Usually comfort noise is played out to the listener to mask the absence of an audio signal from the far-end. Comfort noise is usually modeled on the far-end noise so that there is not a stark contrast between the two. Sometimes there can be unwanted noise generated by VAD. To troubleshoot hissing or static issues that might be because of the VAD, refer to Troubleshooting Hissing and Static, document ID 22388.