Cisco Unified Survivable Remote Site Voicemail -- Troubleshooting
General SRSV Troubleshooting
Using Traces for Troubleshooting
- Use the trace exec mode CLI to turn on the desired traces. Configuring trace srsx srsv-engine all on SRSV-UMG is a good place to start.
- Issue the command show trace buffer tail on the SRSV-UMG CLI in exec mode. The CLI will stop taking input and output trace messages as they are generated.
- Issue Ctrl-C to stop the trace and resume the SRSV-UMG CLI.
- Check the alerts displayed for failures on SRSV-UMG using show srsx alerts on CLI or the dashboard on the GUI.
Error Retrieving Voicemail Common Configuration
- Check the network connectivity test from SRSV-UMG.
- Check if the REST credentials set for the CUC server on SRSV-UMG match the CUC server Admin login credentials.
Error: Hostname not found in DNS
- If you see “Hostname not found in DNS” error while adding devices, make sure that these devices are configured in DNS server in both reverse and forward lookup zones.
- After creating these zones, if you still hit hostname not found errors, give clear ip dns cache in the CLI on SRSV-UMG and retry adding devices.
Unable to Connect to SRSV: Socket Timeout
- Check the network connectivity test from the SRSV-UMG GUI for success for that branch voicemail server.
- if the SRSV-CUE is behind a NAT also check the following
- Check if the SRSV-CUE is registered with the correct PAT port number in the branch voicemail server page from the SRSV-UMG GUI.
- Check the NAT/PAT translations setup on the NAT router.
- Check the TLS security setting on the System > SRSV settings page in the SRSV-UMG GUI.
Language Not Installed
Problem: Language <language> is not installed.
Explanation: Not all Cisco Unity Connection configuration settings apply to the SRSV-CUE device.
Recommended Action 1: Check to see if the system default language on Cisco Unity Connection is installed on the SRSV-CUE device.
Recommended Action 2: Check to see if the subscriber’s default language is installed on the SRSV-CUE device.
Mailbox Count Reached Licensed Mailbox Count
Problem: Mailbox count reached or licensed mailbox count reached.
Solution: Check to see if there are enough mailbox licenses for subscribers. Use the show license CLI command on the SRSV-CUE to see how many mailboxes are licensed. Order and install more mailbox licenses if all installed licenses are active. If licenses are not active use license activate srsv mailboxes <number> on the SRSV-CUE CLI to enable the mailbox licenses.
Error Updating REST Passwords
- Re-register the SRSV-CUE to make the passwords in sync with SRSV-UMG. In SRSV-CUE to register, use the CLI messaging-gateway srsx register <umg-ip-addr> user <name> password <passwd> command.
Security Certificate Troubleshooting
Checking that the Certificates are Copied Correctly
- Check if the crypto certificates of CUC and CUCM are copied onto SRSV-UMG. Use the following CLI command to copy the certificates:
crypto key import trustcacert label CUC80 url ftp:
SSL/TLS Handshake Fails when Getting SRST References
- Check if the services on CUCM are up and running.
- Check that you can ping CUCM from SRSV-UMG.
- Run a network connectivity test from the SRSV-UMG GUI.
- Ensure that the CUCM certificate is copied onto SRSV-UMG.
SRST References Failed
- Check if the CUCM AXL credentials on SRSV-UMG are configured correctly.
- Check that the CUCM services are up and SRSV-UMG can reach CUCM.
- Run a network connectivity test from the SRSV-UMG GUI.
- Ensure that the AXL service is up on CUCM.
- Ensure that the CUCM certificate is set up on SRSV-UMG.
- Use the sh trace buffer tail CLI command.
HTTPS Hostname Mismatch with X.509 Certificate
- Check if the certificate from CUCM has the CN as the fully qualified domain name. For example, CN=cucm80.conn.cisco.com. Can be checked from SRSV-UMG using the CLI command show crypto key all.
- If not, then add the DNS server name and domain name to CUCM. Check Show > Network from the CUCM OS administration. Regenerate the certificate on CUCM from Security > Certificate Management and do the CLI command generate new.
- Reload CUCM and check that the new tomcat certificate has the full CN name.
- Check if the validity time on certificate is not later than the time on SRSV-UMG and then copy the certificate to UMG.
Certificate Installation Fails
If certificate installation fails with “Failed due to importing certificate is not yet valid”, check the validity of the certificate and the clock on SRSV-UMG/SRSV-CUE.
Problem: Provisioning fails because the system has reached the maximum mailbox size.
Solution: Reduce the mailbox size by updating the Mailbox Size parameter on the Site Template Profile page.
Problem: The set time for scheduled provisioning was missed because the system was down for maintenance.
Solution 1: Wait until the next provisioning time.
Solution 2: Perform a manual provision.
Problem: Provisioning fails for a site with the alert "SRSV provisioning failed for site SITE_NAME for reason Not all configuration could be successfully applied".
Solution: Some configuration item from the central voicemail system failed to be provisioned for that site. To determine the cause use the following technique.
- Enable the traces using either the GUI or CLI. The CLI trace commands shown below can be enabled on the GUI from the page "Troubleshoot -> traces".
- trace srsx call-agent-client all
- trace srsx vm-server-client all
- trace srsx srsv-engine all
- Run a provisioning cycle.
- Collect the trace output using the GUI or CLI. One the CLI you can use "show trace buffer tail" and on the GUI you can go to "Troubleshoot -> View -> Trace Buffer".
- Look for 4xx or 5xx failures in the traces. When you find the failure the cause will be found in the response message contents. An example for a problem with a display list name is shown below (The XML has been reformatted here for readability).
--------------- Request --------------- POST http://junebug-nme.tdennler.srsv.bxb.lab/rest/voicemail/publiclists Content-Type: application/xml Accept: application/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <distributionList xmlns:ns2="http://www.cisco.com/aesop/srsv/error" xmlns:ns3="http://www.cisco.com/aesop/cue/srsv/config/UmgSrsvList" xmlns:ns4="http://www.cisco.com/aesop/cue/srsv/config/GlobalConfig" xmlns:ns5="http://www.cisco.com/aesop/srsv/aa"> <listName>ted & bill list</listName> <listNumber>6080</listNumber> </distributionList>
--------------- Response --------------- 406: Not Acceptable Set-Cookie: JSESSIONID=0EDEBC0884F484C0134EB399A6A70436; Path=/rest Transfer-Encoding: chunked X-REST-PI-Version: 220.127.116.11 Content-Type: application/xml Server: Apache-Coyote/1.1 Date: Thu, 17 Feb 2011 13:57:23 GMT
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <errorResponse> <errorCode>996</errorCode> <errorMessage> Validation check failed for bean type com.cisco.aesop.voicemail.rest.model.distributionlist.DistributionList for the following fields: listName[regex_check][regex = [-a-zA-Z0-9._+]*] </errorMessage> </errorResponse>
In this example the distribution list name being provisioned called "ted & bill list" has invalid characters in it. The error response is indicating that only character (A-Z and a-z), numbers (0-9), hyphen (-), underscore (_), dot (.), and plus (+) are allowed. Taking into account HTML escaping, the problem distribution list name uses two invalid characters: space ( ) and ampersand (&). This error can be corrected by renaming the distribution list as "ted_and_bill_list".
The table below shows most of the commonly found problems that can lead to SRSV provisioning problems.
|Invalid Distribution list name||Distribution list names on SRSV-CUE can only contain the characters a-z, A-Z, 0-9, hyphen, underscore, dot, and plus as defined by the regular expression "[-a-zA-Z0-9._+]*". Make sure distribution list names do not contain any characters not in this list and are limited to 64 characters.|
|Invalid subscriber/user alias||SRSV-CUE limits user aliases to start with a letter (a-z, A-Z); must be followed by one or more letters (a-z, A-Z), numbers (0-9), hyphen (-), underscore (_), dot (.); and ended with a letter (a-z, A-Z) or number (0-9) as defined by the regular expression "[a-zA-Z][-a-zA-Z0-9_\.]*[a-zA-Z0-9]". Make sure user names are restricted to the pattern described and have at least 2 characters no more than 31 characters.|
|Invalid spoken name||Recorded spoken names on SRSV-CUE must be recorded in G711 codec and be 10 seconds or less in length. If the spoken names learned from the central voicemail server do not meet these requirements they will not be provisioned.|
Backup and Restore Troubleshooting
Problem: The backup and/or restore process failed.
Solution: Ensure that you correctly configured the backup server.
SRSV-CUE Registration Troubleshooting
Problem: SRSV-CUE device registration failed.
Solution: Ensure that the user who is doing the registering has the srsx privilege.
Problem: SRSV-CUE registration reports "HTTP Failure: Bad Request" such as shown in the following example.
messaging-gateway srsx register 192.168.28.52 user cisco password cisco
Registration failed. HTTP Failure: Bad Request
Solution: The most common cause for this failure is missing DNS configuration. To determine the cause of the failure you are seeing enable "srsv configuration" tracing, run the registration, then dump the trace buffer. An example is shown below (NOTE: XML in traces below has been reformatted from console output to make it more readable).
srsv-test-nme# trace srsv configuration all srsv-test-nme# clear trace srsv-test-nme# messaging-gateway srsx register 192.168.28.52 user cisco password cisco
Registration failed. HTTP Failure: Bad Request junebug-nme# show trace buffer Press <CTRL-C> to exit... 3641 02/13 18:00:10.581 srsv conf 4
--------------- Request --------------- PUT http://SRSV-UMG.srsv.lab:80/umg/srsx/srsv-device/registration Content-Type: application/xml Accept: application/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ns2:RegistrationRequest xmlns:ns2="http://www.cisco.com/aesop/srsv/Registration"> <Hostname>192.168.28.133</Hostname> <BranchCallAgentHostname>junebug.tdennler.srsv.bxb.lab</BranchCallAgentHostname> <DefaultBranchCallAgent>true</DefaultBranchCallAgent> <SerialNumber>FOC14024CWS</SerialNumber> <SoftwareVersion>11.01.19.ref.td</SoftwareVersion> <Module>NME</Module> <Memory>514044</Memory> <MailboxLicense>100</MailboxLicense> <PortLicense>24</PortLicense> <UmgMinVersion>0.0.0.0</UmgMinVersion> </ns2:RegistrationRequest>
--------------- Response --------------- 400: Bad Request Content-Length: 201 Connection: close Set-Cookie: JSESSIONID=B4B6604225E031CB447D77DA86AC85C4; Path=/umg X-REST-PI-Version: 11.02.12.fix.td Content-Type: application/xml Server: Apache-Coyote/1.1 Date: Mon, 14 Feb 2011 00:08:20 GMT
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ns2:Error xmlns:ns2="http://www.cisco.com/aesop/srsv/error"> <Message>SRSV-CUE 192.168.28.133 has not been configured in DNS</Message> </ns2:Error> ----------------------------------------
In this instance the UMG is rejecting registration because the SRSV-CUE at the address 192.168.28.133 is not in DNS (see the Message content in the 400 error response above -- "SRSV-CUE 192.168.28.133 has not been configured in DNS"). To correct this problem add the reverse lookup for 192.168.28.133 to it's fully qualified domain name in DNS. Once in DNS you can use the host or nslookup command on a linux/unix or Windows system respectively to ensure the entry is now in DNS as shown below.
# host 192.168.28.133 18.104.22.168.in-addr.arpa domain name pointer srsv-test-nme.srsv.lab.
Also ensure the forward DNS lookup is configured for the SRSV-CUE then reset the DNS cache on the UMG (NOT THE SRSV-CUE) with the command shown below in CLI exec mode.
SRSV-UMG# clear ip dns cache
Try registering the SRSV-CUE again and this time it should work.
Problem: A subscriber who was just created in Cisco Unity Connection and was provisioned on Cisco Unified SRSV does not know their PIN.
Reason: The subscriber must set up their voicemail preferences on Cisco Unity Connection before they can log in to Cisco Unified SRSV-CUE devices. In rare cases the subscriber may have setup their mailbox but the SRSV system may not have been able to provision the new pin to the subscribers SRSV mailbox before the SRSV-CUE became active.
Solution: A temporary pin can be created on the SRSV-CUE with the CLI user <userid> generate pin. Give the pin number generated to the user to access their voicemail while SRSV-CUE is active. The subscriber will still need to setup their mailbox when CUC service is restored.
Message Waiting Indicator (MWI) Issues
Problem: The MWI feature does not work in SRSV mode for unsolicited.
Solution: Ensure that the MWI feature is enabled for the site template. Then ensure that the correct site template is applied to the site.
Problem: The MWI feature does not work in SRSV mode for sub-notify.
Solution 1: Check the configuration on IOS.
Solution 2:Ensure that the MWI feature is enabled for the site template. Then ensure that the correct site template is applied to the site.
Solution 3:To see if the MWI feature works, enter the CLI command mwi refresh all on the SRSV-CUE device.
Problem: The system prompts subscribers for a PIN when dialing in to their voicemail using the TUI.
Solution 1: Ensure that the PINless login feature has been configured for the subscriber on Cisco Unity Connection by using the following CLI command: show voicemail detail mailbox user_name.
Solution 2: Ensure that the provisioning was successful.
Solution 3: Ensure that you have configured the PINless settings for the subscriber.
Problem: The subscriber cannot log in from any phone without a PIN.
Solution: Manually configure PINless login so that subscribers can log in without a PIN from any phone.
Live Record Issues
Problem: The live record feature does not work.
Solution 1: On the SRSV-UMG GUI, ensure that the live record feature is enabled for the site template. Then ensure that the correct site template is applied to the site. Ensure that provisioning has been enabled for the site and was successful.
Solution 2: Ensure that the live record pilot number is correctly configured for the site template.
Solution 3: On Cisco Unified Communications Manager Express, ensure that a dummy ephone-dn is configured as the live record pilot number and has “call forward all” to the Cisco Unified SRSV-CUE devices voicemail number.
Solution 4: Ensure that the live record feature is performed from the subscriber’s extension.
Problem: The live record beep does not get played in the middle of a live record call.
Solution: Ensure that the live record feature is enabled for the site template on the SRSV-UMG GUI.
Live Reply Issues
Problem: The live reply feature is not enabled on Cisco Unity Express after provisioning.
Solution: On the SRSV-UMG, ensure that the live reply feature is enabled for the site template. Then ensure that the correct site template is applied to the site.
Problem: Subscribers are unable to live reply to a message using the TUI and hear the message “Invalid selection. Try again.”
Solution: Ensure that you are not using an invalid message type for the live reply feature. Live reply is not supported for broadcast and expired messages, non-delivery receipt delayed delivery receipt messages, and messages from local GDM (GTP Director Module).
Problem: Deleted messages on Cisco Unity Express appear as new messages in Cisco Unity Connection after being uploaded.
Possible Cause: In Cisco Unity Connection Release 7.1.3, the system uploads deleted voicemails as new. The subscriber must manually log in to his voicemail on Cisco Unity Connection and delete the message again.
In Cisco Unity Connection Release 8.0, the system uploads deleted voicemails as deleted.
Mailbox Licenses Not Installed
Problem: Mailbox licenses need to be installed.
- View the error messages in the UMG trace buffer or log file (syslog).
- Verify license status on the SRSV-CUE by issuing the CLI command show license status application.
There are two options to enable mailbox creation:
- Install a permanent license on the SRSV-CUE and perform a reload.
- Activate the evaluation (trial period) license and reload.
After reloading, check the license status and show the licenses in-use. If the license state shows "License State: Active, Not in Use", then activate the license with the CLI command license activate srsv mailboxes.
All Mailboxes Not Provisioned to SRSV-CUE
Note: Mailbox provisioning will take place in alphabetic order until the number of mailboxes licensed is exceeded.
- View the error message in the UMG trace buffer.
- Install additional mailbox licenses in the SRSV-CUE equal to or greater than the number of mailboxes to be provisioned to the site.
Users Cannot Access SRSV-CUE Mailbox
Subscribers will encounter a busy tone when attempting to access the SRSV-CUE mailbox if there are no port licenses installed or if there are more subscribers trying to access their mailbox simultaneously than there are port licenses installed.
- View the error messages in the SRSV-CUE messages.log (syslog).
- Verify license status on the SRSV-CUE by issuing the CLI command show license status application
There are two options to enable port creation:
- Install a permanent license on the SRSV-CUE and perform a reload.
- Activate the evaluation (trial period) license and reload.
After reloading, check the license status and show the licenses in-use. If the license state shows "License State: Active, Not in Use", then activate the license with the CLI command license activate srsv ports.
Cannot Register SRSV-CUE
Problem: Not enough SRSV node licenses are available in the SRSV-UMG.
Possible Symptoms: The branch voicemail server will not be assigned to a site and can be viewed through the "Unassigned Branch Voicemail Servers" page on the GUI. You may see error messages in the Alert History page on the GUI or in the log file in the SRSV-UMG GUI.
Recommended Action: Install port licenses.
Voicemail Upload Troubleshooting
Note: The SRSV Activity History Report in the GUI shows how many undeliverable voicemails there are in the system.
Voicemail Upload Fails Due to Connectivity Issues
Note: Use the show voicemail mailbox CLI command to check if a voicemail message was not uploaded from the mailbox of a Cisco Unified SRSV-CUE device.
Problem: Voicemail upload fails due to connectivity issues.
Possible Cause 1: Make sure that the Cisco Unity Connection can be reached from SRSV-UMG and that the SRSV-UMG can be reached from the Cisco Unified SRSV-CUE device.
Recommended Action 1: Run network connectivity tests.
Possible Cause 2: The SRSV-UMG system does not have the correct Cisco Unity Connection credentials configured.
Recommended Action 2: Configure the correct credentials.
Possible Cause 3: You are not using the correct REST password.
Recommended Action 3: Use the correct REST password.
Possible Cause 4: The security certificates are not current or have not been installed.
Recommended Action 4: Check that the security certificates are correctly installed by using the show crypto key all CLI command.
Possible Cause 5: Cisco Unity Connection server information is not configured correctly.
Recommended Action 5: Check the central voicemail server on the GUI or use the CLI command show srsx central-voicemail-server cuc-server-name.
Possible Cause 6: The services on Cisco Unity Connection are not up and running.
Recommended Action 6: Ensure that the conversation manager, MTA, REST, and SMTP services are activated. In Cisco Unity Connection, configure the full SMTP domain name.
Determining if the Voicemail Upload Failed Due to a Full Disk
To determine if the voicemail upload failed because the Cisco Unity Connection disk is full, do one of the following:
- In the GUI, go to the dashboard and look for a message that states the following:
UploadFailedSystemFull - UMG was unable to deliver messages for a Central Voicemail Server because the Voicemail Server is full.
- Enter the CLI command trace srsx upload all. Look for the output that says “Mail system full”.
Voicemail Uploading Problems
Problem: Voicemails uploaded through a secondary SRSV-UMG fail.
Possible Cause 1: The secondary SRSV-UMG system does not have the correct Cisco Unity Connection credentials configured.
Recommended Action 1: Configure the correct credentials.
Possible Cause 2: You are not using the correct SRSV-UMG REST password. The REST passwords must be the same for both the primary and the secondary SRSV-UMG systems.
Recommended Action 2: Use the correct REST password.
Possible Cause 3: The security certificates are not current.
Recommended Action 3: Check that the security certificates are correctly installed by using the show crypto key all CLI command.
Problem: The sender’s mailbox in Cisco Unity Connection has a non-delivery receipt (NDR) message for a message that was not delivered to a VPIM contact. Or you see an error message in the trace that says “No user or contact found.”
Solution 1: Ensure that the VPIM contact was properly created in the Cisco Unity Connection contact list.
Solution 2: Ensure that you properly addressed the VPIM contact.
Solution 3: Ensure that you dial the numbers in the same way that you dial on Cisco Unity Connection to reach a VPIM contact.
Problem: Voicemails uploaded through HTTPS fail.
Solution: Ensure that the certificate generated from SRSV-UMG is installed on the branch voicemail server and vice versa (if HTTPS is enabled on both).
Distribution List Issues
Problem: Distribution list names are not provisioned on a Cisco Unified SRSV-CUE device.
Solution: Turn on traces on Cisco Unified SRSV-UMG by running the show trace buffer tail CLI command. Then use the trace srsx srsv-engine all CLI command to check which distribution lists were provisioned on the system.
Srsv-umg# show trace buffer tail <baseElementList> <baseElement xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns3=http://distributionlist.model.rest.voicemail.aesop.cisco.com xsi:type="ns3:distributionList"> <resourceURI>/voicemail/publiclists/allvoicemailenabledcontacts </resourceURI> <description></description> <listName>allvoicemailenabledcontacts</listName> <listNumber>99992</listNumber>
Auto Attendant Issues
Problem: The auto attendant prompts are not in the subscriber’s language.
Explanation: The Cisco Unified SRSV-CUE system only recognizes the subscriber’s language after the subscriber logs into their mailbox.
Recommended Action: Ensure that the subscriber logs into their mailbox first.
Problem: You cannot reach the site operator, or you get an invalid prompt message.
Possible Cause 1: You have not configured a site operator.
Solution 1: To see if you have configured a site operator, run the following SRSV-CUE CLI command, which will output the site operator extension, if configured: show srsv configuration | i Site
Possible Cause 2: You configured the wrong extension for the site operator.
Solution 2: To change the site operator extension, see the Operator Extension parameter on the SRSV-UMG Site Profile page for the Site you are interested in.
Possible Cause 3: The site operator extension does not work.
Solution 3: Ensure that you can reach the operator extension by dialing the number while SRST is active.