Cisco IOS Voice Troubleshooting and Monitoring -- Troubleshooting Tools
This article presents information about the variety of tools available to assist you in troubleshooting your voice network, including information on using router diagnostic commands, Cisco network management tools, and third-party troubleshooting tools.
Using Router Diagnostic Commands
Cisco routers provide numerous integrated commands to assist you in monitoring and troubleshooting your internetwork.
- show commands help you monitor installation behavior and normal network behavior, and isolate problem areas.
- debug commands help you isolate protocol and configuration problems.
- ping commands help you determine connectivity between devices on your network.
- trace commands provide a method of determining the route by which packets reach their destination.
Using show Commands
show commands are powerful monitoring and troubleshooting tools. You can use show commands to perform a variety of functions:
- Monitor router behavior during initial installation.
- Monitor normal network operation.
- Isolate problem interfaces, nodes, media, or applications.
- Determine when a network is congested.
- Determine the status of servers, clients, or other neighbors.
The following are some of the most common show commands:
- show version-Displays the configuration of the system hardware, the software version, the names and sources of configuration files, and the boot images.
- show running-config-Displays the router configuration currently running.
- show startup-config-Displays the router configuration stored in NVRAM.
- show interfaces-Displays statistics for all interfaces configured on the router or access server. The resulting output varies, depending on the network for which an interface has been configured.
- show controllers-Displays statistics for interface card controllers.
- show flash-Displays the layout and contents of flash memory.
- show buffers-Displays statistics for the buffer pools on the router.
- show memory summary-Displays memory pool statistics and summary information about the activities of the system memory allocator, and gives a block-by-block listing of memory use.
- show process cpu-Displays information about the active processes on the router.
- show stacks-Displays information about the stack utilization of processes and interrupt routines, and the reason for the last system reboot.
- show cdp neighbors-Provides reachability information for directly connected Cisco devices. This is an extremely useful tool for determining the operational status of the physical and data link layers. Cisco Discovery Protocol (CDP) is a proprietary data link layer protocol.
- show debugging-Displays information about the type of debugging that is enabled for your router.
You can always use the ? at the command line for a list of subcommands.
Like the debug commands, some of the show commands listed are accessible only at the router's privileged EXEC mode. Debug command usage is explained further in Using debug Commands and also in the Debug Command Output on Cisco IOS Voice Gateways article.
Hundreds of other show commands are available. For details on using and interpreting the output of specific show commands, refer to the Cisco IOS command references.
Using debug Commands
The debug privileged EXEC commands can provide a wealth of information about the traffic seen (or not seen) on an interface, error messages generated by nodes on the network, protocol-specific diagnostic packets, and other useful troubleshooting data.
The following steps are a summary of debug command usage. For detailed debug command usage, see the Debug Command Output on Cisco IOS Voice Gateways article.
To access and list the privileged EXEC commands, enter this code:
Router> enable Password: XXXXXX Router# ?
Note the change in the router prompts here. The # prompt (instead of the normal > prompt) indicates that the router is in privileged EXEC mode (enable mode).
Use debug commands to isolate problems, not to monitor normal network operation. Because the high processor overhead of debug commands can disrupt router operation, you should use them only when you are looking for specific types of traffic or problems, and have narrowed your problems to a likely subset of causes.
Output formats vary with each debug command. Some generate a single line of output per packet, and others generate multiple lines of output per packet. Some generate large amounts of output, and others generate only occasional output. Some generate lines of text, and others generate information in field format.
To minimize the negative impact of using debug commands, follow this procedure:
- Use the no logging console global configuration command on your router. This command disables all logging to the console terminal.
- Telnet to a router port and enter the enable EXEC command. The enable EXEC command places the router in the privileged EXEC mode. After entering the enable password, you receive a prompt that consists of the router name with a # symbol.
- Use the terminal monitor command to copy debug command output and system error messages to your current terminal display.
By redirecting output to your current terminal display, you can view debug command output remotely, without being connected through the console port.
If you use debug commands at the console port, character-by-character processor interrupts are generated, maximizing the processor load already caused by the use of debug.
If you intend to keep the output of the debug command, spool the output to a file. The procedure for setting up such a debug output file is described in the Debug Command Output on Cisco IOS Voice Gateways article.
This book refers to specific debug commands that are useful when you are troubleshooting specific problems. Complete details regarding the function and output debug commands are provided in the Cisco IOS Debug Command Reference.
In many situations, using third-party diagnostic tools can be more useful and less intrusive than using debug commands. For more information, see the Third-Party Troubleshooting Tools.
Using the ping Commands
To check host reachability and network connectivity, use the ping command, which can be invoked from both user EXEC mode and privileged EXEC mode. After you log in to the router or access server, the router is automatically in user EXEC command mode. The EXEC commands available at the user level are a subset of those available at the privileged level. In general, the user EXEC commands enable you to connect to remote devices, change terminal settings on a temporary basis, perform basic tests, and list system information. The ping command can be used to confirm basic network connectivity on a variety of networks.
For IP, the ping command sends Internet Control Message Protocol (ICMP) echo messages. ICMP is the Internet protocol that reports errors and provides information relevant to IP packet addressing. If a station receives an ICMP Echo message, it sends an ICMP echo reply message back to the source.
The extended command mode of the ping command permits you to specify the supported IP header options. This mode allows the router to perform a more extensive range of test options. To enter ping extended command mode, enter yes at the extended commands prompt of the ping command.
It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions, so that you have a basis for comparison when you are troubleshooting.
For detailed information on using the ping and extended ping commands, refer to the Cisco IOS Configuration Fundamentals Command Reference.
Using the trace Commands
The trace user EXEC command discovers the routes that a router's packets follow when traveling to their destinations. The trace privileged EXEC command permits the supported IP header options to be specified, allowing the router to perform a more extensive range of test options.
The trace command works by using the error message generated by routers when a datagram exceeds its time-to-live (TTL) value. First, probe datagrams are sent with a TTL value of 1. This value causes the first router to discard the probe datagrams and send back "time exceeded" error messages. The trace command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by 1.
Each outgoing packet can result in one of two error messages. A "time exceeded" error message indicates that an intermediate router has seen and discarded the probe. A "port unreachable" error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application. If the timer goes off before a response comes in, trace prints an asterisk (*).
The tracecommand terminates when the destination responds, when the maximum TTL is exceeded, or when you interrupt the trace with the escape sequence.
As with ping, it is a good idea to use the trace command when the network is functioning properly to see how the command works under normal conditions, so that you have a basis for comparison when you are troubleshooting.
For detailed information on using the trace and extended trace commands, refer to the Cisco IOS Configuration Fundamentals Command Reference.
Using Cisco Network Management Tools
Cisco offers the CiscoWorks family of management products that provide design, monitoring, and troubleshooting tools to help you manage your internetwork.
The following tools are useful for troubleshooting internetwork problems:
- CiscoView provides dynamic monitoring and troubleshooting functions, including a graphical display of Cisco devices, statistics, and comprehensive configuration information.
- Internetwork Performance Monitor (IPM) empowers network engineers to proactively troubleshoot network response times utilizing real-time and historical reports.
CiscoView graphical management features provide dynamic status, statistics, and comprehensive configuration information for Cisco internetworking products (switches, routers, hubs, concentrators, and access servers). CiscoView aids network management by displaying a physical view of Cisco devices and color-coding device ports for at-a-glance port status, allowing you to quickly grasp essential information. Features include the following:
- Graphical displays of Cisco products from a central location, giving network managers a complete view of Cisco products. You do not need to physically check devices at remote sites.
- A continuously updated physical view of routers, hubs, switches, or access servers in a network, regardless of physical location.
- Updated real-time monitoring and tracking of key information relating to device performance, traffic, and usage, with metrics such as utilization percentage, frames sent and received, errors, and a variety of other device-specific indicators.
- The capability to modify configurations such as trap, IP route, VLAN, and bridge configurations.
Internetwork Performance Monitor
The Internetwork Performance Monitor (IPM) is a network management application that enables you to monitor the performance of multiprotocol networks. IPM measures the response time and availability of IP networks on a hop-by-hop (router-to-router) basis. It also measures response time between routers and the mainframe in Systems Network Architecture (SNA) networks.
Use IPM to perform the following tasks:
- Troubleshoot problems by checking the network latency between devices.
- Send Simple Network Management Protocol (SNMP) traps and SNA alerts when a user-configured threshold is exceeded, when a connection is lost and reestablished, or when a timeout occurs.
- Analyze potential problems before they occur by accumulating statistics, which are used for modeling future network topologies.
- Monitor response time between two network endpoints.
The IPM product is composed of three parts: the IPM server, the IPM client application, and the response time reporter (RTR) feature of the Cisco IOS software.
Third-Party Troubleshooting Tools
In many situations, third-party diagnostic tools can be more useful than commands that are integrated into the router. For example, enabling a processor-intensive debug command can be extremely detrimental in an environment experiencing excessively high traffic levels. Attaching a network analyzer to the suspect network is less intrusive and is more likely to yield useful information without interrupting the operation of the router. The following are some typical third-party troubleshooting tools used for troubleshooting internetworks:
- Volt-ohm meters, digital multimeters, and cable testers are useful for testing the physical connectivity of your cable plant.
- Time domain reflectometers (TDRs) and optical time domain reflectometers (OTDRs) are devices that assist in the location of cable breaks, impedance mismatches, and other physical cable plant problems.
- Breakout boxes, fox boxes, bit error rate testers (BERTs), and block error rate testers (BLERTs) are useful for troubleshooting problems in peripheral interfaces.
- Network monitors provide an accurate picture of network activity over a period of time by continuously tracking packets crossing a network.
- Network analyzers such as sniffers decode problems at all seven OSI layers. The problems can be identified automatically in real-time and categorized by how critical they are, providing a clear view of network activity.
Volt-Ohm Meters, Digital Multimeters, and Cable Testers
Volt-ohm meters and digital multimeters are at the lower end of the spectrum of cable-testing tools. These devices measure parameters such as AC and DC voltage, current, resistance, capacitance, and cable continuity. They are used to check physical connectivity.
Cable testers (scanners) also enable you to check physical connectivity. Cable testers are available for shielded twisted-pair (STP), unshielded twisted-pair (UTP), 10BASE-T, and coaxial and twinax cables. A given cable tester might be capable of performing any of the following functions:
- Test and report on cable conditions, including near-end crosstalk (NEXT), attenuation, and noise
- Use TDR and perform traffic monitoring and wire map functions
- Display MAC-layer information about LAN traffic, provide statistics such as network utilization and packet error rates, and perform limited protocol testing (for example, TCP/IP tests such as ping)
Similar testing equipment is available for fiber-optic cable. Because of the relatively high cost of this cable and its installation, fiber-optic cable should be tested both before installation (on-the-reel testing) and after installation. Continuity testing of the fiber requires either a visible light source or a reflectometer. Light sources capable of providing light at the three predominant wavelengths-850 nanometers (nm), 1300 nm, and 1550 nm-are used with power meters that can measure the same wavelengths and test attenuation and return loss in the fiber.
TDRs and OTDRs
At the top end of the cable testing spectrum are time domain reflectometers (TDRs). These devices can quickly locate open and short circuits, crimps, kinks, sharp bends, impedance mismatches, and other defects in metallic cables.
A TDR works by bouncing a signal off the end of the cable. Opens, shorts, and other problems reflect the signal back at different amplitudes, depending on the problem. A TDR measures how much time it takes for the signal to reflect and calculates the distance to a fault in the cable. TDRs can be used to measure the length of a cable, and some TDRs can also calculate the propagation rate based on a configured cable length.
Fiber-optic measurement is performed by an optical TDR (OTDR). OTDRs can accurately measure the length of the fiber, locate cable breaks, measure the fiber attenuation, and measure splice or connector losses. An OTDR can be used to take the signature of a particular installation, noting attenuation and splice losses. This baseline measurement can then be compared with future signatures when a problem in the system is suspected.
Breakout Boxes, Fox Boxes, BERTs and BLERTs
Breakout boxes, fox boxes, and bit/block error rate testers (BERTs/BLERTs) are digital interface testing tools used to measure the digital signals present at PCs, printers, modems, the channel service unit/data service unit (CSU/DSU), and other peripheral interfaces. These devices can monitor data line conditions, analyze and trap data, and diagnose problems common to data communication systems. Traffic from data terminal equipment (DTE) through data communications equipment (DCE) can be examined to help isolate problems, identify bit patterns, and ensure that the proper cabling has been installed. These devices cannot test media signals such as Ethernet, Token Ring, or FDDI.
Network monitors continuously track packets crossing a network, providing an accurate picture of network activity at any moment, or a historical record of network activity over a period of time. They do not decode the contents of frames. Monitors are useful for baselining, in which the activity on a network is sampled over a period of time to establish a normal performance profile.
Monitors collect information such as packet sizes, the number of packets, error packets, overall usage of a connection, the number of hosts and their MAC addresses, and details about communications between hosts and other devices. This data can be used to create profiles of LAN traffic and to assist in locating traffic overloads, planning for network expansion, detecting intruders, establishing baseline performance, and distributing traffic more efficiently.
A network analyzer (also called a protocol analyzer or "sniffer") decodes the various protocol layers in a recorded frame and presents the frames as readable abbreviations or summaries. The analyzer indicates which layer is involved (physical, data link, and so forth) and what function each byte or byte content serves.
Most network analyzers can perform many of the following functions:
- Filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured
- Time-stamp captured data
- Present protocol layers in an easily readable form
- Generate frames and send them onto the network
- Incorporate an "expert" system in which the analyzer uses a set of rules, combined with information about the network configuration and operation, to diagnose and solve, or offer potential solutions for network problems