Cisco Application Control Engine (ACE) Troubleshooting Guide -- Show Counter Reference -- Command Set 2

From DocWiki

(Difference between revisions)
Jump to: navigation, search
m
m
 
Line 6: Line 6:
__TOC__
__TOC__
-
 
== show fifo event-history ==
== show fifo event-history ==

Latest revision as of 21:43, 11 March 2011

Guide Contents
Main Article
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting ACE Appliance Ethernet Ports
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Compression
Troubleshooting Performance Issues
ACE Resource Limits
Managing ACE Resources
Show Counter Reference

Contents


show fifo event-history

This command is primarily intended for internal development use. It displays a log of the most recent messages generated by the Packet First-In-First_Out (FIFO) driver. The FIFO driver is the 16-Bit communication path (or interface) between the ACE BCM1250 and CDE.

The output displays the raw hexadecimal (hex dump) of the packets sent through this driver.

This command is used in conjunction with the diagnostic command "debug fifo ...", which has the following command syntax:

switch/Admin# debug fifo ?
  all      Debug Packet Fifo all
  error    Debug Packet Fifo errors
  info     Debug Packet Fifo info
  rxpkt    Debug Packet Fifo received packets
  txpkt    Debug Packet Fifo transmitted packets
  warning  Debug Packet Fifo warnings


Sample Output

2008 Feb  1 15:54:41 kernel: PKT-FIFO TX PACKET (DATA):
2008 Feb  1 15:54:41 kernel: 08 40 00 5c 00 50 80 34 00 00 00 05 00 05 dc 00 
2008 Feb  1 15:54:41 kernel: 00 04 00 00 0c 07 ac 00 00 05 9a 3b 98 81 08 00 
2008 Feb  1 15:54:41 kernel: 45 00 00 3c 53 47 40 00 40 06 13 06 0a 56 d7 35
2008 Feb  1 15:54:41 kernel: 0a 56 e8 8d e0 e2 00 31 49 03 ed 34 00 00 00 00 
2008 Feb  1 15:54:41 kernel: a0 02 16 d0 86 a6 00 00 02 04 05 b4 04 02 08 0a 
2008 Feb  1 15:54:41 kernel: 00 62 be 73 00 00 00 00 01 03 03 00 
2008 Feb  1 15:54:41 kernel: END OF PACKET 

Notes

You can see the MAC/IP/TCP headers in this packet starting at "00 00 0c 07....". The types of kernel Packet FIFO packet that will be displayed are:

  • PKT-FIFO TX PACKET (CTRL)
  • PKT-FIFO TX PACKET (DATA)
  • PKT-FIFO RX PACKET (CTRL)
  • PKT-FIFO RX PACKET (DATA)


Caution Caution: When the debug fifo command diagnostic logging is enabled it can generate huge amounts of data in the debug window. Please use care when enabling the debug fifo command. Generally, this command should be used only while in contact with TAC.

show fragment

Shows statistics related to IP fragmentation and reassembly activities by the ACE. An IP fragment results from datagrams that are broken into smaller pieces because they are larger than the maximum MTU permitted by a traversed link. ACE can generate fragments or reassemble datagrams that arrive as fragments, when needed.

Sample Output

ACE30001/Admin# show fragment vlan 23

Interface vlan23
        Fragment stats: Required 0, OK 0, Failed 0, Created 0
        IP Reassembly stats: Required 144, OK 0, Failed 96

Notes

The Fragment statistics refer to packet fragmentation as performed by the ACE. They include:

Field Description
Required Number of packets that required fragmentation
OK Number of packets that were successfully fragmented
Failed Number of packets that failed fragmentation.
Created Number of fragments that were created

IP Reassembly stats provide information on the fragments received and reassembled by ACE. They include:

Field Description
Required Number of packets that required reassembly
OK Number of packets successfully reassembled
Failed Number of duplicate or overlapping fragments that were dropped


show ft config-error

Indicates whether bulk sync issues were encountered during application of the config to the standby. This command is accessible in any context.

Sample Output

When no bulk sync issues are encountered:

ACE30001/Admin# show ft config-error
No bulk config apply errors
ACE30001/Admin# 

When an issue is encountered:

switch/201-150# show ft config-error 
Thu Apr  9 16:21:46 UTC 2009
`aaa authentication login default group TAC local`

`aaa authentication login error-enable`
--
`username 123456789123456789123456 password 5 $1$8Db5Ei2K$Bh3GCxg8p5TmwzIW8EQ8B/
 role Admin domain default-domain`

Error(s) while applying config.
switch/201-150# 

Notes

Notice the Error message that indicates an error when the ACE attempted to apply the configuration to the peer.


show ft group status

Shows the status of members of the FT group.

Sample Output

ACE5/Admin# show ft group status 

FT Group                     : 1
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Id                      : 1
No. of Contexts              : 1

Notes

There are no counters of troubleshooting interest in this output, since this command simply shows the state of members of the FT group.

Please refer to the Application Control Engine Module Administration Guide for more information.


show ft history cfg_cntlr

This command displays internal messages generated by the ACE when performing a bulk synchronization operation to an FT standby. It is helpful when troubleshooting config sync issues. Errors indicated in the output can be further investigated using the command "show ft config-error".

This command is only available in the Admin context.

Sample Output

With no errors reported:

ACE30001/Admin# show ft history cfg_cntlr

0:0 => Sep 18 19:01:46: main:4697 Waiting for System Manager notification to start config playback
0:1 => Sep 18 19:01:46: main_loop:4490 Entering select loop
0:2 => Sep 18 19:01:46: child_loop:4444 Entering select loop
0:3 => Sep 18 19:01:52: cfgcntlr_handle_mts_msg:3992 Recvd MTS_OPC_SYSMGR_ALL_SCOPES_STARTED. Starting 
                config playback
0:4 => Sep 18 19:01:52: start_config_playback:3846 confreg value is 0x0
0:5 => Sep 18 19:01:52: modify_config_mode_for_ctx:383 Acquired lock for context 0
0:6 => Sep 18 19:01:52: parse_and_apply_startup_cfg_file:3808 Executing first part of Admin context startup 
                configuration file /isan/vegas/work/Admin-startup-config1-bdrGQx
0:7 => Sep 18 19:01:58: ascii_bootstrap_cfg_execute:447 Successfully executed the config commands file 
                /isan/vegas/work/Admin-startu
p-config1-bdrGQx for context (Admin).
0:8 => Sep 18 19:01:58: ascii_bootstrap_cfg_execute:449 Removing log file 
                /isan/vegas/work/Admin-startup-config1-bdrGQx-927-0.log
0:9 => Sep 18 19:01:58: parse_and_apply_startup_cfg_file:3829 Admin first half config applied. Sent 
                MTS_OPC_REQ_CFG_DNLD_STATUS mess
age to local CfgMgr vcid 0
0:10 => Sep 18 19:01:58: start_config_playback:3876 Waiting for HA election to complete. 
                Timeout 240000 msecs
0:11 => Sep 18 14:01:58: cfgcntlr_handle_mts_msg:4196 Received FT_ASSOCIATE_CONTEXT notification 
                for FT Group 1
0:12 => Sep 18 14:01:58: cfg_set_auto_sync_variables:3505 Saved auto-sync mode as enabled.
0:13 => Sep 18 14:02:05: cfgcntlr_handle_mts_msg:4274 Received MTS_OPC_CFG_DNLD_STATUS message 
                for context 0
0:14 => Sep 18 14:02:05: handle_admin_ctx_bootup:2424 handle_admin_ctx_bootup cannot handle ha state: 0
0:15 => Sep 18 14:02:39: ha_save_peer_srg_info:342 Peer SRG received- major 2 minor 1.4a) compatibility 1
0:16 => Sep 18 14:02:39: cfgcntlr_handle_mts_msg:4169 Received STANDBY_CONFIG notification for FT Group 1
0:17 => Sep 18 14:02:39: handle_standby_config_mts:3540 config counter is now 1
0:18 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4048 Received FILE_TRANSFER notification for context 
                name Admin
0:19 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4060 transfer counter incremented to 1
0:20 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4064 Filename modified to /isan/vegas/work/005_Admin
                _0_cfgcntlr-peerbulk-cfg-1
0:21 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4077 FILE_TRANSFER processed for filename 
                /isan/vegas/work/005_Admin_0_cfgcntlr-peerbulk-cfg-1 context Admin type 0
0:22 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4096 Sent 4131 message to local context Admin
0:23 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4112 Recd: CONFIGfilename /isan/vegas/work/005_Admin
                _0_cfgcntlr-peerbulk-cfg-1 context Admin type 0 
0:24 => Sep 18 14:02:40: handle_cfg_sync_tftp_done_mts:2117 Received notification for context id 0
0:25 => Sep 18 14:02:40: process_bulk_config_sync:1766 Generating running config in file /isan/vegas/work/Admin-
                cfgcntlr-rollback-cfg for rollback
0:26 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4048 Received FILE_TRANSFER notification for context name Admin
0:27 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4060 transfer counter incremented to 2
0:28 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4064 Filename modified to /isan/vegas/work/005_Admin_1_
                cfgcntlr-startup-config.new-2
0:29 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4077 FILE_TRANSFER processed for filename /isan/vegas
                /work/005_Admin_1_cfgcntlr-startup-config.new-2 context Admin type 1
0:30 => Sep 18 14:02:40: cfgcntlr_handle_mts_msg:4096 Sent 4132 message to local context Admin
0:31 => Sep 18 14:02:40: process_bulk_config_sync:1789 Created running config in file successfully
0:32 => Sep 18 14:02:40: process_bulk_config_sync:1809 Successfully computed diff for context Admin
0:33 => Sep 18 14:02:42: cfgcntlr_handle_mts_msg:4048 Received FILE_TRANSFER notification for context 
                name spirent_ssg

With errors reported:

switch/Admin# show ft history cfg_cntlr 

0:0 => Apr 09 15:55:07: main:4697 Waiting for System Manager notification to start config playback
0:1 => Apr 09 15:55:07: child_loop:4444 Entering select loop
0:2 => Apr 09 15:55:07: main_loop:4490 Entering select loop
0:3 => Apr 09 15:55:13: cfgcntlr_handle_mts_msg:3992 Recvd MTS_OPC_SYSMGR_ALL_SCOPES_STARTED. Starting 
                config playback
0:4 => Apr 09 15:55:13: start_config_playback:3846 confreg value is 0x0
0:5 => Apr 09 15:55:13: modify_config_mode_for_ctx:383 Acquired lock for context 0
0:6 => Apr 09 15:55:13: parse_and_apply_startup_cfg_file:3808 Executing first part of Admin context startup 
                configuration file /isan/vegas/work/Admin-startup-config1-cjPuPW
0:7 => Apr 09 15:55:15: ascii_bootstrap_cfg_execute:447 Successfully executed the config commands file 
                /isan/vegas/work/Admin-startup-config1-cjPuPW for context (Admin).
0:8 => Apr 09 15:55:15: ascii_bootstrap_cfg_execute:449 Removing log file /isan/vegas/work/Admin-startup-
                config1-cjPuPW-933-0.log
0:9 => Apr 09 15:55:15: parse_and_apply_startup_cfg_file:3829 Admin first half config applied.  Sent 
                MTS_OPC_REQ_CFG_DNLD_STATUS message to local CfgMgr vcid 0
0:10 => Apr 09 15:55:15: start_config_playback:3876 Waiting for HA election to c /245
...skipping
1:245 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 60, bitmap[7]:0xffffffff
1:246 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 61, bitmap[7]:0xffffffff
1:247 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 62, bitmap[7]:0xffffffff
1:248 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 63, bitmap[7]:0xffffffff
1:249 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 64, bitmap[8]:0xffffffff
1:250 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 65, bitmap[8]:0xffffffff
1:251 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 66, bitmap[8]:0xffffffff
1:252 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 67, bitmap[8]:0xffffffff
1:253 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 68, bitmap[8]:0xffffffff
1:254 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 69, bitmap[8]:0xffffffff
1:255 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 70, bitmap[8]:0xffffffff
1:256 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 71, bitmap[8]:0xffffffff
1:257 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 72, bitmap[9]:0x3f
1:258 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 73, bitmap[9]:0x3f
1:259 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 74, bitmap[9]:0x3f
1:260 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 75, bitmap[9]:0x3f
1:261 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 76, bitmap[9]:0x3f
1:262 => Apr 09 15:56:02: cfg_cntrl_set_ctx_booting:238 Setting CTX_BOOTING forcontext id 77, bitmap[9]:0x3f
1:263 => Apr 09 15:56:02: send_ha_trigger_notif:394 MTS_OPC_HA_CTX_TRIGGER message sent to HA manager
1:264 => Apr 09 15:56:02: cfg_cntrl_inform_ha_sync_done:1927 Context id 0 for ctx Admin
1:265 => Apr 09 15:56:02: cfg_cntrl_inform_ha_sync_done:1975 cfg_cntrl_inform_ha_sync_donecalled before 
                ctx_start_sync_done FTGroup 241 vcid:0
1:266 => Apr 09 15:56:02: cfgcntlr_handle_mts_msg:4112 Recd: CONFIGfilename /isan/vegas/work/005_Admin_1_
                cfgcntlr-startup-config.new-2 context Admin type 1 
1:267 => Apr 09 15:56:02: handle_cfg_sync_tftp_done_mts:2117 Received notificati
/error
...skipping
20:3298 => Apr 09 16:21:46: ascii_bootstrap_cfg_execute:437 Error executing the config commands file /isan/
                vegas/work/201-150-cfgcntlr-diff-cfg for context (201-150): error - (null)
20:3299 => Apr 09 16:21:46: ascii_bootstrap_cfg_execute:440 Check errors via 'show config-error'
20:3300 => Apr 09 16:21:46: process_bulk_config_sync:1813 error: could not apply diff file 
                /isan/vegas/work/201-150-cfgcntlr-diff-cfg for context 201-150
20:3301 => Apr 09 16:21:46: cfg_cntrl_inform_ha_sync_done:1927 Context id 1 for ctx 201-150
20:3302 => Apr 09 16:21:46: cfg_cntrl_inform_ha_sync_done:1975 cfg_cntrl_inform_ha_sync_donecalled 
                before ctx_start_sync_done FTGroup 245 vcid:1
20:3303 => Apr 09 16:21:46: cfgcntlr_handle_mts_msg:4112 Recd: CONFIGfilename /isan/vegas/work
                /007_201-150_1_cfgcntlr-startup-config.new-148 context 201-150 type 1 
20:3304 => Apr 09 16:21:46: handle_cfg_sync_tftp_done_mts:2117 Received notification for context id 1
20:3305 => Apr 09 16:21:46: handle_cfg_sync_tftp_done_mts:2201 New startup config file for context 201-150 - 
                filename /isan/vegas/work/007_201-150_1_cfgcntlr-startup-config.new-148 size 5152 bytes
20:3306 => Apr 09 16:21:46: handle_cfg_sync_tftp_done_mts:2229 Copying startup file to /TN-CONFIG/startup_config.new
20:3307 => Apr 09 16:21:46: convert_cmd_for_peer:1083 Transformed lvl2 command peer ip address 50.0.0.12 255.255.255.0


show ft history ha_dp_mgr

This command shows the log of the Fault Tolerant (FT) Dataplane Manager (DM). It is primarily used for internal development purposes.

Sample Output

ACE30001/Admin# show ft history ha_dp_mgr

0:0 => Sep 18 19:01:47: ha_dp_debug_init:2520 Sdwrap sys init successful
0:1 => Sep 18 14:01:58: ha_dp_process_message:1761 Recd. Start HB
0:2 => Sep 18 14:01:58: ha_dp_establish_connections:406 Successfully added encap id 2 for  IF ID 5 for peer. Establishing 
                DP connections now..
0:3 => Sep 18 14:01:58: ha_dp_establish_connections:449 Connection setup passed on try 1. Inserting the table entry for 
                app 0  handle 0x1 in IXP0
0:4 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr: 0xc300000 data: 0x7fff6f28 Len 8
0:5 => Sep 18 14:01:58: ha_dp_establish_connections:489 Connection setup passed on try 1. Inserting the table entry for 
                app 1 handle 0x2 in IXP0
0:6 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr: 0xc300008 data: 0x7fff6f28 Len 8
0:7 => Sep 18 14:01:58: ha_dp_establish_connections:526 Connection setup passed on try 1. Inserting the table entry for 
                app 1  handle 0x10000001 in IXP1
0:8 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr: 0xc300008 data: 0x7fff6f28 Len 8
0:9 => Sep 18 14:01:58: ha_dp_establish_connections:673 Connection setup passed. Inserting the table entry for 
                app 2  handle1 0x3 handle2 0x4 in IXP0
0:10 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr: 0xc300010 data: 0x7fff6f28 Len 8
0:11 => Sep 18 14:01:58: ha_dp_establish_connections:733 Inserting the table entry for app 2  handle1 10000002 
                handle2 10000003 in IXP1
0:12 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr:  0xc300010 data: 0x7fff6f28 Len 8


Notes

This log wraps so it will not use up too much disk space and more importantly the output in the list is preserved over a reboot of the ACE.

While primarily used for internal debugging, this command shows information that can help general troubleshooting as well. For instance, if you have an FT-related problem involving ACE peer synchronization, you would typically be asked to invoke the show tech-support details command. The output of the show tech-support details command would include this FT log and others.

To examine a sample line from the log:

0:12 => Sep 18 14:01:58: ha_dp_write_to_table:298 ha_dp_write_to_table: Base addr: 0xc300010 data: 0x7fff6f28 Len 8

The item: "ha_dp_write_to_table:298" refers to the actual ACE code and "ha_dp_write_to_table" is a routine in the FT process which is executing and printing out information for DE to debug with. For this reason except for obvious errors which are self-explanatory, this log is typically only meaningful to internal development.


show ft history ha_mgr

Displays the HA manager debug log. This information can be useful for checking for correct HA state transitions in the device.

When high availability is enabled, the HA managers on both peers start an internal HA state machine and exchange redundancy protocol packets. Based on the priority configured for both sides, HA managers will go through the states transition, including non-redundant, election, standby-config, standby-bulk and eventually move its state to active and standby-hot state.

Configuration synchronization plays a very important role in the redundancy model. For ACE and ACE appliance, it can be detailed in two parts: bulk config-sync and incremental config-sync.

Sample Output

ACE30001/Admin# show ft history ha_mgr

0:0 => Sep 18 19:01:45: ha_debug_init:3343 Sdwrap sys init successfull
0:1 => Sep 18 19:01:48: ha_create_context:279 Context Admin, Context_id 0, sense 1
0:2 => Sep 18 19:01:48: create_context:2562 Adding Context Admin (0) len 5
0:3 => Sep 18 19:01:48: create_context:2567 Context Admin 0 has been added
0:4 => Sep 18 19:01:56: handle_mts_message:3461 HA MGR: Received MTS notif, from: 0x00000601/3, To: 0x00000601/0, 
                Opcode: MTS_OPC_SYSMGR_TIMEZONE_NOTIF(2509), MsgID: 1144
0:5 => Sep 18 14:01:58: ha_ft_peer_add_or_lookup:45 peer_id 0, sense 1
0:6 => Sep 18 14:01:58: fsm_peer_action:277 FSM: Peer 0, Current State FSM_PEER_STATE_INIT, Event FSM_PEER_EV_CREATE
0:7 => Sep 18 14:01:58: fsm_peer_create:1000 Creating Peer 1
0:8 => Sep 18 14:01:58: peer_state_change:56 Peer 0 Current State: FSM_PEER_STATE_INIT, New State: FSM_PEER_STATE_MY_IPADDR
0:9 => Sep 18 14:01:58: fsm_peer_create:1013 Starting server for Peer 0
0:10 => Sep 18 14:01:58: start_server:124 Configuring the TL Server Peer 0 Tcp Server 2000 2000
0:11 => Sep 18 14:01:58: start_server:140 Created Server Thread ID 1
0:12 => Sep 18 14:01:58: ha_ft_peer_update_heartbeat:193 peer id 0, hb_interval 100, hb_count 10, sense 1
0:13 => Sep 18 14:01:58: fsm_peer_action:265 Peer FSM State Change:No function specified. Peer 0, Current 
                State FSM_PEER_STATE_MY_IPADDR, Event FSM_PEER_EV_HB_UPDATE
0:14 => Sep 18 14:01:58: ha_ft_peer_update_heartbeat:193 peer id 0, hb_interval 100, hb_count 15, sense 1
0:15 => Sep 18 14:01:58: fsm_peer_action:265 Peer FSM State Change:No function specified. Peer 0, Current State 
                FSM_PEER_STATE_MY_IPADDR, Event FSM_PEER_EV_HB_UPDATE
0:16 => Sep 18 14:01:58: ha_ft_peer_update_my_ipaddr:68 peer id 0, ft vlan id 104, my_ipaddr 192.168.4.178, ifid 5, 
                sense 1
0:17 => Sep 18 14:01:58: fsm_peer_action:277 FSM: Peer 0, Current State FSM_PEER_STATE_MY_IPADDR, 
                Event FSM_PEER_EV_MY_IPADDR_ADD
0:18 => Sep 18 14:01:58: peer_state_change:56 Peer 0 Current State: FSM_PEER_STATE_MY_IPADDR, New State: 
                FSM_PEER_STATE_PEER_IPADDR
0:19 => Sep 18 14:01:58: ha_ft_peer_update_peer_ipaddr:122 peer id 0, ft vlan id 104, peer_ipaddr 192.168.4.160, sense 1
0:20 => Sep 18 14:01:58: fsm_peer_action:277 FSM: Peer 0, Current State FSM_PEER_STATE_PEER_IPADDR, Event 
                FSM_PEER_EV_PEER_IPADDR_ADD
0:21 => Sep 18 14:01:58: fsm_peer_send_start_heartbeat:358 Sending Start HB message to HA-DP module
0:22 => Sep 18 14:01:58: peer_state_change:56 Peer 0 Current State: FSM_PEER_STATE_PEER_IPADDR, New State: 
                FSM_PEER_STATE_START_HB
0:23 => Sep 18 14:01:58: ha_ft_peer_update_query_ipaddr:172 peer id 0, query vlan id 23, query ipaddr 10.86.215.160, sense 1
0:24 => Sep 18 14:01:58: fsm_peer_action:277 FSM: Peer 0, Current State FSM_PEER_STATE_START_HB, Event FSM_PEER_EV_HB_UPDATE
0:25 => Sep 18 14:01:58: fsm_peer_send_update_heartbeat:315 Sending Update HB message to  HA-DP module for peer 0
0:26 => Sep 18 14:01:58: ha_ft_group_add_or_lookup:215 ft_group_id 1, sense 1
0:27 => Sep 18 14:01:58: ha_ft_group_update_peer:231 ft_group_id 1, peer_id 0, sense 1
0:28 => Sep 18 14:01:58: ha_ft_group_update_preempt:242 ft_group_id 1, sense 0
0:29 => Sep 18 14:01:58: fsm_ft_action:238 FT FSM State Change:No function specified. Current State FSM_FT_STATE_INIT, 
                Event FSM_FT_EV_UPDATE
0:30 => Sep 18 14:01:58: ha_ft_group_update_config_priority:255 ft_group_id 1, config_priority 100, sense 1
0:31 => Sep 18 14:01:58: ft_group_update_config_priority:2427 Adjusted FT Group 1's net priority 100 FT config_priority 100
0:32 => Sep 18 14:01:58: fsm_ft_action:238 FT FSM State Change:No function specified.  Current State FSM_FT_STATE_INIT, 
                Event FSM_FT_EV_UPDATE

Field descriptions

The states are:

Event Description
FSM_EV_PEER_UP This event is posted when the peer device is detected to be up.
FSM_EV_PEER_DOWN This event is posted when the peer device goes down
FSM_EV_PEER_FT_VLAN_DOWN This event is posted when the peer device is up but the FT VLAN is down.
FSM_EV_FT_STATE This event is posted when an STATE_UPDATE message is received from the peer.
FSM_EV_TIMEOUT This event will indicate a timer expiry.
FSM_EV_CFG_SYNC_DONE This event is posted to the FSM when receiving the configuration data is completed.
FSM_EV_BULK_SYNC_DONE This event is posted to the FSM when receiving the bulk sync data is completed.
FSM_EV_COUP This event is posted when an COUP message is received from the peer.
FSM_EV_RELINQUISH This event is posted when an RELINQUISH message is received from the peer.
FSM_EV_TRACK_STATUS This event is posted when the status of a tracked object is changed


show ft memory

Shows the number of memory allocations as well as the number of bytes of memory used by those allocations for various HA components. This command shows information about tracked memory consumed by HA and the internal libraries it uses.


Sample Output

ACE5/Admin# show ft memory

show mem stats

show mem stats 0

Mem stats for UUID : Malloc track Library(103) Max types: 5
--------------------------------------------------------------------------------
Curr alloc: 168 Curr alloc bytes: 7576(7k)

Mem stats for UUID : Non mtrack users(0) Max types: 31
--------------------------------------------------------------------------------
Curr alloc: 17 Curr alloc bytes: 33689(32k)

Mem stats for UUID : libsdwrap(115) Max types: 24
--------------------------------------------------------------------------------
Curr alloc: 11 Curr alloc bytes: 40388(39k)

Mem stats for UUID : ACE HA Manager(332) Max types: 6
--------------------------------------------------------------------------------
Curr alloc: 4 Curr alloc bytes: 640(0k)

Curr alloc: 200 Curr alloc bytes: 82293 (80k)

Stats Reference

The "alloc" value is the number of objects of that TYPE created.


show ft memory detail

This command is a more verbose version of the show ft memory command. These commands track the memory usage of the Fault Tolerant (FT) application. This detailed form of the command lists the actual name of the FT application doing the memory allocation. The first are those FT application compiled with the MTRACK debugging feature while the second group are non-MTRACK users which are lower level library files.

This command can be used to look for memory leaks within the FT subsystem of the ACE.


Sample Output

ACE30001/Admin# show ft memory detail

show mem stats

show mem stats 1

Mem stats for UUID : Malloc track Library(103) Max types: 5
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
   0 MT_MEM_other                                10     10       3212       321
   1 MT_MEM_mtrack_default                        0      0          0          
   2 MT_MEM_mtrack_hdl                            4      5       2752       290
   3 MT_MEM_mtrack_info                          80    120       1280       192
   4 MT_MEM_mtrack_lib_name                     120    160       3561       474
--------------------------------------------------------------------------------
Total bytes: 10805 (10k)
--------------------------------------------------------------------------------

Mem stats for UUID : Non mtrack users(0) Max types: 40
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
   0 [r-xp]/isan/lib/libavl.so                    3      7         48        11
   1 [r-xp]/isan/lib/libbmp.so.0.0.0              0      0          0          
   2 [r-xp]/isan/lib/libcfgsync.so                0      0          0          
   3 [r-xp]/isan/lib/libdebug_history.so          1      1     204800     20480
   4 [r-xp]/isan/lib/libfileutil.so               0      0          0          
   5 [r-xp]/isan/lib/libglib-2.0.so.0.400.0       3      3       2108       210
   6 [r-xp]/isan/lib/libif_index.so               0      0          0          
   7 [r-xp]/isan/lib/libipcp.so                   0      0          0          
   8 [r-xp]/isan/lib/libmtrack.so                 0      0          0          
   9 [r-xp]/isan/lib/libmts.so.0                  0      0          0          
  10 [r-xp]/isan/lib/libncurses.so.5              0      0          0          
  11 [r-xp]/isan/lib/libpfm_intf.so               0      0          0          
  12 [r-xp]/isan/lib/libpss.so                    4      4      10476      1047
  13 [r-xp]/isan/lib/libsdwrap.so.0               0      0          0          
  14 [r-xp]/isan/lib/libsrg.so                    0      1          0        84
  15 [r-xp]/isan/lib/libsse_common.so             0      0          0          
  16 [r-xp]/isan/lib/libsvir.so                   0      0          0          
  17 [r-xp]/isan/lib/libsyscall.so                0      0          0          
  18 [r-xp]/isan/lib/libsyserr.so                 0      0          0          
  19 [r-xp]/isan/lib/libsyslib.so                 0      0          0          
  20 [r-xp]/isan/lib/libsysmgr.so.0.0.0           0      0          0          
  21 [r-xp]/isan/lib/libsysmgrcmn.so              0      0          0          
  22 [r-xp]/isan/lib/libsysstr.so                 0      0          0          
  23 [r-xp]/isan/lib/libtecla.so                  0      0          0          
  24 [r-xp]/isan/lib/libtl.so                     0      0          0          
  25 [r-xp]/isan/lib/libtnrpc.so                  0      2          0         3
  26 [r-xp]/isan/lib/liburi_copy.so.0.0.0         0      0          0          
  27 [r-xp]/isan/lib/liburi_map.so                0      0          0          
  28 [r-xp]/isan/lib/liburiparse.so               0      0          0          
  29 [r-xp]/isan/lib/libutil_cli.so.0.0.0         0      0          0          
  30 [r-xp]/isan/lib/libutils.so.0                2      2         42         4
  31 [r-xp]/isan/lib/libvdb.so                    0      0          0          
  32 [r-xp]/isan/lib/libvsh_util.so               0      0          0          
  33 [r-xp]/itasca/bin/ha_mgr                     0      0          0          
  34 [r-xp]/lib/ld-2.3.2.so                       0      1          0         8
  35 [r-xp]/lib/libc-2.3.2.so                     6      8        513       126
  36 [r-xp]/lib/libdl-2.3.2.so                    1      1         16         1
  37 [r-xp]/lib/libpthread-0.10.so                2      2        140        14
  38 [rwxp]0x10003000-0x10018000                  0      0          0          
  39 [rwxp]0x7ffd2000-0x7fff8000                  0      0          0          
--------------------------------------------------------------------------------
Total bytes: 218143 (213k)
--------------------------------------------------------------------------------

Mem stats for UUID : libsdwrap(115) Max types: 24
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
   0 LIBSDWRAP_SYSLOG_FLTR_CMI                    0      0          0          
   1 LIBSDWRAP_DBG_ELEM_ARRAY                     5      5       2912       291
   2 LIBSDWRAP_HIST_EVT_T                         1      1        172        17
   3 LIBSDWRAP_DBG_PSS_BUF                        0      0          0          
   4 LIBSDWRAP_DBG_SHOW_BUF                       0      0          0          
   5 LIBSDWRAP_HIST_SHOW_BUF                      0      0          0          
   6 LIBSDWRAP_DBGDUMP_BUF                        0      0          0          
   7 LIBSDWRAP_HIST_HDL                           0      0          0          
   8 LIBSDWRAP_HIST_INSTHDLPTR                    1      1          4          
   9 LIBSDWRAP_HIST_INST_HDL                      1      1        160        16
  10 LIBSDWRAP_HIST_FILESAVE_BUF                  0      0          0          
  11 LIBSDWRAP_HIST_REC                           1      1      32768      3276
  12 LIBSDWRAP_HIST_LOG_TMPBUF                    0      0          0          
  13 LIBSDWRAP_FILE_BUF                           0      0          0          
  14 LIBSDWRAP_FILE_DBGFLAGS_BUF                  0      0          0          
  15 LIBSDWRAP_FILE_EVLOG_BUF                     0      0          0          
  16 LIBSDWRAP_FILE_VTYFLAGS                      0      0          0          
  17 LIBSDWRAP_HIST_EVTBUF                        0      0          0          
  18 LIBSDWRAP_SYS_PSSBUF                         0      0          0          
  19 LIBSDWRAP_SYS_SHOW_BUF                       0      0          0          
  20 LIBSDWRAP_DBGELEM_DESC_ARRAY                 0      0          0          
  21 LIBSDWRAP_DBG_ELEM_ARRAY_PTR                 0      0          0          
  22 LIBSDWRAP_DBG_CONTEXT_STRUCTURE              4      4       6688       668
  23 LIBSDWRAP_FILE_HEADER                        0      0          0          
--------------------------------------------------------------------------------
Total bytes: 42704 (41k)
--------------------------------------------------------------------------------

Mem stats for UUID : ACE HA Manager(332) Max types: 6
--------------------------------------------------------------------------------
TYPE NAME                                           ALLOCS                 BYTES
                                               CURR    MAX       CURR        MAX
   1 HA_MTRACK_ft_group                           4      4       1776       177
   2 HA_MTRACK_context                            4      4        384        38
   3 HA_MTRACK_track                              0      0          0          
   4 HA_MTRACK_track_probe                        0      0          0          
   5 HA_MTRACK_history                            0      0          0          
--------------------------------------------------------------------------------
Total bytes: 2160 (2k)
--------------------------------------------------------------------------------
Grand total bytes: 273812 (267k)

show mem stats rc 0


show ft peer detail

Shows detailed information relating to the peer fault tolerant state.


Sample Output

ACE30001/Admin# show ft peer detail

Peer Id                      : 1
State                        : FSM_PEER_STATE_COMPATIBLE
Maintenance mode             : MAINT_MODE_OFF
FT Vlan                      : 104
FT Vlan IF State             : UP
My IP Addr                   : 192.168.4.178
Peer IP Addr                 : 192.168.4.160
Query Vlan                   : 23
Query Vlan IF State          : UP
Peer Query IP Addr           : 10.86.215.160
Heartbeat Interval           : 100
Heartbeat Count              : 15
Tx Packets                   : 103194
Tx Bytes                     : 23586495
Rx Packets                   : 103253
Rx Bytes                     : 23583317
Rx Error Bytes               : 0
Tx Keepalive Packets         : 103125
Rx Keepalive Packets         : 103126
TL_CLOSE count               : 0
FT_VLAN_DOWN count           : 0
PEER_DOWN count              : 1
SRG Compatibility            : COMPATIBLE
License Compatibility        : COMPATIBLE
FT Groups                    : 4


Notes


Field Description
Peer Id Peer identifier
State Peer state - my be one of the following:
  • FSM_PEER_STATE_INIT – Initial State
  • FSM_PEER_STATE_MY_IPADDR – Local IP address needs to be configured
  • FSM_PEER_STATE_PEER_IPADDR – Peer IP address needs to be configured
  • FSM_PEER_STATE_START_HB – Peer configuration is complete. Heartbeats are sent to check for peer device.
  • FSM_PEER_STATE_TL_SETUP – Peer has been detected and is UP. HA is in the process of establishing a TCP connection to the Peer.
  • FSM_PEER_STATE_SRG_CHECK – Checking for Version compatibility with the Peer
  • FSM_PEER_STATE_LIC_CHECK – Checking for License compatibility with the Peer
  • FSM_PEER_STATE_COMPATIBLE – All checks are complete. The Peer is compatible and ready for redundancy.
  • FSM_PEER_STATE_FT_VLAN_DOWN – FT VLAN is down. Through alternate interface Peer is detected to be alive.
  • FSM_PEER_STATE_DOWN – The Peer device has gone down. Cannot be reached via alternate interface as well.
  • FSM_PEER_STATE_ERROR – Error has occurred as part of State Machine progression such as Version Mismatch and failure in establishing TCP connection to peer.
  • FSM_PEER_STATE_TL_ERROR – TL Connection failure
Maintenance mode May be one of the following:
  • MAINT_MODE_FULL – All contexts on the service blade will become non-redundant causing their peer contexts to switch over to ACTIVE. This mode will be seen right before re-setting/rebooting the service blade mainly for performing hitless upgrades
  • MAINT_MODE_PARTIAL – All STANDBY contexts will transition to STANDBY_COLD state. This mode is entered if configuration sync failure occurs
  • MAINT_MODE_OFF – Maintenance mode is off
Heartbeat Interval Interval in seconds
Heartbeat Count Consecutive misses before declaring a failure
TL_CLOSE count Number of TL_CLOSE messages received from peer
FT_VLAN_DOWN count Number of times FT vlan went down
PEER_DOWN count Number of times peer was declared down
SRG Compatibility Indicates whether the software versions of the local ACE and the peer ACE are compatible. Possible states are:
INIT Compatibility is being negotiated.
COMPATIBLE Software versions of ACE peers are compatible.
INCOMPATIBLE Software versions of peers are incompatible.
WARM_COMPATIBLE See Troubleshooting Redundancy for details.
License Compatibility Indicates whether the licenses of the local ACE and the peer ACE are compatible. Possible states are: INIT, COMPATIBLE, or INCOMPATIBLE. These have similar meanings to the SRG compatibility.
FT Groups Number of FT groups


show ft peer status

Displays the current operating status of the peer. In general, the counters displayed by this command are not relevant to troubleshooting, since this command shows states.


Sample Output

ACE5/Admin# show ft peer status

Peer Id                      : 1
State                        : FSM_PEER_STATE_COMPATIBLE
Maintenance mode             : MAINT_MODE_OFF
SRG Compatibility            : COMPATIBLE
License Compatibility        : COMPATIBLE
FT Groups                    : 1

Notes

Field Description
Peer ID Identifier of the remote context in the FT group.
SRG Compatibility Status of whether the software version of the local ACE and the software version of the peer ACE are compatible. Possible states are:
  • INIT
  • COMPATIBLE
  • INCOMPATIBLE
License Compatibility Status of whether the license of the local ACE and the license of the peer ACE are compatible. Possible states are:
  • INIT
  • COMPATIBLE
  • INCOMPATIBLE
FT Groups Number of FT groups.


show ft stats

Displays FT-related statistics for an FT peer.

Sample Output

ACE5/Admin# show ft stats 
HA Heartbeat Statistics 
------------------------

Number of Heartbeats Sent                 : 242734
Number of Heartbeats Received             : 242787
Number of Heartbeats Missed               : 516   
Number of Unidirectional HB's Received    : 569
Number of HB Timeout Mismatches           : 0
Num of Peer Up Events Sent                : 1
Num of Peer Down Events Sent              : 0
Successive HB's miss Intervals counter    : 0
Successive Uni HB's recv counter          : 0

Notes

The counter of interest here is "Number of Unidirectional HB's Received". This is the number of heartbeats received by the local peer that indicate the remote peer is not receiving HB signals. In other words, the remote peer is sending heartbeats, but not receiving any. Note that both peer modules send heartbeat packets and each packet indicates whether the other peer has been receiving heartbeats.

For more information, see the | Cisco Application Control Engine Module Administration Guide.


show ft track status

Displays the status of failure detection tracking by the ACE of monitored network items.

Sample Output

ACE5/Admin# show ft track status 

FT Group                     : 1
Status                       : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
My Config Priority           : 90
My Net Priority              : 90
My Preempt                   : Disabled
Context Name                 : Admin
Context Id                   : 0

Notes

In general, this command is used for internal development purposes and is not useful for troubleshooting.


show hm-internal evmgr-stats

Provides the queuing information for the Health Manager's internal event manager.

Sample Output

ace3/Admin# show hm-internal evmgr-stats

  Health Monitor Event Manager Stats

Msg Sent             :          0
Msg Dropped          :          0 (0.0%)
Msg Received         :          0
  Good               :          0
  Bad Opcode         :          0
  Bad Event Instance :          0

Msg Allocated        :          0
Msg Allocate Failed  :          0
Msg Freed            :          0

Total Events         :          0
Total Events Skipped :          0 (0.0%)


Notes

Field Description
Msg Sent Event messages sent. This is expected. (Note that a corresponding WrkThread's "Msg Received" counter will be incremented for each "Msg Sent" event.)
Msg Dropped The incrementing of this counter may indicate a problem. Contact Cisco TAC for more information.
Msg Received Messages received.
Good Incrementing of this counter is expected upon completion of a probe. (Note: A corresponding WrkThread's “Msg Send” will be incremented as well.)
Bad Opcode You should contact Cisco TAC if this field is incrementing.
Bad Event Instance You should contact Cisco TAC if this field is incrementing.
Msg Allocated Messages allocated.
Msg Allocate Failed Number of times message allocation failed.
Msg Freed Messages freed
Total Events The total number of events.
Total Events Skipped This event may occur when a probe is still being run and hence the internal event manager does not send a message to WrkThread (i.e., it skips the probe for that interval time). For example, this is possible if the probe interval is less than the open/receive timeout, such as: probe interval (2sec) < open/receive timeout (5sec).

If the server is unreachable, the probe will continue to wait for 5 seconds before it sends a response back to event manager. The event manager in the meantime will skip the probe twice (at 2sec and 4sec).


show hm-internal icmp-stats

Shows statistical information regarding the activities of the Health Manager's ICMP probes.

Sample Output

ace3/Admin# show hm-internal icmp-stats

  Health Monitor ICMP Stats

ICMP packets Sent    :          0
Msg send fails       :          0
ICMP packets recv    :          0
ICMP Host unreachable:          0
ICMP encap-decap err :          0
  num_len            :          0
  Dest unreachable   :          0
  Time exceeded      :          0
  Redirect           :          0
  Other              :          0
  echo_req           :          0
  echo_resp          :          0
  num_stale          :          0
  num_short          :          0
  num_long           :          0

Notes

Field Description
ICMP packets Sent Total ICMP probes sent
Msg send fails Number of times the ICMP probes failed to be sent
ICMP packets recv ICMP probes received
ICMP Host unreachable Number of probe failures due to ICMP Host unreachable
ICMP encap-decap err Number of probe failures due to encap-decap errors
num_len Number of probe failures due to length errors
Dest unreachable Number of probe failures due to destination unreachable
Time exceeded Number of probe failures due to time exceeded
Redirect Number of probe failures due to redirect
Other Number of probe failures due to other failures
echo_req Number of ICMP echo requests
echo_resp Number of ICMP echo responses
num_stale Number of probe failures due to stale
num_short Number of probe failures due to too short
num_long Number of probe failures due to too long


show hm-internal wrkthread-stats

This command shows information on activity of the health monitor process by thread. Threads 1 and 2 are for generic TCP and UDP probes. Thread 3 is for ICMP probe. Thread 4 is for SNMP probe and thread 5 is for scripted probes.

In addition to threads, an event scheduler exists which is mainly responsible for firing probes at configured intervals. The worker threads communicate to the event manager via a pipe.

Sample Output

ace3/Admin# show hm-internal wrkthread-stats

  Health Monitor Worker Threads Stats
                          Thread-1   Thread-2   Thread-3   Thread-4   Thread-5       Total
Msg Sent              :          0          0          0          0          0           0
Msg Dropped           :          0          0          0          0          0           0
Msg Dropped (%)       :       0.0%       0.0%       0.0%       0.0%       0.0%        0.0%
Msg Received          :          0          0          0          0          0           0
  Usage               :          0          0          0          0          0           0
  Bad Opcode          :          0          0          0          0          0           0
  Cancel Probe        :          0          0          0          0          0           0
  Run Probe           :          0          0          0          0          0           0
  Run Probe (Bad Data):          0          0          0          0          0           0
Msg Allocated         :          0          0          0          0          0           0
Msg Allocate Failed   :          0          0          0          0          0           0
Msg Freed             :          0          0          0          0          0           0

Error Create Qnode    :          0          0          0          0          0           0
Error Connect Probe   :          0          0          0          0          0           0
Error Add ICMP Qnode  :          0          0          0          0          0           0

Qnode Created         :          0          0          0          0          0           0
Qnode Destroyed       :          0          0          0          0          0           0
Qnode in All Queues   :          0          0          0          0          0           0
Qnode in Queues < 0   :          0          0          0          0          0           0
Qnode cant be deleted :          0          0          0          0          0           0
Qnode not in queue    :          0          0          0          0          0           0
Invalid socket value while removing
                      :          0          0          0          0          0           0
Invalid socket value while adding
                      :          0          0          0          0          0           0
Active sockets        :          0          0          0          0          0           0


Notes

Field Description
Communication statistics between worker threads and the event Manager:
Msg sent This value is the number of messages sent by the respective worker thread to the event manager.
Msg Dropped This is value represents number of errors encountered during communication from a worker thread to the event manager.
Msg Dropped (%) The sum of Msg Sent and Msg Dropped represents the total number of communication messages sent from the worker thread to the event manager. This value represents the Msg dropped counter as a percentage. Its dervied as follows (Msg Dropped * 100)/(Msg sent + Msg Dropped).
Msg Received This value is the number of messages received from the event manager to the respective Worker thread. The format of the event manager message has OPCODE, probe related data as a part of the message. The following errors pertain to the parsing of the above fields.
Usage This represents the number of messages received with "Usage OPCODE" in the OPCODE field. It is unused.
Bad Opcode The number of messages with invalid OPCODE, which is not recognized by the worker thread.
Cancel Probe The number of messages with "Cancellation OPCODE" from the event manager. Its not used.
Run Probe The number of valid messages received from the event manager. The worker thread can process this message effectively.
Run Probe(Bad Data) The number of message which does not contain valid probe related data.
Errors/counters related to memory (A message is sent back to the event manager after a worker thread processes the probe. This message is allocated by the worker thread and, under normal circumstances, freed in the event manager.)
Msg Allocated This value is the number of messages that are successfully allocated.
Msg Allocate Failed This value is the number of failures during message allocation.
Msg Freed This value represents the number of messages that were freed by the event manager.
A probe that is being run is represented in the form of a qnode. The qnodes are created after receiving a message from the event manager and are destroyed after a message is sent back to the event manager with the probe result.
Qnode Created This value is the number of qnodes that are successfully allocated.
Qnode Destroyed This value is the number of qnodes that are successfully freed.
Error Create Qnode This value is the memory allocation failure encountered during the qnode creation.
For optimization purposes, these qnodes are stored in an array for ICMP probes and as a queue for non-ICMP probes. Since ICMP qnodes are an array, it has a capacity to hold 8K probes at any given point of time.
Error Add ICMP qnode This value is encountered when the number of simultaneous ICMP probes exceeds 8K.
Qnode in All Queues This value is the number of qnodes that are in queue currently in that thread.
Qnode in queues < 0 This value is incremented, when a qnode is getting destroyed twice. In the normal conditions it should not be encountered.
Qnodes cant be deleted This value is incremented, if a qnode we are trying to delete is not in the queue. Under normal conditions, this should not be encountered.
Error connect Probe For TCP/UDP based probes, every probe run creates a socket. If there is any error in the creation of a socket or a subsequent connect with the real server, this error is incremented. Its usually seen in scalable configurations.
Miscellaneous statistics (During a probe run, after socket creation the corresponding qnodes are put in a queue for optimization. The value of the socket is checked for sanity. These counters are incremented using that. These values are for purely internal consumption and should not be considered during debugging.)'
Active Sockets This number indicates the number of sockets that are currently opened.


show hyp stat

Available only from the admin context, this command shows contents of some of the more useful Hyperion registers. Hyperion is the ASIC that passes data between the "ten gig ethernet" (that is, the data path on the cat, which is either 8 GB or 20 GB wide) and the CDE (classification and distribution engine) FPGA which directs traffic to/from all the components of the ACE.

The registers are cleared when read. Thus, every time you read them by issuing this command, you will get the difference since the last time the command was issued.

Sample Output

switch/Admin# show hyp stat
0x6d1 hyp DDR tx -> CDE pkt cnt 0x1
0x6d2 hyp DDR tx -> CDE pkt cnt 0x1
0x679 hyp DDR rx (hi pri) <- CDE pkt cnt 0x0
0x67a hyp DDR rx (hi pri) <- CDE pkt cnt 0x0
0x67b hyp DDR rx (low pri) <- CDE pkt cnt 0x0
0x67c hyp DDR rx (low pri) <- CDE pkt cnt 0x0
0x5a9 hyp DDR rx <- TITAN pkt cnt 0xc050
0x5aa hyp DDR rx <- CDE pkt cnt 0xada8
0x20f DDR forward to fabric pkt cnt (COR) 0xffff
0x210 DDR forward to DDR out pkt cnt (COR) 0xa3a2
0x13d fwd to earl pkt cnt 0x1
0x13e fwd to earl pkt cnt 0x1
0x13f fwd to earl pkt cnt dhpm 0x0
0x140 fwd to earl pkt cnt dhpm 0x0
0x112 Fabric xmit pkt cnt 0x0
0x113 Fabric xmit pkt cnt 0x0

Notes

Notice that it takes two adjacent registers to display "hyp DDR tx -> CDE pkt cnt". This is traffic from the hyperion to the CDE. The arrows depicted in the output are meaningful, in other words.

In general, this should rarely be needed for troubleshooting. It exists primarily for internal development purposes.


show interface

Displays the status of the ACE interface. Typically used in these forms:

  • show interface
  • show interface vlan n

Counter values are organized by VLAN.

To clear the counters, use the following commands:

  • clear interface, to clear all VLAN interfaces
  • clear interface vlan n, to clear a specific VLAN interface


Sample Output

ACE30001/Admin# show interface

vlan23 is up
  Hardware type is VLAN
  MAC address is 00:1d:70:d1:62:91
  Virtual MAC address is 00:0b:fc:fe:1b:01
  Mode : routed
  IP address is 10.86.215.178 netmask is 255.255.255.0
  FT status is active
  Description:Management
  MTU: 1500 bytes
  Last cleared: never
  Alias IP address not set
  Peer IP address is 10.86.215.160 Peer IP netmask is 255.255.255.0
  Assigned from the Supervisor, up on Supervisor
     71627 unicast packets input, 6949140 bytes
     17 multicast, 11 broadcast
     0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
     45928 unicast packets output, 10928026 bytes
     4 multicast, 2109 broadcast
     0 output errors, 0 ignored

Notes

Field Description
INPUT or Received on the VLAN interface
unicast packets input Packets received to a UNICAST address on this VLAN.
bytes The number of UNICAST bytes received on this VLAN.
multicast The number of MULTICAST packets received on this VLAN.
broadcast The number of BROADCAST packets received on this VLAN.
input errors The sum of all errors that prevented the receipt of a packet (or datagrams) and include CRC, Overrun, Underrun and Aborted Frames.
unknown The number of packets dropped on input because of an unknown protocol.
ignored Number of received packets ignored by the VLAN because the interface hardware ran low on internal buffers.
unicast RPF drops The number of UNICAST packets which were dropped due to the "Unicast Reverse Path Forwarding (RPF)" feature being able to verify the IP source address. Related to certain type of Denial of Service (DOS) attacks on the network.
OUTPUT or Transmitted on the VLAN interface
unicast packets output Packets transmitted to a UNICAST address on this VLAN.
bytes The number of UNICAST bytes transmitted on this VLAN.
multicast The number of MULTICAST packets transmitted on this VLAN.
broadcast The number of BROADCAST packets received on this VLAN.
output errors The sum of all errors that prevent a packet from being transmitted on this interface and includes CRC, Overrun, Underun and Aborted Frames.
ignored Number of packets failed to be transmitted by the VLAN because the interface hardware ran low on internal buffers.


show interface internal iftable

This command displays information about the control plane interface table.

Sample Output

show interface internal iftable <interface>

ACE30001/Admin# show interface internal iftable vlan23
vlan23
--------
ifid:           8
Context:        0
ifIndex:        16777239
physid:         23
rmode:          1 (routed)
iftype:         0 (vlan)
IP:             10.86.215.178
Alias IP:       0.0.0.0
Standby IP:     10.86.215.160
bvi_bgid:       0
MTU:            1500
MAC:            00:1d:70:d1:62:91
VMAC:           00:0b:fc:fe:1b:01
Flags:          0xf4028800 (valid, shared, up, admin-up, Active)
ACL In:         10
ACL Out:        0
Route ID:       0
FTgroupID:      1
Zone ID:        8
Sec Level:      0
L2 ACL:         bpdu DENY, ipv6 DENY, mpls DENY, all DENY

LastChange:     1248091338 (Mon Jul 20 12:02:18 2009)
iflookup index: 23
vlan-vmac index:2
Next Shared IF: 19
Lock:           Unlocked, seq 21
Lock errors:    0
Unlock errors:  0
No. of times locked:    21
No. of times unlocked:  21
Current/last owner:     0x40ba8c

Notes

Field Description
ifid Interface ID
Context Context ID
ifIndex Interface Index
physid Physical ID
rmode Routing mode. 1 = routed.
iftype Interface type. 0 = vlan
IP IP address
Alias IP Alias IP address
Standby IP Standby IP address
bvi_bgid Internally translated version of bridge-group id if this interface is a bvi
MTU Maximum Transmission Unit (bytes)
MAC MAC address
VMAC Virtual MAC address
Flags This is typically useful if this table shows something you didn't expect. The meanings of the active flags are displayed along with the flag value.
ACL In Inbound Access Control List ID
ACL Out Outbound Access Control List ID
Route ID Route ID
FTgroupID Fault Tolerant Group ID
Zone ID Zone ID
Sec Level Security Level
L2 ACL Layer 2 Access Control Lists: bpdu DENY, ipv6 DENY, mpls DENY, all DENY
LastChange Time of last change (unix time and human-readable)
iflookup index Interface Lookup Index
vlan-vmac index Virtual MAC address index
Next Shared IF The next interface ID in the chain of shared vlans.
Lock Locked / Unlocked
Lock errors Number of lock errors
Unlock errors Number of unlock errors
No. of times locked Number of times locked
No. of times unlocked Number of times unlocked


show interface internal vlantable

Displays VLAN-related system information for all or a specified VLAN. VLAN information is displayed one line for every possible VLAN ID 1-4094. If you specify a VLAN number, information for just that VLAN is displayed.

Sample Output

ace3/Admin# show interface internal vlantable
vlan sup_en autost type blocked    pri  first_if no_lifs
---- ------ ------ ---- -------    ---  -------- -------
1    0      0      normal     0       1    0        0
2    0      0      normal     0       2    0        0
3    0      0      normal     0       3    0        0
4    0      0      normal     0       4    0        0
5    0      0      normal     0       5    0        0
6    0      0      normal     0       6    0        0
7    0      0      normal     0       7    0        0
8    0      0      normal     0       8    0        0
9    0      0      normal     0       9    0        0
10   1      1      normal     0       10   1        1
... 
4092 0      0      normal     0       4092 0        0
4093 0      0      normal     0       4093 0        0
4094 0      0      normal     0       4094 0        0

Notes

Field Description
vlan VLAN ID
sup_en Is the VLAN enabled on the sup: 0 = no, 1 = yes
autost Autostate on the SUP
type PVLAN type
blocked STP status on the sup: 0 = not blocked, 1 = blocked
pri Primary of this VLANs (pvlans)
first_if First interface ID on this VLAN
no_lifs Number of interfaces (>1 if it's a shared vlan)


show ip fib

Shows the contents of the forwarding information database (FIB).

Sample Output

CE30001/Admin# show ip fib

FIB for Context Admin (RouteId 0)

   Codes: H - host,   I - interface
          S - static,      N - nat
          A - need arp resolve,      E - ecmp

Destination         Interface         EncapId  Flags
------------------------------------------------------------------------
0.0.0.0             vlan23                 8   S [0xc]
224.0.0.0/3         N/A                 DROP   N/A [0x100]
127.1.0.0/16        vlan1                  1   SI [0x18]
10.86.215.0/24      vlan23                 0   IA [0x30]
192.168.2.0/24      vlan102                0   IA [0x30]
192.168.219.0/24    vlan101                0   IA [0x30]
192.168.4.0/24      vlan104                0   IA [0x30]
10.86.215.208/32    vlan23                40   H [0x3]
10.86.215.178/32    N/A                 DROP   N/A [0x10]

...

192.168.2.231/32    vlan102               49   H [0x3]
10.86.215.150/32    vlan23                50   H [0x3]

Total route entries = 43

Notes

Field Description
Destination The destination address for the route and mask
Interface The VLAN interface (N/A for 'drop' entries)
Encap ID The encapsulation identifier. To look at the encapsulation entry, enter the command show np 1/2 me-stats -e0.
Flags These identify the route type and state:
  • H indicates a host route.
  • I indicates interface route.
  • S indicates a static route.
  • N indicates a NAT route.
  • A indicates that the route needs an ARP resolve.
  • E indicates an ECMP route.


show ip route

Shows the IPv4 routing table on the ACE.


Sample Output

switch/Admin# show ip route 

Routing Table for Context Admin (RouteId 0)

   Codes: H - host,   I - interface
          S - static,      N - nat
          A - need arp resolve,      E - ecmp

Destination         Gateway          Interface         Flags
------------------------------------------------------------------------
0.0.0.0             10.86.215.1      vlan5             S [0xc]
10.86.215.0/24      0.0.0.0          vlan5             IA [0x30] 

Total route entries = 2

Notes

Field Description
H Entry is for a remote host
I Entry for locally connected interface/subnet.
S Statically defined route.
N Local NAT Pool address.
A Route is configured but ARP response has not been received.
E Equal cost route (as expected, it's usual to have more than one).


show ip traffic

Shows information related to IP traffic handled by the device.

Sample Output

ACE30001/Admin# show ip traffic
IP statistics:
        Rcvd :  5663692 total, 392622514 bytes
                0 input errors, 0 no route
                0 unknown protocol
        Frags:  0 reassembled, 0 couldn't reassemble
                0 fragmented, 0 couldn't fragment
        Bcast:  45161 received, 0 sent
        Mcast:  112928 received, 0 sent
        Sent :  5589483 total, 398940208 bytes
                0 no route
        Drop :  0 no route, 0 out discarded
ICMP statistics:
        Rcvd : 0 redirects,  2554 unreachable
                125 echo, 1 echo reply, 0 mask requests, 0 mask replies, 0 quench
                0 parameter, 0  timestamp
        Sent : 0 redirects,  24 unreachable, 0 echo, 125 echo reply
                0 mask requests, 0 mask replies, 0 quench, 0 timestamp
                0 parameter, 0  time exceeded 
TCP statistics:
        Rcvd : 0 total,  0 errors
        Sent : 214302 total
UDP statistics:
        Rcvd : 19 total,  0 errors, 24 no port
        Sent : 30389 total
ARP statistics:
        Rcvd : 2224379 packets 0 Errors 9289 requests 9223 responses
        Sent : 16865 packets 0 Errors 7550 requests 9289 responses

Notes

The output includes details for both control plane and data plane traffic. To view only the CP traffic, you can load the dplug and issue the netstat -s command.

The specific reasons for the "couldn't reassemble" counter being incremented can be shown using the show np 1/2 me-stats -sreass | inc Drop command. You can get a breakdown of the total fragment errors by issuing the "show frag" command.

Rating: 2.3/5 (3 votes cast)

Personal tools