Cisco Application Control Engine (ACE) Troubleshooting Guide -- Show Counter Reference -- Command Set 5

From DocWiki

(Difference between revisions)
Jump to: navigation, search
m
 
Line 1: Line 1:
 +
{| align="right" border="1" cellspacing = "0"
 +
|align="center"|'''Guide Contents'''
 +
|-
 +
|[[Cisco Application Control Engine (ACE) Troubleshooting Guide|Main Article]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Overview of ACE Troubleshooting|Overview of ACE Troubleshooting]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Understanding the ACE Module Architecture and Traffic Flow|Understanding the ACE Module Architecture and Traffic Flow]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Preliminary ACE Troubleshooting|Preliminary ACE Troubleshooting]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting ACE Boot Issues|Troubleshooting ACE Boot Issues]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting with ACE Logging|Troubleshooting with ACE Logging]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Connectivity|Troubleshooting Connectivity]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Ethernet Ports|Troubleshooting ACE Appliance Ethernet Ports]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Remote Access|Troubleshooting Remote Access]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Access Control Lists|Troubleshooting Access Control Lists]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Network Address Translation|Troubleshooting Network Address Translation]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting ACE Health Monitoring|Troubleshooting ACE Health Monitoring]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Layer 4 Load Balancing|Troubleshooting Layer 4 Load Balancing]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Layer 7 Load Balancing|Troubleshooting Layer 7 Load Balancing]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Redundancy|Troubleshooting Redundancy]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting SSL|Troubleshooting SSL]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Compression|Troubleshooting Compression]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Troubleshooting Performance Issues|Troubleshooting Performance Issues]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- ACE Resource Limits|ACE Resource Limits]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Managing Resources|Managing ACE Resources]]<br>[[Cisco Application Control Engine (ACE) Troubleshooting Guide -- Show Counter Reference|Show Counter Reference]]<br>
 +
|}
 +
 +
__TOC__
 +
== show probe detail ==
== show probe detail ==

Latest revision as of 21:50, 11 March 2011

Guide Contents
Main Article
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Troubleshooting Connectivity
Troubleshooting ACE Appliance Ethernet Ports
Troubleshooting Remote Access
Troubleshooting Access Control Lists
Troubleshooting Network Address Translation
Troubleshooting ACE Health Monitoring
Troubleshooting Layer 4 Load Balancing
Troubleshooting Layer 7 Load Balancing
Troubleshooting Redundancy
Troubleshooting SSL
Troubleshooting Compression
Troubleshooting Performance Issues
ACE Resource Limits
Managing ACE Resources
Show Counter Reference

Contents


show probe detail

Displays configuration information and statistics for a probe.

Sample Output

ACE30001/rlb_ssg# show probe detail

 probe       : generic-https-probe
 type        : HTTPS
 state       : ACTIVE
 description : 
----------------------------------------------
   port      : 443     address     : 0.0.0.0         addr type  : -           
   interval  : 30      pass intvl  : 30              pass count : 3    
   fail count: 3       recv timeout: 10   
   http method      : GET
   http url         : /index.html
   conn termination : GRACEFUL  
   expect offset    : 0         , open timeout     : 10        
   expect regex     : -
   send data        : -
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health 
   ------------------- ---------------+----------+----------+----------+-------
   serverfarm  : https-test
     real      : FC8-server[0]
                       192.168.2.165   19634      0          19634      SUCCESS 

   Socket state        : CLOSED
   No. Passed states   : 1         No. Failed states : 0
   No. Probes skipped  : 0         Last status code  : 0
   No. Out of Sockets  : 0         No. Internal error: 0
   Last disconnect err :  - 
   Last probe time     : Fri Sep 25 09:40:39 2009
   Last fail time      : Never
   Last active time    : Fri Sep 18 14:03:15 2009

Notes

For a description of the output fields, see chapter "Configuring Health Monitoring" in the Cisco Application Control Engine Module Server Load-Balancing Configuration Guide.


show proc

Displays general information about all processes running on the ACE. This command and "show processes" are identical.

Sample Output

ACE30001/Admin# show proc

PID    State  PC        Start_cnt    TTY   Process
-----  -----  --------  -----------  ----  -------------
    1      S  2ac1964c            1     -  init
    2      S         0            1     -  keventd
    3      S         0            1     -  ksoftirqd_CPU0
    4      S         0            1     -  ksoftirqd_CPU1
    5      S         0            1     -  kswapd
    6      S         0            1     -  bdflush
    7      S         0            1     -  kupdated
   72      S  2ac2968c            1     -  cron
  116      S         0            1     -  loop0
  117      S         0            1     -  kjournald
  123      S         0            1     -  loop1
  124      S         0            1     -  kjournald
  130      S         0            1     -  loop2
  131      S         0            1     -  kjournald
  137      S         0            1     -  loop3
  138      S         0            1     -  kjournald
  144      S         0            1     -  loop4
  145      S         0            1     -  kjournald
  306      S         0            1     -  mts_kutil
  385      S         0            1     -  Pkt_Fifo_Tx
  386      S         0            1     -  Pkt_Fifo_Rx
  387      S         0            1     -  Pkt_Fifo_Util
  400      S         0            1     -  Netio_Encap
  556      S  2ac5b64c            1     -  telnetd
  557      S  2b650f64            1     1  vsh
  612      S  2c9f6fb0            1     -  lcpfw
  625      S         0            1     -  cavium
  733      S  2abe068c            1     -  n2_perf_stats
  742      S         0            1     -  ipcp_pci_rx
  759      S         0            1     -  keventd
  784      S  801010b4            1     -  insmod
  839      S  2b68364c            1     -  httpd
  854      S  2b2d364c            1     -  sysmgr
  872      S  2ceb56b4            1     -  syslogd
  873      S  2aebd64c            1     -  sdwrapd
  877      S  2affffb0            1     -  pfmgr
  881      S  2cd8e64c            1     -  ntp
  882      S  2acf064c            1     -  lmgrd
  883      S  2b0a564c            1     -  fs-daemon
  884      S  2ceb3020            1     -  syslogd
  885      S  2ceb56b4            1     -  syslogd
  886      S  2afb264c            1     -  confcheck
  888      S  2b14d984            1     -  licmgr
  895      S  2acf064c            1     -  cisco
  896      S  2b174984            1     -  vshd
  898      S  2ceb3020            1     -  syslogd
  899      S  2aec664c            1     -  ntpd
  901      S  2b68f8e4            1     -  httpd
  902      S  2b680fb0            1     -  httpd
  906      S  2d4f364c            1     -  xinetd
  907      S  2cb1afb0            1     -  vacd
  908      S  2ae7664c            1     -  ttyd
  909      S  2ae4e87c            1     -  sysinfo
  910      S  2dd956b4            1     -  snmpd
  911      S  2aec0fb0            1     -  sme
  912      S  2b1a364c            1     -  scripted_hm
  913      S  2cf6064c            1     -  radiusd
  914      S  2ae2afb0            1     -  pktcap
  915      S  2adfafb0            1     -  nat_dnld
  916      S  2cdb9020            1     -  itasca_ssl
  917      S  2afabfb0            1     -  itasca_route_mgr
  918      S  2af5cfb0            1     -  itasca_fm
  919      S  2aff1fb0            1     -  ifmgr
  920      S  2b019fb0            1     -  hsrp_track
  921      S  2b43a6b4            1     -  hm
  922      S  2ca2f64c            1     -  ha_mgr
  923      S  2af8164c            1     -  ha_dp_mgr
  924      S  2b01064c            1     -  gslb_proto
  925      S  2b0ae64c            1     -  dhcrelay
  926      S  2af5a64c            1     -  core-dmon
  927      S  2c9be6b4            1     -  config_cntlr
  928      S  2e2bee58            1     -  cfgmgr
  929      S  2ae69fb0            1     -  bpdu
  930      S  2aed4984            1     -  ascii-cfg
  931      S  2cecb020            1     -  arp_mgr
  932      S  2adf0020            1     -  aclmerged
  933      S  2ce9364c            1     -  tacacs
  934      S  2cf1264c            1     -  ldap
  935      S  2cf0f984            1     -  aaa
  939      S  2b438020            1     -  hm
  941      S  2c9bc020            1     -  config_cntlr
  942      S  2c9be6b4            1     -  config_cntlr
  943      S  2cecb020            1     -  arp_mgr
  944      S  2cecb020            1     -  arp_mgr
  945      S  2b43a6b4            1     -  hm
  946      S  2b43a6b4            1     -  hm
  947      S  2cecb020            1     -  arp_mgr
  948      S  2b43a6b4            1     -  hm
  949      S  2e3a0020            1     -  cfgmgr
  951      S  2b43a6b4            1     -  hm
  952      S  2ce9464c            1     -  securityd
  953      S  2ab15024            1     -  cfgmgr
  954      S  2cdb9020            1     -  itasca_ssl
  955      S  2cdbb6b4            1     -  itasca_ssl
  956      S  2cdba984            1     -  itasca_ssl
  957      S  2cdbb6b4            1     -  itasca_ssl
  960      S  2e2bee58            1     -  cfgmgr
  961      S  2e3696dc            1     -  cfgmgr
  962      S  2adf0020            1     -  aclmerged
  963      S  2e3a0020            1     -  cfgmgr
  964      S  2adb96dc            1     -  aclmerged
  965      S  2e3a0020            1     -  cfgmgr
  966      S  2dd93020            1     -  snmpd
  967      S  2dd94984            1     -  snmpd
  968      S  2dd956b4            1     -  snmpd
 1132      S         0            1     -  Peer
 1146      S  2ac0f87c            1     -  klogd
 1794      S  2ac5b64c            1     -  telnetd
 1795      S  2b68a64c            1     0  vsh
 1858      S         0            1     -  Peer
 4866      S  2b689984            1     1  vsh
 4867      S  2ac9f87c            1     1  more
 4868      R  2ac6087c            1     -  ps
20999      S  2b68a64c            1   S00  vsh
    -     NR         -            0     -  installer
    -     NR         -            0     -  session_agent

Notes

The process state value can be one of:

  • D – Uninterruptible sleep (usually I/O related)
  • ER – Error while running
  • NR – Not running
  • R – Running or runnable (on run queue)
  • S – Interruptible sleep (waiting for an event to complete)
  • T – Stopped, either by a job control signal or because it is being traced
  • W – Paging
  • X – Process is dead
  • Z – Defunct ("zombie") process; terminated but not reaped by its parent


show processes cpu

Displays CPU information for the Intel Pentium processor. This command is used for troubleshooting high CPU issues, slow response times on the CLI and performance issues.

Sample Output

ACE30001/Admin# show proc cpu

CPU utilization for five seconds: 2%; one minute: 5%; five minutes: 5%
PID    Runtime(ms)  Invoked   uSecs  1Sec   5 Sec    1 Min   5 Min  Process
-----  -----------  --------  -----  -----   ----    ----    ----  -----------
    1        13737    120542    113    0.0   0.0 %   0.0 %   0.0 %  init
    2            0        32     17    0.0   0.0 %   0.0 %   0.0 %  keventd
    3          172     10815     15    0.0   0.0 %   0.0 %   0.0 %  ksoftirqd_CPU0
    4            0        12      9    0.0   0.0 %   0.0 %   0.0 %  ksoftirqd_CPU1
    5            0         1     10    0.0   0.0 %   0.0 %   0.0 %  kswapd
    6            0         1     11    0.0   0.0 %   0.0 %   0.0 %  bdflush
    7       567639    294614   1926    0.0   0.0 %   0.2 %   0.10%  kupdated
   72         1154     10005    115    0.0   0.0 %   0.0 %   0.0 %  cron
  116          109       185    591    0.0   0.0 %   0.0 %   0.0 %  loop0
  117           20       188    108    0.0   0.0 %   0.0 %   0.0 %  kjournald
  123           90       247    367    0.0   0.0 %   0.0 %   0.0 %  loop1
  124          755       285   2650    0.0   0.0 %   0.0 %   0.0 %  kjournald
  130          148      1900     78    0.0   0.0 %   0.0 %   0.0 %  loop2
  131           83      1990     41    0.0   0.0 %   0.0 %   0.0 %  kjournald
  137           27        46    590    0.0   0.0 %   0.0 %   0.0 %  loop3
  138            1        33     44    0.0   0.0 %   0.0 %   0.0 %  kjournald
  144          391      9856     39    0.0   0.0 %   0.0 %   0.0 %  loop4
  145          435      9912     43    0.0   0.0 %   0.0 %   0.0 %  kjournald
  306        27264    718916     37    0.0   0.0 %   0.0 %   0.0 %  mts_kutil
  385         9627    558762     17    0.0   0.0 %   0.0 %   0.0 %  Pkt_Fifo_Tx
  386       278879   8841533     31    0.0   0.6 %   0.2 %   0.1 %  Pkt_Fifo_Rx
  387            0         1      5    0.0   0.0 %   0.0 %   0.0 %  Pkt_Fifo_Util
  400            1        20     68    0.0   0.0 %   0.0 %   0.0 %  Netio_Encap
  556          516      7287     70    0.0   0.5 %   0.1 %   0.0 %  telnetd
  557         1511      2040    741    0.0   0.9 %   0.1 %   0.0 %  vsh
  612       372224   1315018    283    0.0   0.1 %   0.2 %   0.1 %  lcpfw
  625       461272  58972997      7    0.0   0.4 %   0.3 %   0.0 %  cavium
  733       242945    584082    415    0.0   0.1 %   0.1 %   0.1 %  n2_perf_stats
  742          160      1901     84    0.0   0.0 %   0.0 %   0.0 %  ipcp_pci_rx
  759            0         1      7    0.0   0.0 %   0.0 %   0.0 %  keventd
  784        33075    117935    280    0.0   0.0 %   0.0 %   0.0 %  insmod
  839        21434    589851     36    0.0   0.0 %   0.0 %   0.0 %  httpd
  854       108727   1192089     91    0.0   0.10%   0.2 %   0.0 %  sysmgr
  872      3863585  58964891     65    0.0   0.31%   0.32%   0.29%  syslogd
  873           97        39   2504    0.0   0.0 %   0.0 %   0.0 %  sdwrapd
  877         1368      9864    138    0.0   0.0 %   0.0 %   0.0 %  pfmgr
  881        20740    127451    162    0.0   0.0 %   0.0 %   0.0 %  ntp
  882          108        13   8353    0.0   0.0 %   0.0 %   0.0 %  lmgrd
  883           95        47   2029    0.0   0.0 %   0.0 %   0.0 %  fs-daemon
  884         4394    293365     14    0.0   0.0 %   0.0 %   0.0 %  syslogd
  885        59625    604007     98    0.0   0.0 %   0.0 %   0.0 %  syslogd
  886          173        24   7222    0.0   0.0 %   0.0 %   0.0 %  confcheck
  888         8246    117982     69    0.0   0.0 %   0.0 %   0.0 %  licmgr
  895         3726      9845    378    0.0   0.0 %   0.0 %   0.0 %  cisco
  896        10652     74169    143    0.0   0.0 %   0.0 %   0.0 %  vshd
  898        64927   9827441      6    0.0   0.0 %   0.0 %   0.0 %  syslogd
  899        60989    589699    103    0.0   0.0 %   0.0 %   0.0 %  ntpd
  901           43        48    896    0.0   0.0 %   0.0 %   0.0 %  httpd
  902           23        47    508    0.0   0.0 %   0.0 %   0.0 %  httpd
  906        11593    118067     98    0.0   0.0 %   0.0 %   0.0 %  xinetd
  907        50695     61125    829    0.0   0.0 %   0.0 %   0.0 %  vacd
  908        12475    294862     42    0.0   0.0 %   0.0 %   0.0 %  ttyd
  909         4088     60169     67    0.0   0.90%   0.13%   0.2 %  sysinfo
  910       113389   1188841     95    0.0   0.0 %   0.0 %   0.0 %  snmpd
  911        19414    293510     66    0.0   0.0 %   0.0 %   0.0 %  sme
  912          124        59   2103    0.0   0.0 %   0.0 %   0.0 %  scripted_hm
  913        18147    589714     30    0.0   0.0 %   0.0 %   0.0 %  radiusd
  914           63        34   1858    0.0   0.0 %   0.0 %   0.0 %  pktcap
  915          160       291    553    0.0   0.0 %   0.0 %   0.0 %  nat_dnld
  916          732     10171     72    0.0   0.0 %   0.0 %   0.0 %  itasca_ssl
  917         6675    118193     56    0.0   0.0 %   0.0 %   0.0 %  itasca_route_mgr
  918          280      9878     28    0.0   0.0 %   0.0 %   0.0 %  itasca_fm
  919     43134235    355623  121292   38.5   0.0 %   3.57%   3.68%  ifmgr
  920          265      9840     26    0.0   0.0 %   0.0 %   0.0 %  hsrp_track
  921         3966     39373    100    0.0   0.0 %   0.0 %   0.0 %  hm
  922        55367    254374    217    0.0   0.0 %   0.0 %   0.0 %  ha_mgr
  923         5570     59293     93    0.0   0.0 %   0.0 %   0.0 %  ha_dp_mgr
  924           77        44   1752    0.0   0.0 %   0.0 %   0.0 %  gslb_proto
  925        10551    118031     89    0.0   0.0 %   0.0 %   0.0 %  dhcrelay
  926           95        49   1949    0.0   0.0 %   0.0 %   0.0 %  core-dmon
  927         5270     59172     89    0.0   0.0 %   0.0 %   0.0 %  config_cntlr
  928          711        96   7406    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  929         4460    541616      8    0.0   0.0 %   0.0 %   0.0 %  bpdu
  930         6203     73746     84    0.0   0.0 %   0.0 %   0.0 %  ascii-cfg
  931      8678273   9613436    902    1.3   0.82%   0.73%   0.72%  arp_mgr
  932         2340     10199    229    0.0   0.0 %   0.0 %   0.0 %  aclmerged
  933        29562    590176     50    0.0   0.4 %   0.0 %   0.0 %  tacacs
  934        21326    589708     36    0.0   0.0 %   0.0 %   0.0 %  ldap
  935        33096    591425     55    0.0   0.2 %   0.0 %   0.0 %  aaa
  939         5541    293370     18    0.0   0.0 %   0.0 %   0.0 %  hm
  941         5812    293361     19    0.0   0.0 %   0.0 %   0.0 %  config_cntlr
  942           79        44   1805    0.0   0.0 %   0.0 %   0.0 %  config_cntlr
  943         4725    293357     16    0.0   0.0 %   0.0 %   0.0 %  arp_mgr
  944        55097   1111282     49    0.0   0.0 %   0.0 %   0.0 %  arp_mgr
  945        13398     40204    333    0.0   0.0 %   0.0 %   0.0 %  hm
  946        13377     39867    335    0.0   0.0 %   0.0 %   0.0 %  hm
  947        70507    338432    208    0.0   0.0 %   0.0 %   0.0 %  arp_mgr
  948            0         3     59    0.0   0.0 %   0.0 %   0.0 %  hm
  949         5412    293367     18    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  951            0         3     41    0.0   0.0 %   0.0 %   0.0 %  hm
  952        21731    589850     36    0.0   0.0 %   0.0 %   0.0 %  securityd
  953            0         1    395    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  954         5859    293372     19    0.0   0.0 %   0.0 %   0.0 %  itasca_ssl
  955            0         2    107    0.0   0.0 %   0.0 %   0.0 %  itasca_ssl
  956            0         1    385    0.0   0.0 %   0.0 %   0.0 %  itasca_ssl
  957            0         1    174    0.0   0.0 %   0.0 %   0.0 %  itasca_ssl
  960         2312      1521   1520    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  961        18032    583809     30    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  962         5054    293360     17    0.0   0.0 %   0.0 %   0.0 %  aclmerged
  963          154       147   1052    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  964         9595    293365     32    0.0   0.0 %   0.0 %   0.0 %  aclmerged
  965        53701     16022   3351    0.0   0.0 %   0.0 %   0.0 %  cfgmgr
  966         4743    293360     16    0.0   0.0 %   0.0 %   0.0 %  snmpd
  967         4023      7706    522    0.0   0.0 %   0.0 %   0.0 %  snmpd
  968       199527   5897126     33    0.0   0.1 %   0.1 %   0.1 %  snmpd
 1132            0         4    119    0.0   0.0 %   0.0 %   0.0 %  Peer
 1146           13        26    514    0.0   0.0 %   0.0 %   0.0 %  klogd
 1794           38       127    299    0.0   0.0 %   0.0 %   0.0 %  telnetd
 1795          355       103   3454    0.0   0.0 %   0.0 %   0.0 %  vsh
 1858        16111    118034    136    0.0   0.0 %   0.0 %   0.0 %  Peer
 4898            9         5   1873    0.0   0.0 %   0.0 %   0.0 %  vsh
 4899           20         5   4127    0.0   0.0 %   0.0 %   0.0 %  more
 4900          122         6  20451    0.0   0.0 %   0.0 %   0.0 %  ps
20999         2150     65452     32    0.0   0.0 %   0.0 %   0.0 %  vsh

Notes

This command output format is:

CPU utilization for five seconds: 12%; one minute: 4%; five minutes: 4%
PID    Runtime(ms)  Invoked   uSecs  1Sec   5 Sec    1 Min   5 Min  Process
-----  -----------  --------  -----  -----   ----    ----    ----  -----------
  864     14560628 141874354    102    0.0   0.25%   0.25%   0.24%  syslog

The noteworthy statistics are the CPU averages for the last 1 second, 5 seconds, 1 minute and 5 minutes, as shown in the first line of the display are the CPU averages for all processes running on the ACE:

"CPU utilization for five seconds: 12%; one minute: 4%; five minutes: 4%"

Thereafter, each task is listed individually with its CPU averages.


show processes memory

Displays current memory allocation per process.

Sample Output

ACE30001/Admin# show proc memory

PID    MemAlloc  StackBase/Ptr      Process
-----  --------  -----------------  ----------------
    1     14592  7fff7f40/7fff77d0  init
    2         0         0/0         keventd
    3         0         0/0         ksoftirqd_CPU0
    4         0         0/0         ksoftirqd_CPU1
    5         0         0/0         kswapd
    6         0         0/0         bdflush
    7         0         0/0         kupdated
   72     20928  7fff7e10/7fff7b90  cron
  116         0         0/0         loop0
  117         0         0/0         kjournald
  123         0         0/0         loop1
  124         0         0/0         kjournald
  130         0         0/0         loop2
  131         0         0/0         kjournald
  137         0         0/0         loop3
  138         0         0/0         kjournald
  144         0         0/0         loop4
  145         0         0/0         kjournald
  306         0         0/0         mts_kutil
  385         0         0/0         Pkt_Fifo_Tx
  386         0         0/0         Pkt_Fifo_Rx
  387         0         0/0         Pkt_Fifo_Util
  400         0         0/0         Netio_Encap
  556     17824  7fff7ce0/7fff7870  telnetd
  557    190628  7fff7e30/7fff6ab8  vsh
  612    184064  7fff7e70/7fff7a00  lcpfw
  625         0         0/0         cavium
  733      8144  7fff7e40/7fff6ac8  n2_perf_stats
  742         0         0/0         ipcp_pci_rx
  759         0         0/0         keventd
  784    105880  7fff7e40/801010c4  insmod
  839    838336  7fff7df0/7fff7c70  httpd
  854    892560  7fff7e70/7fff7c98  sysmgr
  872    250656  7fff7cb0/7fff7aa8  syslogd
  873     13960  7fff7ce0/7fff6dd0  sdwrapd
  877    112460  7fff7ce0/7fff7990  pfmgr
  881    220544  7fff7d00/7fff7840  ntp
  882     96472  7fff7c30/7fff6980  lmgrd
  883     96656  7fff7d00/7fff7508  fs-daemon
  884    250656  7fff7cb0/135d0658  syslogd
  885    250656  7fff7cb0/7f7ff4e8  syslogd
  886     30768  7fff7d00/7fff7b98  confcheck
  888    236140  7fff7ce0/7fff7770  licmgr
  895    125720  7fff7c10/7fff7ab8  cisco
  896   2896768  7fff7d10/7fff7778  vshd
  898    250656  7fff7cb0/7f5ff850  syslogd
  899     43888  7fff7cd0/7fff7b90  ntpd
  901    973504  7fff7df0/7fff7b60  httpd
  902    989888  7fff7df0/7fff7b90  httpd
  906    113648  7fff7cb0/7fff7b10  xinetd
  907    310484  7fff7ce0/7fff7990  vacd
  908    112252  7fff7d10/7fff7660  ttyd
  909     30772  7fff7d00/7fff7910  sysinfo
  910   1649136  7fff7ce0/7fff6520  snmpd
  911     70024  7fff7ce0/7fff7960  sme
  912     56676  7fff7cc0/7fff77b0  scripted_hm
  913    343360  7fff7cd0/7fff68b0  radiusd
  914     21856  7fff7ce0/7fff7990  pktcap
  915    457212  7fff7cd0/7fff76a8  nat_dnld
  916    224108  7fff7cc0/7fff7978  itasca_ssl
  917   2379776  7fff7ca0/7fff7720  itasca_route_mgr
  918     55728  7fff7cd0/7fff7988  itasca_fm
  919    252336  7fff7ce0/7fff7948  ifmgr
  920     23968  7fff7cc0/7fff7938  hsrp_track
  921     99112  7fff7cf0/7fff7070  hm
  922     78720  7fff7ce0/7fff7740  ha_mgr
  923     66848  7fff7cd0/7fff7738  ha_dp_mgr
  924     71488  7fff7cc0/7fff7750  gslb_proto
  925     96384  7fff7cf0/7fff7968  dhcrelay
  926     93168  7fff7cf0/7fff7730  core-dmon
  927    409888  7fff7cc0/7fff7718  config_cntlr
  928   8482640  7fff7cd0/7fff64a8  cfgmgr
  929     56416  7fff7ce0/7fff77a0  bpdu
  930     51856  7fff7cf0/7fff78d8  ascii-cfg
  931    672272  7fff7cd0/7fff7310  arp_mgr
  932    347152  7fff7cd0/7fff7760  aclmerged
  933    328636  7fff7cd0/7fff66a0  tacacs
  934    267072  7fff7ce0/7fff6af8  ldap
  935    182544  7fff7ce0/7fff7b18  aaa
  939     99112  7fff7cf0/10039e90  hm
  941    409888  7fff7cc0/1005a678  config_cntlr
  942    409888  7fff7cc0/7f7ff7e8  config_cntlr
  943    672272  7fff7cd0/127557c0  arp_mgr
  944    672272  7fff7cd0/7f7ff328  arp_mgr
  945     99112  7fff7cf0/7f7ff400  hm
  946     99112  7fff7cf0/7f5ff400  hm
  947    672272  7fff7cd0/7f5ff828  arp_mgr
  948     99112  7fff7cf0/7f3ff400  hm
  949   8482640  7fff7cd0/105b6f98  cfgmgr
  951     99112  7fff7cf0/7f1ff400  hm
  952    153864  7fff7cd0/7fff6b60  securityd
  953   8482640  7fff7cd0/7f7ffa68  cfgmgr
  954    224108  7fff7cc0/1001baf8  itasca_ssl
  955    224108  7fff7cc0/7f7ff340  itasca_ssl
  956    224108  7fff7cc0/7f5ff180  itasca_ssl
  957    224108  7fff7cc0/7f3ff338  itasca_ssl
  960   8482640  7fff7cd0/7f5f5550  cfgmgr
  961   8482640  7fff7cd0/7f3fa690  cfgmgr
  962    347152  7fff7cd0/10029da0  aclmerged
  963   8482640  7fff7cd0/7f1fa870  cfgmgr
  964    347152  7fff7cd0/7f7ff8d8  aclmerged
  965   8482640  7fff7cd0/7eff0600  cfgmgr
  966   1649136  7fff7ce0/10846c4c  snmpd
  967   1649136  7fff7ce0/7f7ffa70  snmpd
  968   1649136  7fff7ce0/7f5ff7d8  snmpd
 1132         0         0/0         Peer
 1146     59632  7fff7ca0/7fff7be8  klogd
 1794     17824  7fff7ce0/7fff7870  telnetd
 1795    170148  7fff7e30/7fff6a60  vsh
 1858         0         0/0         Peer
 4912    190628  7fff7e30/7fff56d8  vsh
 4913      5400  7fff7e10/7fff7b20  more
 4914         0  7fff7c80/7fff7ad8  ps
20999    198820  7fff7e70/7fff6aa0  vsh

Notes

A statistic that is relevant to troubleshooting in this output is the MemAlloc column. MemAlloc shows the amount of fixed memory allocated by each process. By watching this value, you can discover processes that are using an increasing amount of memory, possibly indicative of a memory usage problem. This would be done by issuing the command at fixed intervals checking for large increases in the allocated memory.


show resource allocation

This output shows resource allocation.

Sample Output

ace19/Admin# sho resource allocation
---------------------------------------------------------------------------
Parameter                 Min      Max         Class
---------------------------------------------------------------------------

acl-memory                0.00%    100.00%    default 

syslog buffer             0.00%    100.00%    default 

conc-connections          0.00%    100.00%    default 

mgmt-connections          0.00%    100.00%    default

proxy-connections         0.00%    100.00%    default

bandwidth                 0.00%    100.00%    default

connection rate           0.00%    100.00%    default

inspect-conn rate         0.00%    100.00%    default

syslog rate               0.00%    100.00%    default

regexp                    0.00%    100.00%    default

sticky                    0.00%    100.00%    default

xlates                    0.00%    100.00%    default

ssl-connections rate      0.00%    100.00%    default

mgmt-traffic rate         0.00%    100.00%    default

mac-miss rate             0.00%    100.00%    default

acc-connections           0.00%    100.00%    default

http-comp rate            0.00%    100.00%    default

throughput                0.00%    100.00%    default

Notes

For more information, see the Managing Resources section of the Troubleshooting Guide.


show resource usage all

When executed from the Admin context, this command displays the resource usage for all other contexts.

Sample Output

ACE30001/Admin# show resource usage all
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-------------------------------------------------------------------------------
Context: Admin
  conc-connections              9        339      80000    2000000          0
  mgmt-connections             18         28       1000      25000          0
  proxy-connections             0        118      10486     262144          0
  xlates                        0          0      10486     262144          0
  bandwidth                  2072    6920309    5000000  250000000          0
    throughput               1058    6907078    5000000  125000000          0
    mgmt-traffic rate        1014      13231          0  125000000          0
  connections rate              0       2896      10000     250000          0
  ssl-connections rate          0          0         10        250          0
  mac-miss rate                 0          1         20        500          0
  inspect-conn rate             0          0         60       1500          0
  acl-memory                33920      33920     784000   19652608          0
  sticky                        1          2      41942          0          0
  regexp                        0          0      10486     262144          0
  syslog buffer            272384     272384      40960    1007616          0
  syslog rate                   0         14       1000      25000          0
Context: rlb_csg
  conc-connections             24        114    2000000    2000000          0
  mgmt-connections              0          0      25000      25000          0
  proxy-connections             0         11     262144     262144          0
  xlates                        0          0     262144     262144          0
  bandwidth                   164       3630  125000000  250000000          0
    throughput                164       3630  125000000  125000000          0
    mgmt-traffic rate           0          0          0  125000000          0
  connections rate              0         32     250000     250000          0
  ssl-connections rate          0          0        250        250          0
  mac-miss rate                 0          0        500        500          0
  inspect-conn rate             0          0       1500       1500          0
  acl-memory                 5520       5520   19650480          0          0
  sticky                        0          0    1048576          0          0
  regexp                       14         14     262144          0          0
  syslog buffer                 0          0    1048576          0          0
  syslog rate                   0          0      25000          0          0
Context: rlb_ssg
  conc-connections              0         11    2000000    2000000          0
  mgmt-connections              0          0      25000      25000          0
  proxy-connections             0         11     262144     262144          0
  xlates                        0          0     262144     262144          0
  bandwidth                   842      12402  125000000  250000000          0
    throughput                842      12402  125000000  125000000          0
    mgmt-traffic rate           0          0          0  125000000          0
  connections rate             10         42     250000     250000          0
  ssl-connections rate          0          2        250        250          0
  mac-miss rate                10         11        500        500          0
  inspect-conn rate             0          0       1500       1500          0
  acl-memory                11096      11096   19650480          0          0
  sticky                        0          0        838          0          0
  regexp                       18         18     262144          0          0
  syslog buffer           1046528    1046528    1048576          0          0
  syslog rate                   0         10      25000          0          0
Context: spirent_ssg
  conc-connections              0          4    2000000    2000000          0
  mgmt-connections              0          0      25000      25000          0
  proxy-connections             0          4     262144     262144          0
  xlates                        0          0     262144     262144          0
  bandwidth                    82       2378  125000000  250000000          0
    throughput                 82       2378  125000000  125000000          0
    mgmt-traffic rate           0          0          0  125000000          0
  connections rate              0         25     250000     250000          0
  ssl-connections rate          0          0        250        250          0
  mac-miss rate                 0          0        500        500          0
  inspect-conn rate             0          0       1500       1500          0
  acl-memory                 9456       9456   19650480          0          0
  sticky                        0          0        838          0          0
  regexp                       14         14     262144          0          0
  syslog buffer                 0          0    1048576          0          0
  syslog rate                   0          0      25000          0          0

Notes

Note the mac-miss rate. Mac-miss messages are sent when the dataplane does not find an encap-entry on a route/mac-lookup and ratelimited is 2k/sec. It's possible to see denied messages if incoming traffic rate is higher than that, since CP can take some time to resolve and program the encap entries. The "mac-miss rate" value then is the number of times ACE got a packet from an unknown mac-address, which the CP needs to learn from these packets to populate the neighbors


show rserver detail

The rserver stats fields are similar to those from show serverfarm detail; the only difference is that "total conn-failures" value appears at the bottom of the output.

Note that this command displays rserver information followed by a separate set of information on every serverfarm (that is, every sfarm-real instance).

Sample Output

ACE30001/Admin# show rserver detail

 rserver              : 165, type: HOST
 state                : OPERATIONAL (verified by arp response)
 description          : -
 max-conns            : -         ,  out-of-rotation count  : -
 min-conns            : -         
 conn-rate-limit      : 350000    ,  out-of-rotation count  : 0
 bandwidth-rate-limit : 300000000 ,  out-of-rotation count  : 0
 weight               : 8
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total               
   ---+---------------------+------+------------+----------+--------------------
   serverfarm: IPSEC_GWs
       192.168.2.165:0       8      OPERATIONAL  0          2437322             
         max-conns            : -         ,  out-of-rotation count  : -
         min-conns            : -         
         conn-rate-limit      : -         ,  out-of-rotation count  : -
         bandwidth-rate-limit : -         ,  out-of-rotation count  : -
         total conn-failures  : 16062


 rserver              : 231, type: HOST
 state                : OPERATIONAL (verified by arp response)
 description          : -
 max-conns            : -         ,  out-of-rotation count  : -
 min-conns            : -         
 conn-rate-limit      : 350000    ,  out-of-rotation count  : 0
 bandwidth-rate-limit : 300000000 ,  out-of-rotation count  : 0
 weight               : 8
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total               
   ---+---------------------+------+------------+----------+--------------------
   serverfarm: IPSEC_GWs
       192.168.2.231:0       8      OPERATIONAL  0          1188792             
         max-conns            : -         ,  out-of-rotation count  : -
         min-conns            : -         
         conn-rate-limit      : -         ,  out-of-rotation count  : -
         bandwidth-rate-limit : -         ,  out-of-rotation count  : -
         total conn-failures  : 6156


 rserver              : 233, type: HOST
 state                : OPERATIONAL (verified by arp response)
 description          : -
 max-conns            : -         ,  out-of-rotation count  : -
 min-conns            : -         
 conn-rate-limit      : 350000    ,  out-of-rotation count  : 0
 bandwidth-rate-limit : 300000000 ,  out-of-rotation count  : 0
 weight               : 8
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total               
   ---+---------------------+------+------------+----------+--------------------
   serverfarm: IPSEC_GWs
       192.168.2.233:0       8      OPERATIONAL  0          233321              
         max-conns            : -         ,  out-of-rotation count  : -
         min-conns            : -         
         conn-rate-limit      : -         ,  out-of-rotation count  : -
         bandwidth-rate-limit : -         ,  out-of-rotation count  : -
         total conn-failures  : 1217


show scp stats

Displays information related to the Switch Module Control Protocol (SCP) statistics. SCP is the protocol that the SUP (IOS) and ACE communicate on over the EOBC channel.

Sample Output

ACE30001/Admin# show scp stats 
SCP statistics:
Tx packets                                 773704 
Rx packets                                 773704 
Tx bytes                                195106600 
Rx packets                               48449378 
TX Stats 
TX Errors                                       0 
TX Timeout                                      0 
Not Connected                                   0 
Tx No memory                                    0 
RX Stats 
Rx Error                                        1 
Rx No memory                                    0 
Rx no buffers                                   0 
Rx offline                                      0 
Rx message size                                 0 
Rx Kthread enqueue                         606026 
Rx Kthread dequeue                         606026 
Rx unknown SAP                                  0 
Rx MTS alloc fail                               0 
Rx MTS enqueue                             606026 
Rx MTS enqueue fail                             0 

Notes

The error-related counters are for the most part self explanatory (see below). In general these errors incrementing would be cause for concern and could be a hardware issue.

There are two main types of errors shown, transmit errors and receive errors. Of note are the following.

  • Transmit (TX) Errors:
    • timeout — Message sent to SUP timed out and the message was discarded.
    • not connected — The EOBC channel between the ACE/SUP is dead. This is a serious error that would most likely result in the ACE not coming up or rebooting without a core.
    • no memory — There is no fixed memory available and the message was discarded.
  • Receive (RX) Errors:
    • no memory — There is no fixed memory available and the message was discarded.
    • no buffers — There was no buffer available to receive packets from the SUP and the incoming message is discarded.
    • offline — The EOBC channel between the ACE/SUP is dead. This is a serious error that would most likely result in the ACE not coming up or rebooting without a core.
    • message size — The incoming message from the SUP exceeded the maximum for the EOBC channel. This indicates a serious communication issue between the SUP and ACE.


show serverfarm detail

Displays statistics for an entire serverfarm, followed by stats for the individual rservers in the serverfarm.

Sample Output

ACE30001/Admin#  show serverfarm detail
 serverfarm     : IPSEC_GWs, type: HOST
 total rservers : 3
 active rservers: 3
 description    : -
 state          : ACTIVE
 predictor      : LEASTCONNS
   slowstart    : 0 secs
 failaction     : -
 back-inservice    : 0
 partial-threshold : 0
 num times failover       : 0
 num times back inservice : 1
 total conn-dropcount : 0
 ---------------------------------
                                                ----------connections-----------
       real                  weight state        current    total      failures 
   ---+---------------------+------+------------+----------+----------+---------
   rserver: 165
       192.168.2.165:0       8      OPERATIONAL  0          2437322    16062
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -         
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -
         load value           : 0         

   rserver: 231
       192.168.2.231:0       8      OPERATIONAL  0          1188792    6156
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -         
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -
         load value           : 0         

   rserver: 233
       192.168.2.233:0       8      OPERATIONAL  0          233321     1217
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -         
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -
         load value           : 0         

Notes

The serverfarm-related stats are:

Field Description
serverfarm Name of the serverfarm and the configured server farm type, HOST or REDIRECT.
total rservers Total number of real servers associated with the server farm.
active rservers Number of real servers that are active in the server farm.
description User-entered text description of the server farm with a maximum of 240 alphanumeric characters.
state Current state of the server farm. Possible values are ACTIVE or INACTIVE.
predictor Configured load-balancing method and values for various fields associated with the predictor. Possible predictor values are:
  • HASH-ADDRSRC
  • HASH-ADDRDEST
  • HASH-COOKIE
  • HASH-HEADER
  • HASH-HTTP-CONTENT
  • HASH-LAYER4-PAYLOAD
  • HASH-URL
  • LEASTBANDWIDTH
  • LEASTCONNS
  • LEASTLOADED
  • RESPONSE
  • ROUNDROBIN
slowstart Configured slowstart value.
failaction Action that the ACE takes for connections if a real server fails in a server farm. Possible actions are purge or none.
back-inservice Configured value of the back-inservice keyword of the partial-threshold command. Specifies the minimum percentage of real servers in the primary server farm that must be active again for the ACE to place the server farm back in service.
partial-threshold Configured value of the partial-threshold command. Specifies the minimum percentage of real servers in the primary server farm that must remain active for the server farm to stay up.
num times failover Number of times that the server farm failed over to the backup server farm.
num times back inservice Number of times that the ACE placed the server farm back in service after a failover.
total conn-dropcount Total number of connections that the ACE discarded because the number of connections exceeded the configured conn-limit max value.


The rserver-related stats are:

Field Description
Rserver Name of the real server associated with the server farm.
IP Address:Port IP address and port of the real server.
Weight Weight assigned to the real server in the server farm.
State Current state of the real server. Possible states are OPERATIONAL or OUTOFSERVICE.
Current Connections Number of active connections to the real server.
Total Connections Total number of connections to the specified server farm.
Maxconns Configured maximum allowable number of active connections to a real server.
Minconns Configured minimum number of connections that the number of connections must fall below before sending more connections to a server after it has exceeded the maximum connections threshold.
Out-of-rotation-count Number of times that the real server was not considered for load balancing because the number of connections, connection rate, or bandwidth rate exceeded the configured limits of the server.
Conn-rate-limit Configured connection rate limit of the real server in connections per second.
Bandwidth-rate-limit Configured bandwidth rate limit of the real server in bytes per second.
Connections failures This is incremented if the connection is closed for the following reasons:
  • SYN timeout
  • RST received
  • Internal exception

An internal exception event can happen for a variety of reasons, such as failure to start or restart a timer, destination NAT failure, redundant connection, etc.


show service-policy detail

There are two forms of the command:

  • show service-policy detail
  • show service-policy [name] details

The second displays detailed service-policy counters. The show service-policy [name] command gives a similar though less verbose output for the named service policies.

Sample Output

ACE30001/Admin# show service-policy detail

Policy-map : LB
Status     : ACTIVE
Description: -----------------------------------------
Interface: vlan 101 
  service-policy: LB
    class: IPSEC-CLASS
      nat:
        nat dynamic 1 vlan 102
        curr conns       : 0         , hit count        : 3882870   
        dropped conns    : 388       
        client pkt count : 23025431  , client byte count: 1548137256          
        server pkt count : 23003930  , server byte count: 9373406823          
        conn-rate-limit      : -         , drop-count : -         
        bandwidth-rate-limit : -         , drop-count : -         
     VIP Address:    Protocol:  Port:
     192.168.219.150 udp        eq    500  
     192.168.219.150 udp        eq    4500 
     192.168.219.150 50
     192.168.219.150 tcp        eq    80   
      loadbalance:
        L7 loadbalance policy: IPSEC-GW
        VIP Route Metric     : 77
        VIP Route Advertise  : DISABLED
        VIP ICMP Reply       : ENABLED
        VIP State: INSERVICE
        curr conns       : 0         , hit count        : 3882870   
        dropped conns    : 388       
        client pkt count : 23025431  , client byte count: 1548137256          
        server pkt count : 23003930  , server byte count: 9373406823          
        conn-rate-limit      : -         , drop-count : -         
        bandwidth-rate-limit : -         , drop-count : -         
        L7 Loadbalance policy : IPSEC-GW
          class/match : class-default
             LB action: : 
               sticky group: SeGW_GRP
                  primary serverfarm: IPSEC_GWs
                    state: UP
                  backup serverfarm : -
            hit count        : 3882870   
            dropped conns    : 0         
        Parameter-map(s):
          UDP-TIMEOUT


Notes


Field Description
Status Current operational state of the service policy. Possible states are ACTIVE or INACTIVE.
Description User-entered description of the policy map.
Interface VLAN ID of the interface to which the policy map has been applied
Service Policy Unique identifier of the policy map.
Class Name of the class map associated with the service policy. There can be many classes associated with one service-policy.
NAT Dynamic Pool Identifier of the NAT dynamic pool and the associated VLAN.
NAT curr conns Number of active connections in this NAT Pool.
NAT hit count Number of times a connection was established through this NAT Pool.
NAT dropped conns Number of connections that the ACE discarded because the NAT Pool was out of resources.
NAT client pkt count Number of packets received from the network through this NAT Pool.
NAT client byte count Number of bytes received from the client side network through this NAT Pool.
NAT server pkt count Number of packets received from the service side network through this NAT Pool.
NAT server byte count Number of bytes received from the service side network through this NAT Pool.
NAT conn-rate-limit drop-count Number of connections that the ACE discarded because the rservers where at max connections through the NAT Pool.
NAT bandwidth-rate-limit drop-count Number of connections that the ACE discarded because the allocated bandwidth was exceeded through the NAT Pool.
Virtual IP (VIP) The VIP address associated with this class. The VIP is specified by VIP, Protocol and Port. There can be multiple VIPs associated with each class.
L7 Policy Name of the Layer 7 policy map associated with the service policy.
VIP Route Metric Specifies the distance metric for the route as specified with the loadbalance vip advertise command. The ACE writes the value configured to its routing table. Possible values are integers from 1 to 254.
VIP Route Advertise Operational state of the loadbalance vip advertise command: ENABLED or DISABLED. This command is used with Route Health Injection (RHI) to allow the ACE to advertise the availability of a VIP address throughout the network.
VIP ICMP Reply Operational state of the loadbalance vip icmp-reply command. Possible states are: ENABLED, DISABLED, ENABLED-WHEN-ACTIVE, or ENABLED-WHEN-PRIMARY-SF-UP.
VIP State Operational state of the virtual server: INSERVICE or OUTOFSERVICE.
Curr Conns Number of active connections to the VIP.
Dropped Conns Number of connections that the ACE discarded to this VIP. This is incremented whenever a connection hitting that VIP gets dropped/rejected. Common reasons that the connection hitting the VIP will be dropped are:
  • if all the rservers in the serverfarm associated to VIP goes down, then the VIP also will go down. So all the incoming connections will be dropped.
  • if a request in the connection requests some unknown URL to the VIP, then the connection will be rejected.
  • if the server which is picked up by the LB to load-balance the connection won't respond to the request, then after max retires, the connection will be dropped.

The counter itself is cumulative and that the value could be made up of entries from the following counters:

  1. sh stats loadbalance
    • Total Layer4 rejections
    • Total Layer7 rejections
    • Total Layer4 LB policy misses
    • Total Layer7 LB policy misses
    • Total times rserver was unavailable
  2. sh stats connection
    • Total Connections Timed-out
    • Total Connections Failed
  3. The "failures" counter from show serverfarm <serverfarm>.
  4. sh stats inspect
    • Total drop decisions
Client Pkt Count Number of packets received from the client side network to the VIP.
Server Pkt Count Number of packets received from the server side network to the VIP.
Hit Count Number of times a connection was established with this VIP.
Client Byte Count Number of bytes received from the client side network to the VIP.
Server Byte Count Number of bytes received from the server side network to the VIP.
Conn-rate-limit drop-count Number of connections that the ACE discarded to this VIP because the rservers where at max connections
Bandwidth-rate-limit drop-count Number of connections that the ACE discarded to this VIP because the allocated bandwidth was exceeded.
Primary server farm The name of the primary server farm.
Current state of the primary server farm The serverfarm can be either operational UP or DOWN.
Hit Count (Primary Server Farm) Cumulative number of connections to the primary server farm.
Dropped Conns (Primary Server Farm) Number of attempted connections to the primary server farm that the ACE discarded.
Backup server farm The name of the backup server farm.
Current state of the backup server farm The backup serverfarm can be either operational UP or DOWN.
Hit Count (Backup Server Farm) Cumulative number of connections to the primary server farm.
Dropped Conns (Backup Server Farm) Number of attempted connections to the primary server farm that the ACE discarded.
Parameter-map(s) The name of any configured parameter maps which have been associated with this class.


show stats loadbalance

Displays statistics related to load balancing activities of both np1 and np2. That is, it represents the sum of the values shown in show np 1 me-stats -slb and show np 2 me-stats -slb.


Sample Output

ACE/Admin# show stats loadbalance 
+------------------------------------------+
+------- Loadbalance statistics -----------+
+------------------------------------------+
 Total version mismatch              : 0
 Total Layer4 decisions              : 0
 Total Layer4 rejections             : 0
 Total Layer7 decisions              : 0
 Total Layer7 rejections             : 0
 Total Layer4 LB policy misses       : 0
 Total Layer7 LB policy misses       : 0
 Total times rserver was unavailable : 0
 Total ACL denied                    : 0
 Total IDMap Lookup Failures         : 0


Notes


Field Description
Total version mismatch Vserver version on the proxy info or in the vserver state does not match that expected for the L7 connection. This occurs when new (reconfigured) vserver info is passed to the dataplane. LB is unable to return a decision, and the connection is close or not formed.
Total Layer4 decisions LB loadBalanced, stuck, or forwarded an L4 connection.
Total Layer4 rejections LB rejected an L4 connection. This is often in conjunction with other counters telling why.
Total Layer7 decisions LB loadBalanced, stuck, or forwarded an L7 connection.
Total Layer7 rejections LB rejected an L7 decision for any of a multitude of reasons, including:
  • no available real server
  • acl deny
  • VIP not inservice
  • VIP config version mismatch (similar to HTTP below, but for LB)
  • any of the http errors below
    • connection invalid
    • no valid policy
    • error receiving sticky info from other IXP
    • exceed max capacity on rserver

For RADIUS traffic, if the client-side UDP connection and resulting pmap entry is removed before the RADIUS response is received by the ACE, ACE will not know where to forward the response. It therefore drops the packet and increments this counter.

Total Layer4 LB policy misses LB was unable to find a policy which satisfies all the conditions of the L7 connection.
Total Layer7 LB policy misses LB was unable to find a policy which satisfies all the conditions of the L7 connection.
Total times rserver was unavailable Policy and vserver were OK, but there was no acceptable rserver available for LB to send the connection to.
Total ACL denied The configured policy associated with this connection indicates it should be dropped. This can occur for either an L4 or an L7 policy.
Total IDMap Lookup Failures Fault Tolerant (FT) ID MAP lookup failures.


show stats crypto client

Sample Output

ACE30001/Admin# show stats crypto client
SSL Client Statistics: 
------------------
SSL alert CLOSE_NOTIFY rcvd:                      0
SSL alert UNEXPECTED_MSG rcvd:                    0
SSL alert BAD_RECORD_MAC rcvd:                    0
SSL alert DECRYPTION_FAILED rcvd:                 0
SSL alert RECORD_OVERFLOW rcvd:                   0
SSL alert DECOMPRESSION_FAILED rcvd:              0
SSL alert HANDSHAKE_FAILED rcvd:                  0
SSL alert NO_CERTIFICATE rcvd:                    0
SSL alert BAD_CERTIFICATE rcvd:                   0
SSL alert UNSUPPORTED_CERTIFICATE rcvd:           0
SSL alert CERTIFICATE_REVOKED rcvd:               0
SSL alert CERTIFICATE_EXPIRED rcvd:               0
SSL alert CERTIFICATE_UNKNOWN rcvd:               0
SSL alert ILLEGAL_PARAMETER rcvd:                 0
SSL alert UNKNOWN_CA rcvd:                        0
SSL alert ACCESS_DENIED rcvd:                     0
SSL alert DECODE_ERROR rcvd:                      0
SSL alert DECRYPT_ERROR rcvd:                     0
SSL alert EXPORT_RESTRICTION rcvd:                0
SSL alert PROTOCOL_VERSION rcvd:                  0
SSL alert INSUFFICIENT_SECURITY rcvd:             0
SSL alert INTERNAL_ERROR rcvd:                    0
SSL alert USER_CANCELED rcvd:                     0
SSL alert NO_RENEGOTIATION rcvd:                  0
SSL alert CLOSE_NOTIFY sent:                      0
SSL alert UNEXPECTED_MSG sent:                    0
SSL alert BAD_RECORD_MAC sent:                    0
SSL alert DECRYPTION_FAILED sent:                 0
SSL alert RECORD_OVERFLOW sent:                   0
SSL alert DECOMPRESSION_FAILED sent:              0
SSL alert HANDSHAKE_FAILED sent:                  0
SSL alert NO_CERTIFICATE sent:                    0
SSL alert BAD_CERTIFICATE sent:                   0
SSL alert UNSUPPORTED_CERTIFICATE sent:           0
SSL alert CERTIFICATE_REVOKED sent:               0
SSL alert CERTIFICATE_EXPIRED sent:               0
SSL alert CERTIFICATE_UNKNOWN sent:               0
SSL alert ILLEGAL_PARAMETER sent:                 0
SSL alert UNKNOWN_CA sent:                        0
SSL alert ACCESS_DENIED sent:                     0
SSL alert DECODE_ERROR sent:                      0
SSL alert DECRYPT_ERROR sent:                     0
SSL alert EXPORT_RESTRICTION sent:                0
SSL alert PROTOCOL_VERSION sent:                  0
SSL alert INSUFFICIENT_SECURITY sent:             0
SSL alert INTERNAL_ERROR sent:                    0
SSL alert USER_CANCELED sent:                     0
SSL alert NO_RENEGOTIATION sent:                  0
SSLv2 client hello received:                      0
SSLv3 client hello received:                      0
TLSv1 client hello received:                      0
SSLv3 negotiated protocol:                        0
TLSv1 negotiated protocol:                    19855
SSLv3 full handshakes:                            0
SSLv3 resumed handshakes:                         0
Cipher sslv3_rsa_rc4_128_md5:                     0
Cipher sslv3_rsa_rc4_128_sha:                     0
Cipher sslv3_rsa_des_cbc_sha:                     0
Cipher sslv3_rsa_3des_ede_cbc_sha:                0
Cipher sslv3_rsa_exp_rc4_40_md5:                  0
Cipher sslv3_rsa_exp_des40_cbc_sha:               0
Cipher sslv3_rsa_exp1024_rc4_56_md5:              0
Cipher sslv3_rsa_exp1024_des_cbc_sha:             0
Cipher sslv3_rsa_exp1024_rc4_56_sha:              0
Cipher sslv3_rsa_aes_128_cbc_sha:                 0
Cipher sslv3_rsa_aes_256_cbc_sha:                 0
TLSv1 full handshakes:                        19855
TLSv1 resumed handshakes:                         0
Cipher tlsv1_rsa_rc4_128_md5:                 19855
Cipher tlsv1_rsa_rc4_128_sha:                     0
Cipher tlsv1_rsa_des_cbc_sha:                     0
Cipher tlsv1_rsa_3des_ede_cbc_sha:                0
Cipher tlsv1_rsa_exp_rc4_40_md5:                  0
Cipher tlsv1_rsa_exp_des40_cbc_sha:               0
Cipher tlsv1_rsa_exp1024_rc4_56_md5:              0
Cipher tlsv1_rsa_exp1024_des_cbc_sha:             0
Cipher tlsv1_rsa_exp1024_rc4_56_sha:              0
Cipher tlsv1_rsa_aes_128_cbc_sha:                 0
Cipher tlsv1_rsa_aes_256_cbc_sha:                 0
Total SSL client authentications:                 0
Failed SSL client authentications:                0
SSL client authentication cache hits:         19841
SSL static CRL lookups:                           0
SSL best effort CRL lookups:                      0
SSL CRL lookup cache hits:                        0
SSL revoked certificates:                         0
Total SSL server authentications:             19855
Failed SSL server authentications:                0
Internal error:                                   0
Handshake FlushRX operations:                     0
Handshake FlushTX operations:                     0
Xscale messages rcvd from ME:                 99275
Xscale messages sent to ME:                  119130
Finish msg split across ssl recs:                 0
Fasttx msg ring full:                             0
SSL_ME tx msg ring full:                          0
N2 encrypt_record:                                0
N2 decrypt_record:                            19855
N2 random:                                    39710
N2 handshake_hash:                            19855
N2 hash:                                          0
N2 gpop_master:                               19855
N2 gpop_import_master_secret:                     0
N2 gpop_pkcs1v15enc:                          19855
N2 gpop_pkcs1v15enc_crt:                          0
N2 gpop_finish:                               19855
N2 gpop_verify:                                   0
N2 gpop_pkcs1v15dec:                              0
N2 gpop_pkcs1v15dec_crt:                          0
N2 rsa_server_full:                               0
N2 resume:                                        0


Notes

See show stats crypto server for a description of these counters.


show stats crypto server

Sample Output

ACE30001/Admin# show stats crypto server
SSL Server Statistics: 
------------------
SSL alert CLOSE_NOTIFY rcvd:                      0
SSL alert UNEXPECTED_MSG rcvd:                    0
SSL alert BAD_RECORD_MAC rcvd:                    0
SSL alert DECRYPTION_FAILED rcvd:                 0
SSL alert RECORD_OVERFLOW rcvd:                   0
SSL alert DECOMPRESSION_FAILED rcvd:              0
SSL alert HANDSHAKE_FAILED rcvd:                  0
SSL alert NO_CERTIFICATE rcvd:                    0
SSL alert BAD_CERTIFICATE rcvd:                   0
SSL alert UNSUPPORTED_CERTIFICATE rcvd:           0
SSL alert CERTIFICATE_REVOKED rcvd:               0
SSL alert CERTIFICATE_EXPIRED rcvd:               0
SSL alert CERTIFICATE_UNKNOWN rcvd:               0
SSL alert ILLEGAL_PARAMETER rcvd:                 0
SSL alert UNKNOWN_CA rcvd:                        0
SSL alert ACCESS_DENIED rcvd:                     0
SSL alert DECODE_ERROR rcvd:                      0
SSL alert DECRYPT_ERROR rcvd:                     0
SSL alert EXPORT_RESTRICTION rcvd:                0
SSL alert PROTOCOL_VERSION rcvd:                  0
SSL alert INSUFFICIENT_SECURITY rcvd:             0
SSL alert INTERNAL_ERROR rcvd:                    0
SSL alert USER_CANCELED rcvd:                     0
SSL alert NO_RENEGOTIATION rcvd:                  0
SSL alert CLOSE_NOTIFY sent:                      0
SSL alert UNEXPECTED_MSG sent:                    0
SSL alert BAD_RECORD_MAC sent:                    0
SSL alert DECRYPTION_FAILED sent:                 0
SSL alert RECORD_OVERFLOW sent:                   0
SSL alert DECOMPRESSION_FAILED sent:              0
SSL alert HANDSHAKE_FAILED sent:                  0
SSL alert NO_CERTIFICATE sent:                    0
SSL alert BAD_CERTIFICATE sent:                   0
SSL alert UNSUPPORTED_CERTIFICATE sent:           0
SSL alert CERTIFICATE_REVOKED sent:               0
SSL alert CERTIFICATE_EXPIRED sent:               0
SSL alert CERTIFICATE_UNKNOWN sent:               0
SSL alert ILLEGAL_PARAMETER sent:                 0
SSL alert UNKNOWN_CA sent:                        0
SSL alert ACCESS_DENIED sent:                     0
SSL alert DECODE_ERROR sent:                      0
SSL alert DECRYPT_ERROR sent:                     0
SSL alert EXPORT_RESTRICTION sent:                0
SSL alert PROTOCOL_VERSION sent:                  0
SSL alert INSUFFICIENT_SECURITY sent:             0
SSL alert INTERNAL_ERROR sent:                    0
SSL alert USER_CANCELED sent:                     0
SSL alert NO_RENEGOTIATION sent:                  0
SSLv2 client hello received:                      0
SSLv3 client hello received:                      0
TLSv1 client hello received:                      0
SSLv3 negotiated protocol:                        0
TLSv1 negotiated protocol:                        0
SSLv3 full handshakes:                            0
SSLv3 resumed handshakes:                         0
Cipher sslv3_rsa_rc4_128_md5:                     0
Cipher sslv3_rsa_rc4_128_sha:                     0
Cipher sslv3_rsa_des_cbc_sha:                     0
Cipher sslv3_rsa_3des_ede_cbc_sha:                0
Cipher sslv3_rsa_exp_rc4_40_md5:                  0
Cipher sslv3_rsa_exp_des40_cbc_sha:               0
Cipher sslv3_rsa_exp1024_rc4_56_md5:              0
Cipher sslv3_rsa_exp1024_des_cbc_sha:             0
Cipher sslv3_rsa_exp1024_rc4_56_sha:              0
Cipher sslv3_rsa_aes_128_cbc_sha:                 0
Cipher sslv3_rsa_aes_256_cbc_sha:                 0
TLSv1 full handshakes:                            0
TLSv1 resumed handshakes:                         0
Cipher tlsv1_rsa_rc4_128_md5:                     0
Cipher tlsv1_rsa_rc4_128_sha:                     0
Cipher tlsv1_rsa_des_cbc_sha:                     0
Cipher tlsv1_rsa_3des_ede_cbc_sha:                0
Cipher tlsv1_rsa_exp_rc4_40_md5:                  0
Cipher tlsv1_rsa_exp_des40_cbc_sha:               0
Cipher tlsv1_rsa_exp1024_rc4_56_md5:              0
Cipher tlsv1_rsa_exp1024_des_cbc_sha:             0
Cipher tlsv1_rsa_exp1024_rc4_56_sha:              0
Cipher tlsv1_rsa_aes_128_cbc_sha:                 0
Cipher tlsv1_rsa_aes_256_cbc_sha:                 0
Total SSL client authentications:                 0
Failed SSL client authentications:                0
SSL client authentication cache hits:             0
SSL static CRL lookups:                           0
SSL best effort CRL lookups:                      0
SSL CRL lookup cache hits:                        0
SSL revoked certificates:                         0
Total SSL server authentications:                 0
Failed SSL server authentications:                0
Internal error:                                   0
Handshake FlushRX operations:                     0
Handshake FlushTX operations:                     0
Xscale messages rcvd from ME:                 99285
Xscale messages sent to ME:                  119142
Finish msg split across ssl recs:                 0
Fasttx msg ring full:                             0
SSL_ME tx msg ring full:                          0
N2 encrypt_record:                                0
N2 decrypt_record:                            19857
N2 random:                                    39714
N2 handshake_hash:                            19857
N2 hash:                                          0
N2 gpop_master:                               19857
N2 gpop_import_master_secret:                     0
N2 gpop_pkcs1v15enc:                          19857
N2 gpop_pkcs1v15enc_crt:                          0
N2 gpop_finish:                               19857
N2 gpop_verify:                                   0
N2 gpop_pkcs1v15dec:                              0
N2 gpop_pkcs1v15dec_crt:                          0
N2 rsa_server_full:                               0
N2 resume:                                        0


Notes

Field Description
SSL alert [type] rcvd/sent Number of times that the particular SSL alert message was received or sent.
SSLv2/v3 client hello received Number of ClientHello message received.
SSLv3/TLSv1 negotiated protocol Number of the times that the version used in the connection.
SSLv3 full handshakes Number of handshakes completed without errors.
SSLv3 resumed handshakes Number of handshakes resumed by using a session ID.
Cipher sslv3... Number of times that the cipher suite is used in the connection.
TLSv1 full handshakes Number of handshakes completed without errors.
TLSv1 resumed handshakes Number of handshakes resumed by using a session ID.
Cipher tlsv1... Number of times that the cipher suite is used in the connection.
Total SSL client authentications Number of authenticated client connections. This field increments only when displaying server statistics.
Failed SSL client authentications Number of client connections that failed authentication. This field increments only when displaying server statistics.
SSL client authentication cache hits Number of times that an authenticated client reconnects and a cache entry is found. This field increments only when displaying server statistics.
SSL static CRL lookups Number of lookups against a statically defined CRL.
SSL best effort CRL lookups Number of lookups using the best effort.
SSL CRL lookup cache hits Number of CRL lookups where the cache result was used.
SSL revoked certificates Number of revoked certificates encountered.
SSL CRL download failed Number of CRL downloads that failed.
Total SSL server authentications Number of server certificate authentications that the ACE attempted to perform. This field increments only when displaying client statistics.
Failed SSL server authentications Number of server certificate authentications that failed. This field increments only when displaying client statistics.
Handshake FlushRX/TX operations Number of times that the SSL handshake finished.
Xscale messages rcvd/sent for ME Number of messages passed between the SSL processors during the SSL handshake.
Xscale rcvd abort msg before hdshk Number of times that the SSL handshake was aborted.
Finish msg split across ssl recs Number of times that the SSL Finished message was split by the client into multiple SSL records.
SSLv3 client hello received Number of SSL V3 client hello messages received.
SSLv3 negotiated protocol Number of times the the SSL v3 protocol was selected in the handshake.
Cipher sslv3_rsa_rc4_128_md5 Number of times this cipher suite was selected.
N2 rsa_server_full N2 rsa_server_full is called as a normal part of the SSL handshake.

Notes

If "SSLv3 full handshakes" is less than rsa_server_full, then the difference is the number of connections that did not complete the SSL handshake. This should be close to the value of "Handshake failure alert".

The sum of "SSLv3 client hello received" and "TLSv1 client hello received", less "N2 rsa_server_full" gives you the number of handshakes that failed prior to getting to "N2 rsa_server_full".

See also show np 1 me-stats -snitrox and show np 1 me-stats -scrypto.


show stats loadbalance radius

Sample Output

slot3/radius# show stats loadbalance radius 
+-------------------------------------------------+
+--------- Radius Loadbalance statistics ---------+
+-------------------------------------------------+
 Total requests received              : 0
 Total responses received             : 0
 Total retry packets received         : 0
 Total header parse results received  : 0
 Total body parse results received    : 0
 Total data parse results received    : 0
 Total packets sent out               : 0
 Total sessions allocated             : 0
 Total sessions deleted               : 0
 Total username sticky added          : 0
 Total calling-station sticky added   : 0
 Total framed-ip sticky added         : 0
 Total end-user packet sticky success : 0
 Total end-user packet sticky failure : 0
 Total Acct-On/Off requests received  : 0
 Total Acct-On/Off responses received : 0
 Total Acct-On/Off with no rules      : 0
 Total Acct-On/Off req processing done: 0
 Total NULL packet received errors    : 0
 Total parse errors                   : 0

Notes

While most of the counters are self-explanatory, note the following points regarding several of the counters:

Field Description
Total proxy mapper errors This counter increments if there is no Radius header in the Radius message or if in other ways the message is malformed in such as way as to affect the header.
Total memory allocation failures This counter increments when a pmap entry cannot be allocated for the Radius transaction. There are 32K pmap entries for each IXP. A pmap entry is created for each Radius transaction. Note that there can be multiple radius transactions per UDP connection. If this stat is incrementing, then radius messages are being dropped, and the recommended workaround is to reduce the UDP inactivity timeout. (Make it UDP ports 1812 and 1813 for minimal impact on other connections).

WARNING: If the timeout is too short, the client-side connection and resulting pmap entry might be removed before the radius response message is received by the ACE. In this case, the response is dropped and "Total Layer7 rejections" counter will be incremented under show stats loadbalance radius.

Total end-user packet sticky failure This counter is incremented every time a Radius end-user packet is received which does not hit any sticky entry. In this case the pkt will be forwarded (routed) to the server and this counter will be incremented. To help clarify, this would happen with a config such as:
    sticky radius framed-ip GRP1
      replicate sticky
      serverfarm sf1

    class-map match-all catch_all_vip
      2 match virtual-address 0.0.0.0 0.0.0.0 any

    policy-map type loadbalance first-match L7_P
      class class-default
        sticky-serverfarm GRP1
 
    policy-map multi-match RAD_L7_VIP
      class RAD_L7-VIP
        loadbalance vip inservice
        loadbalance policy RAD_L7_pmap
        loadbalance vip icmp-reply
      class catch_all_vip
        loadbalance vip inservice
        loadbalance policy L7_P
        loadbalance vip icmp-reply
 
    interface vlan 440
      ip address 44.44.44.149 255.255.255.0
      service-policy input RAD_L7_VIP
      no shutdown

In this scenario, if there were no Radius FIP sticky entry, and then an end-user sent a packet to a server that matched the catch_all_vip (for example using directly the server IP address, not the VIP), it will match the catch_all_vip and the L7_P. A sticky lookup will be done but no sticky entry is found. Then this counter is incremented.

Total Valid Sticky Key This will increase every time an access-accept hits a pmap entry that has a valid sticky key in it (which basically means that the previous access-request was a retransmission).


show stats probe

Displays probe-related statistics.


Sample Output

ace-1/Admin# show stats probe

+------------------------------------------+
+----------- Probe statistics -------------+
+------------------------------------------+
 ----- icmp probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- tcp probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- udp probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- http probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- https probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- ftp probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- telnet probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- smtp probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- pop probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- dns probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- echo probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- imap probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- radius probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- finger probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

 ----- script probe ----
 Total probes sent       : 0            Total send failures   : 0
 Total probes passed     : 0            Total probes failed   : 0
 Total connect errors    : 0            Total conns refused   : 0
 Total RST received      : 0            Total open timeouts   : 0
 Total receive timeout   : 0

Notes

For a description of the fields, see the "Displaying Probe Information" chapter of the Application Control Engine Module Server Load-Balancing Configuration Guide.


show syn-cookie

Displays the SYN cookie-related statistics for each configured interface. A TCP SYN cookie is an initial sequence number calculated in response to a SYN request from a client and inserted in the SYN-ACK response. It provides an authentication mechanism of sorts for preventing SYN floods from a rogue client.


Sample Output

ACE30001/Admin# show syn-cookie

Interface vlan23
        Configured TCP Embryonic Connection Limit: 0
        Current number of Embryonic Connections: 0
        Number of TCP Syns Intercepted by SYN COOKIE: 0
        Number of TCP Acks Successfully Processed by SYN COOKIE: 0
        Failed Number of TCP Acks Processed by SYN COOKIE: 0
Interface vlan101
        Configured TCP Embryonic Connection Limit: 0
        Current number of Embryonic Connections: 0
        Number of TCP Syns Intercepted by SYN COOKIE: 0
        Number of TCP Acks Successfully Processed by SYN COOKIE: 0
        Failed Number of TCP Acks Processed by SYN COOKIE: 0
Interface vlan102
        Configured TCP Embryonic Connection Limit: 0
        Current number of Embryonic Connections: 0
        Number of TCP Syns Intercepted by SYN COOKIE: 0
        Number of TCP Acks Successfully Processed by SYN COOKIE: 0
        Failed Number of TCP Acks Processed by SYN COOKIE: 0



show system internal mts buffers details

Provides detailed information on MTS buffers. MTS Buffers are used for internal communications between processes and threads with the ACE. In general a buffer is allocated, sent, processed and freed quickly.

Sample Output

ACE30001/Admin# show system internal mts buffers details
Node/Sap/queue  Age(ms)         SrcNode         SrcSAP  DstNode         DstSAP  OPC     MsgId   MsgSize
sup/1106/nper   26497           0x601           1085    0x601           1106    4001    927650  300     
sup/1085/nper   26497           0x601           1085    0x601           1106    4001    927650  300     

Notes

It would be normal to see a handful of buffers in the output; however, it is most likely a problem when many are observed or the "Age" is very large. This would indicate that the buffer has been leaked. Once the ACE runs low on these MTS buffer, the ACE can reboot without a core, possibly making the Control Plane (CP) become unreachable which would lead to management access problems or other unusual issues.

In general, if there are a lot of buffers on this output with a large age you should notify Cisco TAC.


show system internal mts memory

Sample Output

ACE30001/Admin# show system internal mts memory
mts buffer manager statistics 

shared memory pool at c111b000-c211b000 (size: 16777216 bytes)
request_hi:      41476
request_lo:      0
mem_in_use:      0
mem_free:        16777216
num_buffers:     0
mem_in_use_hi:   131328
num_allocs:      1655510
num_frees:       1655510
num_successes:   1655510
num_failures:    0
frag_waste:      0
frag_high:       55308


Notes

Field Description
request_hi Highest sized memory block requested
request_lo Lowest sized memory block requested
mem_in_use Total memory allocated
mem_free Total memory free
num_buffers Total number of buffers allocated currently
mem_in_use_hi Highest level of memory in use ever
num_alloc Number of allocation requests
num_frees Number of free requests
num_successes Number of successful alloc requests
num_failures Number of failures
frag_waste Space currently wasted in internal fragmentation
frag_high Highest level of fragmentation ever reached


show system internal sysmgr service all details

Displays all the available services that have started on the ACE. Some examples of a service (or more accurately, "daemon") that run on ACE are: TACACS or RADIUS.


Sample Output

switch/Admin# show system internal sysmgr service all details

Service "Tacacs Daemon" ("tacacs", 2):
        UUID = 0xB6, PID = 923, SAP = 112
        State: SRV_STATE_HANDSHAKED (entered at time Fri Dec 28 20:12:25 2007).
        Restart count: 1
        Time of last restart: Fri Dec 28 20:12:23 2007.
        The service never crashed since the last reboot.	

Notes

The fields of interest are:

  • "Restart count" — This should always be a value of one on a customer ACE. In production environments if a service were to crash it would bring down ACE and generate a core dump.
  • "Time of last restart" — On a customer ACE this will be the time of the last core dump for this server if it has ever caused the ACE to crash and reboot. The reboot reason will be on the second line.

A process core will contain this output for the service that cored. On a running ACE this command would not be of interest.


show system kcache

Displays information on the cache used by various processes.

Sample Output

ACE30001/Admin# show system kcache
slabinfo - version: 1.1 (SMP)
kmem_cache            80     80    244    5    5    1 :  252  126
ipcp_pp_req_cache    118    118     64    2    2    1 :  252  126
ipcp_qentry_cache    213    339     32    3    3    1 :  252  126
mts-qentry-cache     213    339     32    3    3    1 :  252  126
mts-sapinfo-cache     14     14   1152    2    2    2 :   60   30
tl_mts_cache         339    339     32    3    3    1 :  252  126
scp_mts_cache        328    404     16    2    2    1 :  252  126
tcp_tw_bucket        280    280     96    7    7    1 :  252  126
tcp_bind_bucket      213    339     32    3    3    1 :  252  126
tcp_open_request     118    118     64    2    2    1 :  252  126
inet_peer_cache       59     59     64    1    1    1 :  252  126
ip_fib_hash          226    226     32    2    2    1 :  252  126
ip_dst_cache         260    260    192   13   13    1 :  252  126
arp_cache             30     30    128    1    1    1 :  252  126
blkdev_requests      160    160     96    4    4    1 :  252  126
nfs_write_data         0      0    416    0    0    1 :  124   62
nfs_read_data          0      0    384    0    0    1 :  124   62
nfs_page               0      0     96    0    0    1 :  252  126
journal_head         186    312     48    4    4    1 :  252  126
revoke_table         252    253     12    1    1    1 :  252  126
revoke_record        226    226     32    2    2    1 :  252  126
dnotify cache          0      0     20    0    0    1 :  252  126
file lock cache      120    120     96    3    3    1 :  252  126
fasync cache           0      0     16    0    0    1 :  252  126
uid_cache            226    226     32    2    2    1 :  252  126
skbuff_head_cache   1474   1600    192   80   80    1 :  252  126
sock                 496    496    896  124  124    1 :  124   62
sigqueue             261    261    132    9    9    1 :  252  126
cdev_cache           118    118     64    2    2    1 :  252  126
bdev_cache           118    118     64    2    2    1 :  252  126
mnt_cache            118    118     64    2    2    1 :  252  126
inode_cache         2254   2254    512  322  322    1 :  124   62
dentry_cache        2880   2880    128   96   96    1 :  252  126
filp                3150   3150    128  105  105    1 :  252  126
names_cache           22     22   4096   22   22    1 :   60   30
buffer_head        18698  28400     96  468  710    1 :  252  126
mm_struct            312    312    160   13   13    1 :  252  126
vm_area_struct      8788   9040     96  222  226    1 :  252  126
fs_cache             354    354     64    6    6    1 :  252  126
files_cache          198    198    416   22   22    1 :  124   62
signal_act            80     87   4128   27   29    4 :    0    0
size-131072(DMA)       0      0 131072    0    0   32 :    0    0
size-131072            3      4 131072    3    4   32 :    0    0
size-65536(DMA)        0      0  65536    0    0   16 :    0    0
size-65536            20     20  65536   20   20   16 :    0    0
size-32768(DMA)        0      0  32768    0    0    8 :    0    0
size-32768            15     15  32768   15   15    8 :    0    0
size-16384(DMA)        0      0  16384    0    0    4 :    0    0
size-16384            12     12  16384   12   12    4 :    0    0
size-8192(DMA)         0      0   8192    0    0    2 :    0    0
size-8192             22     25   8192   22   25    2 :    0    0
size-4096(DMA)         0      0   4096    0    0    1 :   60   30
size-4096             31     31   4096   31   31    1 :   60   30
size-2048(DMA)         0      0   2048    0    0    1 :   60   30
size-2048           1210   1210   2048  605  605    1 :   60   30
size-1024(DMA)         0      0   1024    0    0    1 :  124   62
size-1024            462    524   1024  118  131    1 :  124   62
size-512(DMA)          0      0    512    0    0    1 :  124   62
size-512            1696   1696    512  212  212    1 :  124   62
size-256(DMA)          0      0    256    0    0    1 :  252  126
size-256             324    450    256   26   30    1 :  252  126
size-128(DMA)          0      0    128    0    0    1 :  252  126
size-128            1890   1890    128   63   63    1 :  252  126
size-64(DMA)           0      0     64    0    0    1 :  252  126
size-64              641    767     64   12   13    1 :  252  126
size-32(DMA)           0      0     32    0    0    1 :  252  126
size-32             2938   2938     32   26   26    1 :  252  126

Notes

This is the same as the linux command 'cat /proc/slabinfo'.


show system kmem

Displays system kernel memory statistics. Using this command to monitor memory use over the course of several hours can help identify memory leaks in the system.

Sample Output

ACE30001/Admin# show system kmem
        total:    used:    free:  shared: buffers:  cached:
Mem:  847564800 591683584 255881216        0  3371008 168300544
Swap:        0        0        0
MemTotal:       827700 kB
MemFree:        249884 kB
MemShared:           0 kB
Buffers:          3292 kB
Cached:         164356 kB
SwapCached:          0 kB
Active:           4436 kB <<<
Inactive:       514964 kB
HighTotal:      655360 kB
HighFree:       166460 kB
LowTotal:       172340 kB
LowFree:         83424 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Committed_AS:   948556 kB


Notes

This output displays is the same information that is displayed via the linux command cat /proc/meminfo.

Memory values are in units of kBytes.

Field Description
MemTotal Total usable RAM: physical memory minus reserved memory and the kernel binary code.
MemFree The amount of physical RAM left unused by the system.
Buffers The amount of physical RAM used for file buffers.
Cached The amount of physical RAM used as cache memory. Memory in the pagecache (diskcache) minus SwapCache.
SwapCache The amount of Swap used as cache memory. Memory that once was swapped out, is swapped back in, but is still in the swapfile.
Active The total amount of buffer or page cache memory, that is active. This part of the memory is used recently and usually not reclaimed unless absolutely necessary.
Inactive The total amount of buffer or page cache memory that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes by the paging algorithm.
HighTotal The total amount of memory in the high region. This can vary based on the type of kernel used. Kernel uses indirect tricks to access the high memory region. Data cache can go in this memory region.
LowTotal The total amount of non-highmem memory.
LowFree The amount of free memory of the low memory region. This is the memory the kernel can address directly. All kernel datastructures need to go into low memory.
SwapTotal Total amount of physical swap memory.
SwapFree Total amount of swap memory free.
Dirty The total amount of memory waiting to be written back to the disk.
Writeback The total amount of memory actively being written back to the disk.
Committed_AS An estimate of how much RAM you would need to make a 99.99% guarantee that there never is Out of Memory (OOM) for this workload. Normally the kernel will overcommit memory. This parameter represents the worst case scenario value and also includes swap memory.


show system kmemtrack

Enables you to track memory allocations and frees in the kernel loadable modules (drivers).

Sample Output

ACE30001/Admin# show system kmemtrack

Kernel Module Memory Tracking 
-------------------------------------------------------------
Module           kmalloc       kfree         diff
klm_cpu_util     00000008      00000008      00000000
klm_ipcp         00001811      00001811      00000000
n2_drv           00000339      00000227      00000112
klm_bf_nvram     00000016      00000016      00000000
klm_idprom       00000071      00000071      00000000
klm_encdec       00000011      00000000      00000011
klm_modlock      00000001      00000001      00000000
klm_sysmgr-hb    00000002      00000000      00000002
klm_utaker       00006656      00006656      00000000
klm_mts          00248775      00247519      00001256
klm_scp          00000001      00000000      00000001
klm_tl           00119846      00119843      00000003
klm_sdwrap       00001838      00000018      00001820
klm_resdrv       00000001      00000001      00000000
klm_klib         00000004      00000000      00000004


Notes

The "diff" field is useful if you suspect that there is a memory leak in the kernel. Not just the absolute value in the diff column but continuously increasing value across successive invocations of this command.

  • show proc memory is for the user space processes
  • show proc kmemtrack is for drivers alone
  • show proc kmem is for whole system.

The operator should start with show system kmem to see if system is running low on memory. If it is one of the other two commands can shows who process or driver is taking up too much memory.


show system resources

Sample Output

ACE30001/Admin# show system resources
Load average:   1 minute: 0.06   5 minutes: 0.09   15 minutes: 0.09
Processes   :   116 total, 3 running
CPU states  :   50.5% user,   2.0% kernel,   47.5% idle
Memory usage:    827700K total,    587560K used,    240140K free
                   3332K buffers,  173304K cache

Notes

  • Load average: These stats correspond to the load averages from the linux "w" command
  • Processes: Total number of processes - do 'show proc cpu' for details
  • CPU states: CPU Percentages
  • Memory usage: These stats correspond to the linux command cat /proc/meminfo



show system skbtrack

This command displays the socket buffer (network buffer, or skbuffs) allocations in the kernel loadable modules. Skbuffs are the buffers in which the OS kernel handles network packets. The packet is received by the network card, put into an skbuff and then passed to the network stack, which uses the skbuff.


Sample Output

ACE30001/Admin# show system skbtrack  

Kernel Module skbuff Tracking 
-------------------------------------------------------------
Module           alloc         free          diff
n2_drv           00001090      00000000      00001090
klm_pseudo       00397987      00397987      00000000
klm_mts          00779257      00779257      00000000


Notes

  • Module: kernel loadable module (KLM) name
  • alloc: network buffer allocations
  • free: network buffer frees
  • diff: memory allocated minus free


show xlate

Sample Output

ACE30001/Admin# show xlate
TCP PAT from vlan32:10.25.49.131/14311 to vlan32:172.16.182.170/1304
NAT from vlan32:192.168.25.73 to vlan55:172.16.183.33 count:1

Notes

In the PAT line, the listed elements are as follows:

  • The client IP is 10.25.49.131
  • TCP port is 14311
  • Client VLAN is 32
  • Server VLAN is 32
  • PAT address is 172.16.182.170
  • PAT port = 1304

Similarly, in the NAT line, the client IP and VLAN is on the left and the server IP and VLAN is on the right.

Rating: 4.0/5 (5 votes cast)

Personal tools