Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later -- Troubleshooting the NFS AO

From DocWiki

Revision as of 00:29, 11 October 2012

This article describes how to troubleshoot the NFS AO.

NFS Accelerator Troubleshooting

The NFS accelerator optimizes NFSv3 traffic. Other NFS versions are not optimized by the NFS AO.

You can verify the general AO configuration and status with the show accelerator and show license commands, as described in the Troubleshooting Application Acceleration article. The Enterprise license is required for NFS accelerator operation.

Next, verify the status specific to the NFS AO by using the show accelerator nfs command, as shown in Figure 1. You want to see that the NFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem.

Figure 1. Verifying the NFS Accelerator Status

Use the show running-config command to verify that the NFS traffic policy is properly configured. You want to see accelerate nfs for the File-System application classifier NFS action and you want to see appropriate match conditions listed for the NFS classifier, as follows:

WAE674# sh run | include NFS
      name File-System classifier NFS action optimize full accelerate nfs       <-------------

WAE674# sh run | begin NFS
...skipping
   classifier NFS
      match dst port eq 2049                                                    <-------------
   exit

Use the show statistics connection optimized nfs command to check that the WAAS device is establishing optimized NFS connections. Verify that "N" appears in the Accel column for NFS connections, which indicates that the NFS AO was used.

WAE674# sh stat conn opt nfs
D:DRE,L:LZ,T:TCP Optimization,
C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO,

ConnID  Local IP:Port         Remote IP:Port        PeerID             Accelerator 
582     10.56.94.101:33606    10.56.94.80:2049      0:1a:64:d3:2f:b8   NTDL         <-----Look for "N"

Use the show statistics accelerator nfs command to verify the following:

• The NFS traffic is NFSv3. Look at the Total RPC Calls per NFS Version field. The output of that field is an array of 5 values, and you want to see mostly NFSv3 traffic, which is reported in the 4th counter. High numbers in other array positions signify other NFS versions.
• NFS traffic is not encrypted. Look at the Total RPC Calls per Authentication Flavor field. The output of that field is an array of 4 values, and you want to see mostly unencrypted traffic, which corresponds to the first 3 counters. A high number in the last counter signifies encrypted NFS traffic. Also check the Total RPC Calls with Unknown Authentication Flavor field, where you want to see 0 or a small number because these connections are not optimized.
• The NFS connection is asynchronous. Verify that the Percentage of Requests Served Locally field is nonzero.

WAE# sh statistics accelerator nfs
 
NFS:
  Global Statistics
  -----------------
  Time Accelerator was started:                                      Fri Oct 23
16:40:06 2009
  Time Statistics were Last Reset/Cleared:                           Fri Oct 23
16:40:06 2009
  Total Handled Connections:                                         170      
  Total Optimized Connections:                                       170      
  Total Connections Handed-off with Compression Policies Unchanged:  0        
  Total Dropped Connections:                                         0        
  Current Active Connections:                                        0        
  Current Pending Connections:                                       0        
  Maximum Active Connections:                                        13       
  Total RPC Calls per Authentication Flavor:                         65        
298544    0         0                                                          <----Should see 0 or few in last field 
  Total RPC Calls with Unknown Authentication Flavor:                0         <----Should see 0 or few
  Total RPC Calls per NFS Version:                                   0         
0         0         298609    0                                                <----Should see 0 or few in first two and last fields
  Total RPC Calls with Unknown NFS Version:                          0         <----Should see 0 or few
  Total Requests:                                                    298609   
  Total Local Replies:                                               191713   
  Percentage of Requests Served Locally:                             64        <----Should be nonzero
  Percentage of Requests Served Remotely:                            36       
  Average Time to Generate Local READ Reply (ms):                    15       
  Average Time to Generate Local WRITE Reply (ms):                   0        
  Average Time to Generate Local GETATTR Reply (ms):                 0        
  Average Time to Generate Local Reply (ms):                         0        
  Average Time to Receive Remote Reply (ms):                         10       
  Meta-Data Cache Access Count:                                      206017   
  Meta-Data Cache Hit Count:                                         191673   
  Remaining Number Of Entries in Meta-Data Cache:                    128926   
  Meta-Data Cache Hit Ratio:                                         93        

You can view the NFS connection statistics by using the show statistics connection optimized nfs detail command as follows:

WAE674# show stat conn opt nfs detail
Connection Id:            1916
   Peer Id:                  00:14:5e:84:24:5f
   Connection Type:          EXTERNAL CLIENT
   Start Time:               Thu Jun 25 07:09:09 2009
   Source IP Address:        10.10.10.20
   Source Port Number:       928
   Destination IP Address:   10.10.100.102
   Destination Port Number:  2049
   Application Name:         File-System                                  <-----Should see File-System
   Classifier Name:          NFS                                          <-----Should see NFS
   Map Name:                 basic
   Directed Mode:            FALSE
   Preposition Flow:         FALSE
   Policy Details:
          Configured:        TCP_OPTIMIZE + DRE + LZ
             Derived:        TCP_OPTIMIZE + DRE + LZ
                Peer:        TCP_OPTIMIZE + DRE + LZ
          Negotiated:        TCP_OPTIMIZE + DRE + LZ
             Applied:        TCP_OPTIMIZE + DRE + LZ
   Accelerator Details:
               Configured:   NFS                                          <-----Should see NFS configured
                  Derived:   NFS
                  Applied:   NFS                                          <-----Should see NFS applied
                     Hist:   None
                                               Original            Optimized
                                   -------------------- --------------------
   Bytes Read:                                     5120                 4639
   Bytes Written:                                 28136                 1407
. . .

NFS : 1916

   Time Statistics were Last Reset/Cleared:                           Thu Jun 25
 07:09:09 2009
   Total Bytes Read:                                                  5120
28136
   Total Bytes Written:                                               28136
5120
   Bit Flags for I/O state:                                           19
   Histogram of Buffers Read From Local Endpoint:                     31
1         0         0         0
   Total NFS Requests:                                                32
   Total Replies Served Locally:                                      4
   Percentage of Requests Served Locally:                             12
   Percentage of Requests Served Remotely:                            88
   Average Time to Generate Local READ Reply (ms):                    0
   Average Time to Generate Local WRITE Reply (ms):                   0
   Average Time to Generate Local GETATTR Reply (ms):                 0
   Average Time to Generate Local Reply (ms):                         0
   Average Time to Receive Remote Reply (ms):                         103
   Total RPC Procedure Calls:                                         0
9         0         10        7         0         4         1         0
0         0         0         0         0         0         0         0
1         0         0         0         0
. . .

 Total Unknown RPC Procedure Calls:                                    0
  Total Write RPCs Using Stable-how Enumerated Values:               0
0         1
  Total WRITE RPCs with Invalid Stable-how Value:                    0
  Bytes Buffered for READ Purpose:                                   0
  Start Time of Session:                                             Thu Jun 25
07:09:09 2009

  Meta-Data Cache Access Count:                                      9
  Meta-Data Cache Hit Count:                                         4
  Remaining Number Of Entries in Meta-Data Cache:                    1000
  Meta-Data Cache Hit Ratio:                                         44
  Current number of entries in Meta-Data Cache:                      0
. . . 

NFS AO Logging

The following log files are available for troubleshooting NFS AO issues:

• Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
• Debug log files: /local1/errorlog/nfsao-errorlog.current (and nfsao-errorlog.*)

For easier debugging, you should first set up an ACL to restrict packets to one host.

WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any
WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10

To enable transaction logging, use the transaction-logs configuration command as follows:

wae(config)# transaction-logs flow enable
wae(config)# transaction-logs flow access-list 150

You can view the end of a transaction log file by using the type-tail command.

To set up and enable debug logging of the NFS AO, use the following commands.

NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a production environment.

You can enable detailed logging to the disk as follows:

WAE674(config)# logging disk enable
WAE674(config)# logging disk priority detail

You can enable debug logging for connections in the ACL as follows:

WAE674# debug connection access-list 150

The options for NFS AO debugging are as follows:

WAE674# debug accelerator nfs ?
  all               enable all accelerator debugs
  async-write       enable async write optimization debugs
  attributes-cache  enable attributes-cache optimization debugs
  nfs-v3            enable NFSv3 layer debugs
  read-ahead        enable read ahead optimization debugs
  rpc               enable RPC layer debugs
  shell             enable shell (infra) debugs
  utils             enable utils debugs

You can enable debug logging for NFS connections and then display the end of the debug error log as follows:

WAE674# debug accelerator nfs all
WAE674# type-tail errorlog/nfsao-errorlog.current follow