Cisco WAAS Troubleshooting Guide for Release 4.1.3 and Later -- Troubleshooting the MAPI AO
This article describes how to troubleshoot the MAPI AO.
The MAPI accelerator optimizes Microsoft Outlook Exchange e-mail traffic. Exchange uses the EMSMDB protocol, which is layered on MS-RPC, which in turn uses either TCP or HTTP (unsupported) as the low level transport.
The MAPI AO supports Microsoft Outlook 2000 through 2007 clients for both cached and noncached mode traffic. Secure connections that use message authentication (signing) or encryption are not accelerated by the MAPI AO. Such connections and connections from older clients are handed off to the generic AO for TFO optimizations. Additionally, Outlook Web Access (OWA) and Exchange-Exchange connections are not supported.
Note: Microsoft Outlook 2007 has encryption enabled by default. You must disable encryption to benefit from the MAPI application accelerator. In Outlook, choose Tools > E-mail Accounts, choose View or Change Existing E-mail Accounts, and then click Next. Choose the Exchange account, and then click Change. Click More Settings, and then click the Security tab. Uncheck the Encrypt data between Microsoft Office Outlook and Microsoft Exchange Server check box, as shown in Figure 1.
Alternatively, you can disable encryption for all users of an Exchange Server by using a Group Policy.
- Figure 1. Disabling Encryption in Outlook 2007
In the following cases, the MAPI AO does not handle a connection:
- Encrypted connection (handed off to the generic AO)
- Unsupported client (handed off to the generic AO)
- Unrecoverable parsing error. All TCP connections between the client and server service are disconnected. When the client reconnects, all connections are handed off to the generic AO.
- Client attempts to establish a new association group on the connection when the WAE is overloaded.
- Client establishes a connection when the WAE is overloaded and MAPI reserved connection resources are not available.
The Outlook client and server interact in a session over a group of TCP connections called an association group. Within an association group, object accesses can span across any connection and connections are dynamically created and released as needed. A client can have more than one association group open at the same time to different servers or the same server. (Public folders are deployed on different servers from the mail store.)
It is essential that all MAPI connections within an association group go through the same pair of WAEs in the branch and data center. If some connections within an association group do not go through the MAPI AO on these WAEs, the MAPI AO would not see the transactions performed on those connections and the connections are said to "escape" the association group. For this reason, the MAPI AO should not be deployed on serially clustered inline WAEs that form a high availability group.
The symptoms of MAPI connections that escape their WAE association group are Outlook error symptoms such as duplicate messages or Outlook stops responding.
During a TFO overload condition, new connections for an existing association group would be passed through and escape the MAPI AO, so the MAPI AO reserves a number of connection resources in advance to minimize the impact of an overload condition. For more details about reserved MAPI connections and their impact on device overload, see the section "MAPI Application Accelerator Reserved Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
Verify the general AO configuration and status with the show accelerator and show license commands, as described in the Troubleshooting Application Acceleration article. The Enterprise license is required for MAPI accelerator operation and the EPM application accelerator must be enabled.
Next, verify the status specific to the MAPI AO by using the show accelerator mapi command, as shown in Figure 2. You want to see that the MAPI AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem.
- Figure 2. Verifying the MAPI Accelerator Status
Use the show statistics accelerator epm command to verify that the EPM AO is functional. Check that the Total Handled Connections, Total Requests Successfully Parsed, and Total Responses Successfully Parsed counters increase when a client is started.
Use the show running-config command to verify that the MAPI and EPM traffic policies are properly configured. You want to see accelerate mapi for the Email-and-Messaging application action and you want to see the MS-EndPortMapper classifier and traffic policy defined, as follows:
WAE674# sh run | include mapi map adaptor EPM mapi name Email-and-Messaging All action optimize full accelerate mapi WAE674# sh run | begin MS-EndPointMapper ...skipping classifier MS-EndPointMapper match dst port eq 135 exit WAE674# sh run | include MS-EndPointMapper classifier MS-EndPortMapper name Other classifier MS-EndPortMapper action optimize DRE no compression none accelerate MS-port-mapper
Use the show policy-engine application dynamic command to verify that dynamic match rules exist, as follows:
- Look for a rule with User ID: EPM and Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da.
- The Flows field indicates the total number of active connections to the Exchange service.
- For each MAPI client you should see a separate entry with the User ID: MAPI.
Use the show statistics connection optimized mapi command to check that the WAAS device is establishing optimized MAPI connections. Verify that "M" appears in the Accel column for MAPI connections, which indicates that the MAPI AO was used, as follows:
WAE674# show stat conn opt mapi Current Active Optimized Flows: 2 Current Active Optimized TCP Plus Flows: 1 Current Active Optimized TCP Only Flows: 1 Current Active Optimized TCP Preposition Flows: 0 Current Active Auto-Discovery Flows: 0 Current Reserved Flows: 12 <---------- Added in 4.1.5 Current Active Pass-Through Flows: 0 Historical Flows: 161 D:DRE,L:LZ,T:TCP Optimization RR:Total Reduction Ratio A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO ConnID Source IP:Port Dest IP:Port PeerID Accel RR 342 10.56.94.101:4506 10.10.100.100:1456 0:1a:64:d3:2f:b8 TMDL 61.0% <-----Look for "M"
Note: In version 4.1.5, the Current Reserved Flows counter was added in the output. This counter refers to the number of reserved MAPI connection resources on the WAE that are currently unused but set aside for future MAPI connections. For more details about reserved MAPI connections and their impact on device overload, see the section "MAPI Application Accelerator Reserved Connections Impact on Overload" in the Troubleshooting Overload Conditions article.
If you observe connections with "TGDL" in the Accel column, these connections were pushed down to the generic AO and optimized with transport optimizations only. If these are connections that you expected to be handled by the MAPI AO, it may be because they are encrypted MAPI connections. To check on the number of encrypted MAPI connections that have been requested, use the show statistics accelerator mapi command as follows:
wae# sh stat accel mapi MAPI: Global Statistics ----------------- Time Accelerator was started: Thu Nov 5 19:45:19 2009 Time Statistics were Last Reset/Cleared: Thu Nov 5 19:45:19 2009 Total Handled Connections: 8615 Total Optimized Connections: 8614 Total Connections Handed-off with Compression Policies Unchanged: 0 Total Dropped Connections: 1 Current Active Connections: 20 Current Pending Connections: 0 Maximum Active Connections: 512 Number of Synch Get Buffer Requests: 1052 Minimum Synch Get Buffer Size (bytes): 31680 Maximum Synch Get Buffer Size (bytes): 31680 Average Synch Get Buffer Size (bytes): 31680 Number of Read Stream Requests: 3844 Minimum Read Stream Buffer Size (bytes): 19 Maximum Read Stream Buffer Size (bytes): 31744 Average Read Stream Buffer Size (bytes): 14556 Minimum Accumulated Read Ahead Data Size (bytes): 0 Maximum Accumulated Read Ahead Data Size (bytes): 1172480 Average Accumulated Read Ahead Data Size (bytes): 594385 Local Response Count: 20827 Average Local Response Time (usec): 250895 Remote Response Count: 70486 Average Remote Response Time (usec): 277036 Current 2000 Accelerated Sessions: 0 Current 2003 Accelerated Sessions: 1 Current 2007 Accelerated Sessions: 0 Secured Connections: 1 <-----Encrypted connections Lower than 2000 Sessions: 0 Higher than 2007 Sessions: 0
You can find the IP addresses of clients requesting encrypted MAPI connections in the syslog by searching for messages like the following:
2009 Jan 5 13:11:54 WAE512 mapi_ao: %WAAS-MAPIAO-3-132104: (929480) Encrypted connection. Client ip: 10.36.14.82
You can view the MAPI connection statistics by using the show statistics connection optimized mapi detail command as follows:
WAE674# show stat conn opt mapi detail Connection Id: 1830 Peer Id: 00:14:5e:84:24:5f Connection Type: EXTERNAL CLIENT Start Time: Thu Jun 25 06:32:27 2009 Source IP Address: 10.10.10.10 Source Port Number: 3774 Destination IP Address: 10.10.100.101 Destination Port Number: 1146 Application Name: Email-and-Messaging <-----Should see Email-and-Messaging Classifier Name: **Map Default** Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da <-----Should see this UUID Directed Mode: FALSE Preposition Flow: FALSE Policy Details: Configured: TCP_OPTIMIZE + DRE + LZ Derived: TCP_OPTIMIZE + DRE + LZ Peer: TCP_OPTIMIZE + DRE + LZ Negotiated: TCP_OPTIMIZE + DRE + LZ Applied: TCP_OPTIMIZE + DRE + LZ Accelerator Details: Configured: MAPI <-----Should see MAPI configured Derived: MAPI Applied: MAPI <-----Should see MAPI applied Hist: None Original Optimized -------------------- -------------------- Bytes Read: 4612 1973 Bytes Written: 4086 2096 . . .
Local and remote response counts and average response times are shown in this output:
. . . MAPI : 1830 Time Statistics were Last Reset/Cleared: Thu Jun 25 06:32:27 2009 Total Bytes Read: 46123985 Total Bytes Written: 40864046 Number of Synch Get Buffer Requests: 0 Minimum Synch Get Buffer Size (bytes): 0 Maximum Synch Get Buffer Size (bytes): 0 Average Synch Get Buffer Size (bytes): 0 Number of Read Stream Requests: 0 Minimum Read Stream Buffer Size (bytes): 0 Maximum Read Stream Buffer Size (bytes): 0 Average Read Stream Buffer Size (bytes): 0 Minimum Accumulated Read Ahead Data Size (bytes): 0 Maximum Accumulated Read Ahead Data Size (bytes): 0 Average Accumulated Read Ahead Data Size (bytes): 0 Local Response Count: 0 <---------- Average Local Response Time (usec): 0 <---------- Remote Response Count: 19 <---------- Average Remote Response Time (usec): 89005 <---------- . . .
MAPI AO Logging
The following log files are available for troubleshooting MAPI AO issues:
- Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
- Debug log files: /local1/errorlog/mapiao-errorlog.current (and mapiao-errorlog.*)
For easier debugging, you should first set up an ACL to restrict packets to one host.
WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10
To enable transaction logging, use the transaction-logs configuration command as follows:
wae(config)# transaction-logs flow enable wae(config)# transaction-logs flow access-list 150
You can view the end of a transaction log file by using the type-tail command as follows:
wae# type-tail tfo_log_10.10.11.230_20090715_130000.txt Wed Jul 15 19:12:35 2009 :2289 :10.10.10.10 :3740 :10.10.100.101 :1146 :OT :END :EXTERNAL CLIENT :(MAPI) :822 :634 :556 :706 Wed Jul 15 19:12:35 2009 :2289 :10.10.10.10 :3740 :10.10.100.101 :1146 :SODRE :END :730 :605 :556 :706 :0 Wed Jul 15 19:12:35 2009 :2290 :10.10.10.10 :3738 :10.10.100.101 :1146 :OT :END :EXTERNAL CLIENT :(MAPI) :4758 :15914 :6436 :2006 Wed Jul 15 19:12:35 2009 :2290 :10.10.10.10 :3738 :10.10.100.101 :1146 :SODRE :END :4550 :15854 :6436 :2006 :0 Wed Jul 15 19:12:35 2009 :2284 :10.10.10.10 :3739 :10.10.100.101 :1146 :OT :END :EXTERNAL CLIENT :(MAPI) :1334 :12826 :8981 :1031
To set up and enable debug logging of the MAPI AO, use the following commands.
NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a production environment.
You can enable detailed logging to the disk as follows:
WAE674(config)# logging disk enable WAE674(config)# logging disk priority detail
You can enable debug logging for connections in the ACL as follows:
WAE674# debug connection access-list 150
The options for MAPI AO debugging are as follows:
WAE674# debug accelerator mapi ? all enable all MAPI accelerator debugs Common-flow enable MAPI Common flow debugs DCERPC-layer enable MAPI DCERPC-layer flow debugs EMSMDB-layer enable MAPI EMSMDB-layer flow debugs IO enable MAPI IO debugs ROP-layer enable MAPI ROP-layer debugs ROP-parser enable MAPI ROP-parser debugs RPC-parser enable MAPI RPC-parser debugs shell enable MAPI shell debugs Transport enable MAPI transport debugs Utilities enable MAPI utilities debugs
You can enable debug logging for MAPI connections and then display the end of the debug error log as follows:
WAE674# debug accelerator mapi Common-flow WAE674# type-tail errorlog/mapiao-errorlog.current follow