Contact Center Networking: Offload, Receive Side Scaling and Chimney

From DocWiki

(Difference between revisions)
Jump to: navigation, search
(Checking Windows 2003)
(Checking Windows 2008)
 
Line 88: Line 88:
*Open a CMD session with elevated Administrator rights and issue the following command: 
*Open a CMD session with elevated Administrator rights and issue the following command: 
-
<pre>netsh int sh offload
+
<pre>netsh int ip sh offload
</pre>  
</pre>  
If any offload feature is enabled it will be listed underneath each NIC that it is enabled for.  
If any offload feature is enabled it will be listed underneath each NIC that it is enabled for.  

Latest revision as of 18:08, 19 October 2012


Contents

What are they?

Offload: The TCP Offload Engine

The TCP/IP Offload Engine (TOE) was introduced in the early 1990's however take up within the industry did not really start until early 2000. It was originally designed to provide the capability to alleviate the mainboard processor(s) from performing checksum calculations to ASICs located on a TOE-supporting Network Interface Controller (NIC).

As a general rule of thumb, for TCP/IP it takes 1 CPU cycle (1Hz) to process 1 bit in 1 second. This means that at gigabit speeds, you are looking at roughly 1GHz used to process 1Gb/s of traffic. This is not so much of a problem at FastEthernet or even GigabitEthernet speeds but does become a serious concern with 10GigabitEthernet Networks.

Receive Side Scaling

This feature was introduced by Microsoft to prevent all packets from being handled and processed by a single CPU core in a multi-core computer. It uses a complex hashing algorithm to force the routing of packets to different cores.

Microsoft TCP Chimney Offload & NetDMA

Microsoft TCP Chimney Offload is a partial offload feature that Microsoft introduced with Windows 2003 & Vista. A good overview of the architecture of TCP Chimney can be found here: MSDN Microsoft TCP Chimney Architecture

Network (NetDMA) or Remote Direct Memory Access (RDMA) over TCP allows local applications to use the TOE-enabled NIC to bypass the network stack and place data directly in to/out of a remote applications memory.


Please also see:

Cisco Field Notice: FN - 63215 - ICM/UCCE Potential Loss of Server Network Connection

Cisco Field Notice: FN - 62839 - Windows 2003 With SP2 - System Freezes With Blue Screen When Running CSA and Some Hardware

The Microsoft Windows Server 2003 Scalable Networking Pack release

Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008

Receive-Side Scaling Enhancements in Windows Server 2008

Why is this important?

While the intent of these features is to increase network performance, they have been known to cause significant performance degradation as load increases on Windows 2003 and Windows 2008 servers running any Cisco Unified Contact Center software. Cisco has released a number of field notices regarding these issues and steps to address the settings. There is also plenty of anecdotal evidence on the Internet that other non-Cisco applications are impacted by inconsistencies between the OS and the TOE-enabled NIC.

Cisco's Position on these Settings

Cisco advises that your first step to address any issues that are potentially associated with any of these features is to update NIC drivers & firmware. If the symptoms persist, then you can look to disable these settings per the instructions below.

IMPORTANT! While no adverse affect has been reported over time in the field with these changes being made, Cisco hasn't tested the impact of these changes thus cannot guarantee that overall system performance will not be degraded.

How to Check they are enabled?

You have a number of means to check that these features are enabled.

Checking Windows 2003

For Windows 2003 you have the following means to check:

Checking NIC Offload via CMD:

  • Open a CMD session with elevated Administrator rights and issue the following command:
netsh int ip sh offload

If any offload feature is enabled it will be listed underneath each NIC that it is enabled for.

You also need to confirm what values have been set in the NIC device driver. To do this you need to complete the following checks:

  1. Click Start, click Run, type ncpa.cpl, and then click OK.
  2. Right-click a network adapter object, and then click Properties.
  3. Click Configure, and then click the Advanced tab.
  4. In the Property list, click Receive Side Scaling to confirm if it is enabled or not.
  5. In the Property list, click each Offload option (IPv4 Checksum/TCP Offload/Large Send Offload, etc) to confirm if it is enabled or not.
  6. Repeat steps for Public and Private interfaces.

Below are two examples of the NIC settings where various Offload features are listed:

HP NC7781 NIC.JPG
Intelpro1000mt.JPG

To check whether RSS & TCP Chimney is enabled in the Registry you need check the following registry settings:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 
    EnableTCPChimney = 1 (or it does not exist) 
    EnableRSS = 1 (or it does not exist) 
    EnableTCPA = 1 (or it does not exist) 
    DisableTaskOffload = 0 (or it does not exist)

Checking Windows 2008

For Windows 2008 you have the following means to check: 

Checking Offload via CMD: 

  • Open a CMD session with elevated Administrator rights and issue the following command: 
netsh int ip sh offload

If any offload feature is enabled it will be listed underneath each NIC that it is enabled for.

Checking RSS & MS TCP Chimney Offload via CMD:

  • Open a CMD session with elevated Administrator rights and issue the following command:
netsh int tcp show global

As with Windows 2003, you also need confirm whether RSS and Offload features have been enabled in the NIC. To do this you need follow the below steps:

  • Click Start, click Run, type ncpa.cpl, and then click OK.
  • Right-click a network adapter object, and then click Properties.
  • Click Configure, and then click the Advanced tab.
  • In the Property list, click Receive Side Scaling to confirm if it is enabled or not.
  • In the Property list, click each Offload option (IPv4 Checksum/TCP Offload/Large Send Offload, etc) to confirm if it is enabled or not.
  • Repeat steps for Public and Private interfaces.

How to Disable these features?

Depending on the operating system used you will need to disable these features in different places. Below we have split out the requirements for Windows 2003 and Windows 2008. Note that when we request you disable checksum offload in the Network Interface Card, you should disable ANY configurable option with the word Offload.

Windows 2003

Here are the collated recommendations for Windows 2003 SP2 servers if they are part of a Contact Centre deployment.

  • NIC Device driver updated to latest version
  • Ensure KB948496 is downloaded, installed and the server rebooted
  • Disable RSS and TCP Offload in the network adapter driver, with these steps:
    1. Click Start, click Run, type ncpa.cpl, and then click OK.
    2. Right-click a network adapter object, and then click Properties.
    3. Click Configure, and then click the Advanced tab.
    4. In the Property list, click Receive Side Scaling, click Disable in the Value list, and then click OK.
    5. In the Property list, click each Offload option (IPv4 Checksum/TCP Offload/Large Send Offload, etc.), click Disable or Off in the Value list, and then click OK. Any option with Offload in the description should be disabled.
    6. Repeat steps for Public and Private interfaces.
  • Check the registry to confirm settings applied after KB patch. You should see the following three registry values:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters     
    EnableTCPChimney = 0 
    EnableRSS = 0 
    EnableTCPA = 0 
  • Add the following key in the registry to globally disable the TCP Offload Engine:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters     
    DisableTaskOffload = 1 (DWORD) 
  • You must reboot the server after making this change and check settings in TCP using the following command:
netsh int ip show offload

You should only see a description of each NIC installed in the server with nothing listed below each NIC entry.

These changes have shown to provide a significant improvement in buffering and latency issues across private network links.


Windows 2008

Here is the recommendations for Windows 2008 servers if they are part of a Contact Centre deployment.

  • Ensure that the NIC device driver is updated to latest version
  • Disable RSS and TCP Offload in the network adapter driver, with these steps:
    1. Click Start, click Run, type ncpa.cpl, and then click OK.
    2. Right-click a network adapter object, and then click Properties.
    3. Click Configure, and then click the Advanced tab.
    4. In the Property list, click Receive Side Scaling, click Disable in the Value list, and then click OK.
    5. In the Property list, click each Offload option (IPv4 Checksum/TCP Offload/Large Send Offload, etc), click Disable or Off in the Value list, and then click OK. Any option with Offload in the description should be disabled.
  • Repeat steps for Public and Private interfaces.
  • Open a CMD shell with elevated Administrator rights
  • Issue the following command:
netsh int tcp set global chimney=disabled rss=disabled
  • Add the following key in the registry to globally disable the TCP Offload Engine:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters     
    DisableTaskOffload = 1 (32bit DWORD) 
  • You must reboot the server after making this change and check settings in TCP using the following command:
netsh int ip show offload

You should only see a description of each NIC installed in the server with nothing listed below each NIC entry.

Rating: 4.7/5 (18 votes cast)

Personal tools