WAAS - Operations Guide and Automation

From DocWiki

Jump to: navigation, search

Contents

Operations TOIs

Common errors in WAAS Central Manager

Accelerator SSL_ITPS not licensed.

Fix:


Accelerator WAN_SECURE keepalive timeout.

Fix:


AMG host lookup failed.

Fix:


amg_time_skew (error from "sh alarms" on CLI of device)

Fix: Configure NTP servers on the WAVE device (see example below from RTP Infinite Video Cluster)


rtp10-cd-rbb-wave10#config t

rtp10-cd-rbb-wave10(config)#ntp server 10.81.254.202

rtp10-cd-rbb-wave10(config)#ntp server 72.163.32.44

rtp10-cd-rbb-wave10(config)#end


Authentication failed while communicating with AppNav-XE Controller device.

Fix:


Identity not configured for domain.

Fix: Login to device


Kernel crash files were detected.

Fix: Open TAC case.


Licensing errors

Fix: Connect to physical / ISR WAAS via command line. Input "crypto pki managed-store initialize" command.


low_disk_space (error from "sh alarms" on CLI of device)

Fix: Input "sh disks details" from WAAS CLI. Check if any of the mounted file systems are close to 100% in use. If so, attempt to remove some TFO log files using the below command:


hgh01-waas-wae1#delfile tfo_log_<WAAS_IP_ADDRESS>_<RECENT_DATE - e.g. 20170131>*

Do you really want to delete all the matching files? [y/n]y

hgh01-waas-wae1#


Management server is unreachable.

Fix:


Management server is offline.

Fix: Reinstall ISR-WAAS from scratch.


mstore_key_retrieval (error from "sh alarms" on CLI of device)

Fix: Connect to physical / ISR WAAS via command line. Input "crypto pki managed-store initialize" command.


User space Core files were detected.

Fix: On devices with this error in CM, they will have a core dump file on the device. You can confirm this by logging into CLI and checking in the core_dir directory:

vmct04-wan-gw2-ISR-WAAS#dir core_dir

     size          time of last change             name               

------------------------- -----------
       618174  Fri May 12 04:32:05 2017           core.srserverd64.6.2.3a.b39.cnbuild.911.tar.gz


You can do a sh alarms on the device, to confirm the presence of an alarm indicating that there is a core dump file stored in the core_dir directory:

vmct04-wan-gw2-ISR-WAAS#sh alarms

Critical Alarms:


None

Major Alarms:


       Alarm ID                 Module/Submodule               Instance
  ---------------             --------------------          ---------------
  1 core_dump                 sysmon                       core                     

Minor Alarms:


None

From here, if you delete the core file from the core_dir directory, you should see the alarm clear within a short space of time, first from sh alarms on the CLI and then from the Central Manager:

vmct04-wan-gw2-ISR-WAAS#delfile core_dir/core.srserverd64.6.2.3a.b39.cnbuild.911.tar.gz

You can see that after a few minutes, the alarm has disappeared:

vmct04-wan-gw2-ISR-WAAS#sh alarms

Critical Alarms:


None

Major Alarms:


None

Minor Alarms:


None


WAAS product license is missing.

Fix: Go to device on Central Manager, Admin > License Management, check Enterprise box.

How to upgrade WAAS hardware appliance (WAVE) to new code version


1) Login to WAAS Central Manager.
2) Go to "Devices" tab, and find the device you wish to upgrade.
3) Click on the "Edit" icon next to the name of the device.
4) Click on "(Update)" next to the current software version number.
5) From the next screen, you will see a list of versions to upgrade to. Select your new version number and click "Submit" at the bottom of the screen.
6) The device will begin downloading the latest version of software. This process will take some time, you can view its status from the Devices dashboard. You are looking to see the new version under the "Software Version" column when the new version has been installed. You should also see Management Status as "Online" and Device Status as Green. If you do not see the new software version, and instead see "ReloadNeeded", proceed to Step 7.
7) A manual reload of the device is also required. You will need to login to the device via CLI, and issue a reload command. The device can take a while to reload, keep an eye on its status from the Devices dashboard.
8) If following a reload, the device appears in the Devices dashboard with a Management Status of "Offline", you may need to wait longer. WAVE appliances have taken 24-36 hours to fully reload with correct version during past upgrades. If the device is still Offline after 24-36 hours you should try to access the WAAS device via the local site's console device. From here you may be able to tell whether the device is stuck in a boot cycle, or whether you may need to engage local hands for a hard reboot :

How to upgrade ISR-WAAS appliance to new code version


1) Login to WAAS Central Manager.
2) Go to "Devices" tab, and find the device you wish to upgrade.
3) Click on the "Edit" icon next to the name of the device.
4) Click on "(Update)" next to the current software version number.
5) From the next screen, you will see a list of versions to upgrade to. Select your new version number and click "Submit" at the bottom of the screen.
6) The device will begin downloading the latest version of software. This process will take some time, you can view its status from the Devices dashboard. You are looking to see the new version under the "Software Version" column when the new version has been installed. You should also see Management Status as "Online" and Device Status as Green. If you do not see the new software version, and instead see "ReloadNeeded", proceed to Step 7.
7) A manual reload of the device is also required. You will need to login to the device via CLI, and issue a reload command. The device can take a while to reload, keep an eye on its status from the Devices dashboard.
8) If following a reload, the device appears in the Devices dashboard with a Management Status of "Offline", you can log in directly to the WAN gateway and run the following commands:

e.g. to fix vmct04-wan-gw2-ISR-WAAS

Perform a show inventory command to check which slot has the NIM-SSD that is running ISR-WAAS installed:

vmct04-wan-gw2#sh inv

..

NAME: "NIM subslot 0/3", DESCR: "NIM SSD Module" PID: NIM-SSD , VID: V01, SN: FOC19057CW1

Next, you will need to reload this subslot and keep a close eye on the Devices dashboard to see if it brings the device back online. Again, this may take some time.

vmct04-wan-gw2#hw-module subslot 0/3 reload

If this also fails, then you can deactivate and re-activate the ISR-WAAS, using the following method:

vmct04-wan-gw2#sh virtual-service list Virtual Service List:


Name Status Package Name


vmct04_wan_gw2_WAAS Activated ISR-WAAS-5.5.3.59.ova


vmct04-wan-gw2#conf t vmct04-wan-gw2(config)#virtual-service vmct04_wan_gw2_WAAS vmct04-wan-gw2(config-virt-serv)#no activate

Exit from configuration mode, and after a few seconds you will see:

vmct04-wan-gw2#sh virtual-service list Virtual Service List:


Name Status Package Name


vmct04_wan_gw2_WAAS Deactivated ISR-WAAS-5.5.3.59.ova

Next, re-activate the service:

vmct04-wan-gw2#conf t vmct04-wan-gw2(config)#virtual-service vmct04_wan_gw2_WAAS vmct04-wan-gw2(config-virt-serv)#activate

Check again that the service has been re-activated:

vmct04-wan-gw2#sh virtual-service list Virtual Service List:


Name Status Package Name


vmct04_wan_gw2_WAAS Activated ISR-WAAS-5.5.3.59.ova

After a short while you should be able to see from the Central Manager whether this process has been successful.

If you are still unable to bring the ISR-WAAS online from following the above steps you will likely need to raise a TAC / LIT case.


How to re-install ISR-WAAS appliance

Refer to page 15 of WAN Opt Design Guide.

How to fix WAAS hosts "Offline" in Central Manager

How to fix devices where interception is disabled

How to fix devices where service-nodes are configured incorrectly

How to fix "service-insertion service-context waas/1- not enabled"

Fix: Log onto site WAN gateway and do show run | sec service-insertion and check whether "enable" is configured under "service-insertion service-context waas/1"


Automation

Feature Description Contact Expected Completion Complete?
Integrate WAAS with Cisco Prime Add description here Add contact Add date
Traffic packet analysis using Prime NAM Add description here Add contact Add date
Syslog dashboards using Splunk Add description here Add contact Add date
Integrate WAAS 6.3 with APIC-EM Add description here Add contact Add date
Management of Core WAAS WCCP ACL using ACI Add description here Add contact Add date
Automate: Discover and fix WAAS hosts down Add description here Add contact Add date
Automate: Discover and fix devices where interception is disabled Add description here Add contact Add date
Automate: Discover and fix incorrectly configured service nodes Add description here Add contact Add date
Automate: Discover and fix "service-insertion service-context waas/1- not enabled" Add description here Add contact Add date


Progress on Automation Features

Integrate WAAS with Cisco Prime

Function Description Contact Expected Completion Complete?
Compute Application Response Time using Netflow on Cisco Prime


Traffic packet analysis using Prime NAM

Function
TBC


Syslog dashboards using Splunk

Function
TBC


Integrate WAAS 6.3 with APIC-EM

Function Description Contact Expected Completion Complete?
Test WAAS Appnav-XE and WAAS WCCP ACL in APIC-EM on lab kit


Management of Core WAAS WCCP ACL using ACI

Function
TBC


Automate: Discover and fix WAAS hosts down

Function
TBC


Automate: Discover and fix devices where interception is disabled

Function
TBC


Automate: Discover and fix incorrectly configured service nodes

Function
TBC


Automate: Discover and fix "service-insertion service-context waas/1- not enabled"

Function
TBC

Rating: 0.0/5 (0 votes cast)

Personal tools