How To Collect Core Files - ASR9K

From DocWiki

Jump to: navigation, search

This process details how to collect core files from an ASR9K router as requested by Development.

  1. Disable sparse core dumps
  2. Ensure sufficient disk space for the dump
  3. Dump the process core from the appropriate node
  4. Find the dumped core files
  5. Upload the files to the case


Contents

Disable sparse core dumps

Config

Sparse cores are faster, but do not contain the complete memory space. Often, for large processes like BGP, complete core files are requested. Disable sparse core dumps using the following configuration commands.

RP/0/RSP0/CPU0:ASR9000# configure
RP/0/RSP0/CPU0:ASR9009(config)# exception sparse off
RP/0/RSP0/CPU0:ASR9000(config)# commit

Verification

RP/0/RSP0/CPU0:ASR9000# show exception
 Exception path for choice 1 is not configured or removed 
 Exception path for choice 2 is not configured or removed 
 Exception path for choice 3 is not configured or removed 
 Choice fallback one path = dumper_harddisk:/dumper compress = on filename = <process_name>
 Choice fallback two path = dumper_disk1a:/dumper compress = on filename = <process_name>
 Choice fallback three path = dumper_disk0a:/dumper compress = on filename = <process_name>
 Kernel dump not configured 
 Tftp route for kernel core dump not configured 
 Dumper packet memory in core dump enabled
 Sparse core dump disabled
 No config for sprsize tuple 
 No config for coresize tuple 
 No config for memory-threshold tuple 
 No config for core-verification tuple
RP/0/RSP0/CPU0:ASR9000# show run formal | i sparse
Building configuration...
exception sparse off

Ensure sufficient disk space

Sometimes sparse cores (or no cores) are created when harddisk: is full. Ensure that the disk has enough space for the dump.

RP/0/RSP0/CPU0:ASR9000# show filesystem 
File Systems:

     Size(b)     Free(b)        Type  Flags  Prefixes
           -           -     network     rw  qsm/dev/fs/tftp:
           -           -     network     rw  qsm/dev/fs/rcp:
           -           -     network     rw  qsm/dev/fs/ftp:
  1027604480     2597888  dumper-lnk     rw  qsm/dumper_disk0a:
  3082813440           0  dumper-lnk     rw  qsm/dumper_harddisk:
      224256      165888  dumper-lnk     rw  qsm/dumper_nvram:
  4110401536  2730731008  dumper-lnk     rw  qsm/dumper_disk0:
    28966912    28943732  dumper-lnk     rw  qsm/dumper_configflash:
    44695552    24136420  dumper-lnk     rw  qsm/dumper_bootflash:
  4110401536  2730731008  flash-disk     rw  disk0:
  3082813440           0    harddisk     rw  harddisk:
  1027604480     2597888  flash-disk     rw  disk0a:
    44695552    24136420       flash     rw  bootflash:
    28966912    28943732       flash     rw  configflash:
      224256      165888       nvram     rw  nvram:
      

In the above case, there seems to be no free space in harddisk, and this is due to a process crashing frequently, creating core dumps as and when it crashes.

RP/0/RSP0/CPU0:ASR9000# cd harddisk:/dumper
RP/0/RSP0/CPU0:ASR9000# dir

Directory of harddisk:/dumper

11904       -rw-  12541       Mon Oct 13 08:32:17 2014  first.mibd_interface_330.by.dumper_gen.20141013-083152.node0_RSP0_CPU0.ppc.txt
11902       -rw-  175647      Mon Oct 13 08:32:17 2014  first.mibd_interface_330.by.dumper_gen.20141013-083152.node0_RSP0_CPU0.ppc.cpu_info.Z
11870       -rw-  9349        Tue Jan  6 00:07:07 2015  first.ipv4_ma_279.by.dumper_gen.node0_RSP0_CPU0.ppc.txt
11876       -rw-  140743      Tue Jan  6 00:07:07 2015  first.ipv4_ma_279.by.dumper_gen.node0_RSP0_CPU0.ppc.cpu_info.Z
686648      -rw-  6785        Tue Apr 26 19:48:16 2016  zl30160_log
42996619    -rw-  14201       Thu Sep 24 16:00:12 2015  first.pim_1169.by.dumper_gen.sparse.20150924-160006.node0_RSP0_CPU0.ppc.txt
42996620    -rw-  149646      Thu Sep 24 16:00:13 2015  first.pim_1169.by.dumper_gen.sparse.20150924-160006.node0_RSP0_CPU0.ppc.cpu_info.Z
42996624    -rw-  3656        Thu Oct  8 16:59:41 2015  first.aipc_proxy_115.by.sysmgr.sparse.20151008-165934.node0_RSP0_CPU0.ppc.txt
42996626    -rw-  301064      Thu Oct  8 16:59:42 2015  first.aipc_proxy_115.by.sysmgr.sparse.20151008-165934.node0_RSP0_CPU0.ppc.cpu_info.Z
42996625    -rw-  2798830     Mon Oct 19 02:00:14 2015  aipc_proxy_115.20151019-014633.node0_RSP0_CPU0.ppc.Z
42996627    -rw-  3799        Thu Oct  8 17:01:08 2015  aipc_proxy_115.by.sysmgr.sparse.20151008-170101.node0_RSP0_CPU0.ppc.txt
44385132    -rw-  2792303     Sun Oct 18 12:52:20 2015  aipc_proxy_115.20151018-124752.node0_RSP0_CPU0.ppc.Z
42996629    -rw-  3652        Thu Oct  8 17:10:30 2015  aipc_proxy_115.by.sysmgr.sparse.20151008-171025.node0_RSP0_CPU0.ppc.txt
42996688    -rw-  2791451     Sun Oct 18 07:49:42 2015  aipc_proxy_115.20151018-073751.node0_RSP0_CPU0.ppc.Z
42996636    -rw-  12163       Thu Oct  8 20:29:01 2015  l2vpn_mgr_1171.by.l2vpn_mgr.sparse.20151008-202849.node0_RSP0_CPU0.ppc.txt
42996637    -rw-  2793431     Sun Oct 18 20:27:38 2015  aipc_proxy_115.20151018-201146.node0_RSP0_CPU0.ppc.Z
42996634    -rw-  3889        Fri Oct  9 07:24:12 2015  aipc_proxy_115.sparse.20151009-072410.node0_RSP0_CPU0.ppc.txt
42996633    -rw-  3727        Fri Oct  9 07:24:20 2015  aipc_proxy_115.sparse.20151009-072413.node0_RSP0_CPU0.ppc.txt
47300599    -rw-  2793179     Sun Oct 18 10:25:29 2015  aipc_proxy_115.20151018-102059.node0_RSP0_CPU0.ppc.Z
42996639    -rw-  3727        Fri Oct  9 07:24:24 2015  aipc_proxy_115.sparse.20151009-072417.node0_RSP0_CPU0.ppc.txt
45311496    -rw-  2793347     Sun Oct 18 15:23:48 2015  aipc_proxy_115.20151018-151846.node0_RSP0_CPU0.ppc.Z
42996641    -rw-  3727        Fri Oct  9 07:24:30 2015  aipc_proxy_115.sparse.20151009-072425.node0_RSP0_CPU0.ppc.txt
45630540    -rw-  2798758     Sun Oct 18 23:39:21 2015  aipc_proxy_115.20151018-233252.node0_RSP0_CPU0.ppc.Z
42996643    -rw-  3727        Fri Oct  9 07:24:46 2015  aipc_proxy_115.sparse.20151009-072441.node0_RSP0_CPU0.ppc.txt
42996762    -rw-  2799430     Mon Oct 19 05:21:49 2015  aipc_proxy_115.20151019-042816.node0_RSP0_CPU0.ppc.Z
42996645    -rw-  3727        Fri Oct  9 07:27:10 2015  aipc_proxy_115.sparse.20151009-072705.node0_RSP0_CPU0.ppc.txt
42996646    -rw-  2792331     Sun Oct 18 06:27:41 2015  aipc_proxy_115.20151018-062343.node0_RSP0_CPU0.ppc.Z
42996647    -rw-  3727        Fri Oct  9 07:30:16 2015  aipc_proxy_115.sparse.20151009-073011.node0_RSP0_CPU0.ppc.txt

<snip>

To make space, delete unwanted core files or other files from harddisk:/dumper. In this example I'm deleting all the crash-related files associated with apic_proxy.

RP/0/RSP0/CPU0:ASR9001-B# run

Check the directory

# pwd
/harddisk:/dumper

Delete the files

# rm *aipc_proxy_*

Exit

# exit

Dump the Core

Doing a running coredump does not cause impact. Make sure to dump from the appropriate node where the issue is seen.

This core took a few minutes to complete. This example is BGP, but any running process on RSPs or linecards can be dumped.

Example. This dumps the BGP core on RSP0:

RP/0/RSP0/CPU0:ASR9000# dumpcore running bgp location 0/RSP0/CPU0

Find the Core

This command shows where the core files are stored.

Make sure the cores collected are NOT sparse. The core file name should not contain the word 'sparse'.

RP/0/RSP0/CPU0:ASR9000# show exception
 Exception path for choice 1 is not configured or removed 
 Exception path for choice 2 is not configured or removed 
 Exception path for choice 3 is not configured or removed 
 Choice fallback one path = dumper_harddisk:/dumper compress = on filename = <process_name>
 Choice fallback two path = dumper_disk1a:/dumper compress = on filename = <process_name>
 Choice fallback three path = dumper_disk0a:/dumper compress = on filename = <process_name>
 Kernel dump not configured 
 Tftp route for kernel core dump not configured 
 Dumper packet memory in core dump enabled
 Sparse core dump disabled
 No config for sprsize tuple 
 No config for coresize tuple 
 No config for memory-threshold tuple 
 No config for core-verification tuple

The above highlighted section shows that the cores are stored in harddisk:/dumper

RP/0/RSP0/CPU0:ASR9001-B# dir harddisk:/dumper | i bgp

11966       -rw-  41639739    Wed Apr 16 14:19:59 2014  first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.Z
9348106     -rw-  18424       Wed Apr 16 14:20:01 2014  first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt
9348135     -rw-  151247      Wed Apr 16 14:20:01 2014  first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.cpu_info.Z

Copy the core files

FTP - Prompt for Password and Username

  • FTP server
    • IP: 10.0.0.1
    • Username: lab
    • Password: lab
RP/0/RSP0/CPU0:ASR9000# copy harddisk:/dumper/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.Z ftp://10.0.0.1

Destination username: [anonymous]?lab
Destination password: 
Destination filename [first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.Z]?
Writing ftp://lab:*@10.0.0.1/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.Z
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
41639739 bytes copied in     11 sec (  3536583)bytes/sec

RP/0/RSP0/CPU0:ASR9000# copy harddisk:/dumper/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt ftp://10.0.0.1

Destination username: [anonymous]?lab
Destination password: 
Destination filename [first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt]?
Writing ftp://lab:*@10.0.0.1/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt
C
18424 bytes copied in      0 sec 

RP/0/RSP0/CPU0:ASR9000# copy harddisk:/dumper/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.cpu_info.Z ftp://10.0.0.1

Destination username: [anonymous]?lab
Destination password: 
Destination filename [first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.cpu_info.Z]?
Writing ftp://lab:*@10.0.0.1/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.cpu_info.Z
C
151247 bytes copied in      0 sec

FTP - Password and Username in Command

You can also use the format: ftp://<username>:<password>@<ip-address> to bypass the login portion of the copy command if desired:

RP/0/RSP0/CPU0:ASR9000# copy harddisk:/dumper/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt ftp://lab:lab@10.0.0.1
Wed Apr 13 10:16:38.590 UTC
Destination filename [first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt]?
Writing ftp://lab:*@10.0.0.1/first.bgp_1049.by.dumper_gen.node0_RSP0_CPU0.ppc.txt
C
18424 bytes copied in      0 sec

TAC Upload Tool

This tool can be used for large uploads.

Support Case Uploader

Rating: 5.0/5 (1 vote cast)

Personal tools