Minidump Checklist

From DocWiki

What is Minidump?

•Mini-Dump is a user mode post mortem debugging technology that allows engineers to 'replay' exceptions that occurred on a customer system as if they were running locally. •With just a MiniDump file (.mdmp) and a properly configured symbol store and source code, a post-mortem can be replayed. All of the relevant binaries, associated symbols and source code will be loaded on the fly.

Who has introduced Minidump Technology ?

•Mini-Dump technology was first introduced by Microsoft circa Windows 2000.

Which Release of ICM used Minidump Technology ?

•7.0 will be first release to leverage this technology.

Which was Technology used before the use of Minidump?

•Dr. Watson.

What is Dr.Watson ?

http://cisco.com/en/US/partner/products/sw/custcosw/ps1001/products_tech_note09186a0080153ffa.shtml

Why did Cisco Embrace Minidump Technology from ICM 7.0 rather than Dr.Watson ?

•Dr. Watson is limited and very old technology. •Dr. Watson cannot decode the new Visual Studio .NET 2003 symbol files. •Microsoft dropped support for embedding COFF symbols in binaries after Visual Studio 6.0. •MiniDump is fully configurable by the application as to the amount of information captured in a dump. •Custom user-defined streams can be added to a MiniDump (not yet implemented in ICM). •File management is defined by the application. •Dr. Watson can create CrashDumps (.dmp), that also can be used similarly to a MiniDump, but it is has limitations.

How is Minidump Configured ?

Checklists to be followed by TAC before escalating the case to CCBU for Minidump analysis.

1.TAC need to provide callflow and versions of ICM with SR/ES installed

   ( if any ).

2.TAC needs to collect Logs of Matching Time stamps at the time

   Minidump was generated.

3.TAC need to Collect the following files :- a) If router is crashing and generating a Mini Dump, Collect the Following files :- •Router Mini Dump files (.mdmp files ) •router.exe ( under icm\bin ) •router.pdb ( under icm\bin )

b) If Logger process is crashing and generating a Mini Dump, Collect the Following files :-

•Logger Minidump files (.mdmp files ) •.exe file ( under icm\bin ) •.PDB file (under icm\bin )

c) If CTIOS Server is crashing and generating a Mini Dump, Collect the Following files :-

•CTIOS Minidump files (.mdmp files ) •ctiosservernode.exe ( under icm\ctios_bin ) •ctiosservernode.pdb ( under icm\ctios_bin )

d) If OPC Process is crashing and generating a Mini Dump, Collect the Following files :-

•OPC Minidump files (.mdmp files ) •opc.exe ( under icm\bin ) •opc.pdb ( under icm\bin )

e) If PIM process is crashing and generating a Mini Dump, Collect the Following files :-

•PIM Minidump files •.exe file ( under icm\bin depending on type of PIM ) •.PDB file (under icm\bin depending on type of PIM )

f) If NIC process is crashing and generating a Mini Dump, Collect the Following files :-

•NIC Minidump files •.exe file ( under icm\bin ) •.PDB file (under icm\bin )

3) Here is a Sample example of how to search for known cases/defects when a process crashes generating a Minidump.

    Logs show the following:
    15:09:32:132 pg12A-opc Trace: ConfirmInvocation - Fail to find the
    Invocation Object by InvokeID 0xf000136b
    15:09:32:163 pg12A-opc Trace: ConfirmInvocation - Fail to find the
    Invocation Object by InvokeID 0xf000136c
    15:09:32:163 pg12A-opc Trace: ConfirmInvocation - Fail to find the
    Invocation Object by InvokeID 0xf000136d
    15:09:32:163 pg12A-opc Trace: ConfirmInvocation - Fail to find theInvocation Object by InvokeID 0xf000136e
    15:09:32:382 pg12A-opc Trace: CExceptionHandlerEx::GenerateMiniDump - A
    Mini Dump File is available at logfiles\opc.exe_20070726150932351.mdmp
    15:09:32:663 pg12A-opc Unhandled Exception: Exception code: C0000005
    ACCESS_VIOLATION
     Fault address:  004E9FA1 01:000E8FA1 C:\icm\bin\opc.exe
     Registers:
    EAX:00000000
    EBX:00000000
    ECX:01EC8008
    EDX:01ED1C38
    ESI:0000002C
    EDI:0472CD98
    CS:EIP:001B:004E9FA1
    SS:ESP:0023:0012E454  EBP:0068B594
    DS:0023  ES:0023  FS:003B  GS:0000
    Flags:00010202
     Call stack:
    Address   Frame
    004E9FA1  0012E45C  Peripheral::HashInActiveCall+41004EBABB  0012E46C  Peripheral::AddActiveCall+3B
    00452139  0012E620  Call::ConnectionCleared+21D9
    0044FFFD  0012E7CC  Call::ConnectionCleared+9D
    00466865  0012FDF4  CSTAConnectionClearedEvent+315
    004605EE  0012FE04  CallEventControl::Initialize+46E
    0050485D  0012FE20
    CNFGMSGGetConfigPermissionInd::SetConfigSequenceNumber+AD
    00504F4F  0012FE3C  ProcessPIMMsgs+17F
    00552E44  0012FE94  MDSGetHandle+154
    00555EFD  0012FEAC  MDSStartClient+2AD
    00491F0C  0012FEB0  OPCInput::MDSInput+FC
    00491F55  0012FEC8  OPCInput::ProcessInput+35
    004924DF  0012FEF4  OPCInput::StartInput+1BF
    004A30C2  0012FF24  OPCMain::Execute+4A2
    0054430A  0012FF38  Reader::SingleCommand+CA
    004A2A8E  0012FF60  main+8E
    005C823B  0012FFC0  mainCRTStartup+143
    77E523E5  0012FFF0  IsProcessorFeaturePresent+9E.
    * *

Tips for Troubleshooting :-

1.Do a Topic search (topic.cisco.com )on the following keywords : "OPC assert Peripheral::HashInActiveCall" 2.Follow the resolution summary if a matching case/defect is found. Else escalate to CCBU.