DocWiki - User contributions [en]

How to analyze the core file

2010-09-24T06:16:17Z

Sdandu: New page: {| border="1" |- ! '''Problem Summary''' | How to analyze the core file? |- ! '''Error Message''' | N/A. |- ! '''Where to find''' | Refer to How to check if a process crashed for the...

{| border="1"
|-
! '''Problem Summary'''
| How to analyze the core file?
|-
! '''Error Message'''
| N/A.
|-
! '''Where to find'''
| Refer to [[ How to check if a process crashed ]] for the location of the core dumps and how to collect them.
|-
! '''Recommended Action'''
|
* Copy the dump to a machine running similar CCX version.
* run 'gdb -c <core file path> <exe path>' (exe path is the path of the service executable)
** Executable name would be part of the dump file name (core.19841.6.'''UCCX_Engine'''.1267009557)
** Executable file path of a few CCX services
*** Engine : /opt/cisco/uccx/bin/UCCX_Engine
*** CVD : /opt/cisco/uccx/bin/UCCX_Cvd
*** etc
* Output of gdb would be something similar
gdb -c /var/log/active/core/core.19841.6.UCCX_Engine.1267009557 /opt/cisco/uccx/bin/UCCX_Engine
...
...
Loaded symbols for /usr/local/thirdparty/java/jdk1.6.0_17/jre/lib/i386/libnio.so
Reading symbols from /opt/cisco/uccx/lib/libdriverManager.so...done.
Loaded symbols for /opt/cisco/uccx/lib/libdriverManager.so
#0 0x008037a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb)
* run backtrace 'bt' command for the full stack trace
(gdb) bt
#0 0x008037a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x0025b825 in raise () from /lib/tls/libc.so.6
#2 0x0025d289 in abort () from /lib/tls/libc.so.6
#3 0x0102fb9f in os::abort () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#4 0x01152b81 in VMError::report_and_die () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#5 0x01153771 in crash_handler () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#6 <signal handler called>
#7 0x01009d42 in methodOopDesc::name_and_sig_as_C_string () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#8 0x00dec813 in frame::print_on_error () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#9 0x01152107 in VMError::report () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#10 0x01152aba in VMError::report_and_die () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#11 0x010363ac in JVM_handle_linux_signal () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#12 0x01032624 in signalHandler () from /usr/local/thirdparty/java/j2sdk/jre/lib/i386/server/libjvm.so
#13 <signal handler called>
#14 0x05a368e1 in '''Java_com_cisco_alarmutil_GenericAlarmJNI_sendAlarmwithDataType'''
(env=0x74732c6d, obj=0x3d657461, message=0x54554853, messageParams=0x4e574f44,
_catalogID=0x32202c5d) at /usr/lib/gcc/i386-redhat-linux/3.4.6/include/jni.h:1384
* Report to DE and provide the stack trace and the core dump file.
* DE would debug the problematic code reported in the stack trace.

|-
! '''Release'''
| Release 8.0(1)
|-
! '''Associated CDETS #'''
| NA
|-
|}

How to check if a process crashed

2010-09-24T06:15:17Z

Sdandu: New page: {| border="1" |- ! '''Problem Summary''' | How to check if a process crashed? |- ! '''Error Message''' | 53: uccx-ha-n01.localdomain: Feb 09 2010 13:03:04.809 UTC : %UC_SERVICEMANAGER-2-Se...

{| border="1"
|-
! '''Problem Summary'''
| How to check if a process crashed?
|-
! '''Error Message'''
| 53: uccx-ha-n01.localdomain: Feb 09 2010 13:03:04.809 UTC : %UC_SERVICEMANAGER-2-ServiceFailed: %[ServiceName=Cisco Unified CCX Engine][Reason=Service stopped abruptly][ClusterID=][NodeID=uccx-ha-n01]: Service terminated.
|-
! '''Where to find'''
| Look for the above Service Manager alarm
* Login into RTMT -> SysLog Viewer -> Select a Node -> Application Logs -> CiscoSyslog -> Filter -> App ID = Cisco Service Manager -> UC_SERVICEMANAGER-2-ServiceFailed
|-
! '''Recommended Action'''
| Check if a core file exists
* Login into RTMT -> Trace & Log Central -> Remote Browse -> Crash Dumps -> Service Name -> Finish
* Browse the nodes and check for any core files. If there is a core file, download and debug it. Refer to [[ How to analyze the core file ]].
|-
! '''Release'''
| Release 8.0(1)
|-
! '''Associated CDETS #'''
| NA
|-
|}

RCA for Engine failover

2010-09-24T06:14:29Z

Sdandu: New page: Abrupt Engine mastership failover to the HA node can happen for multiple reasons * '''Engine process crashed''' ** How to find it? Refer to How to check if a process crashed ** How t...

Abrupt Engine mastership failover to the HA node can happen for multiple reasons
* '''Engine process crashed'''
** How to find it? Refer to [[ How to check if a process crashed ]]
** How to analyze the core file? Refer to [[ How to analyze the core file ]]

* '''CVD process crashed'''. Engine service is dependent upon CVD service. So if CVD crashes/restarted, Engine too gets restarted, thereby causing mastership failover to the other node.

* '''Engine ran into OutOfMemory'''
** Check the MIVR logs for the OOM reason.
*** ''java.lang.OutOfMemoryError: GC overhead limit exceeded''
** Debug it based upon the OOM reason. Refer to [[ How to debug OutOfMemoryError ]]

* '''Nodes went into island mode (multiple masters) and recovered'''. Upon recovery publisher node retains mastership.
** Check the MCVD logs for the failover logs.
* '''Application error happened''', and Engine decided to shutdown
** Look for ''com.cisco.wfapi.WFKeepAliveException: KeepAliveException in ManagerManagerImpl'' in MIVR logs.

How to debug the root cause for high CPU usage

2010-09-24T06:10:41Z

Sdandu: New page: Unified CCX is not a CPU intensive application. Apart from a few background threads, most of the processing in the CCX processes happen only during the execution of an application workflow...

Unified CCX is not a CPU intensive application. Apart from a few background threads, most of the processing in the CCX processes happen only during the execution of an application workflow (call). Then why is the CPU usage of a CCX service high?

'''Probable Reasons'''
* High call load.
* Application thread looping continuously.
* Application thread busy in processing some spurious events.

'''Where to Check?'''
* Login into RTMT and browse 'Server -> Process -> Sort by %CPU'.
* Find the culprit process.

'''How to find the root cause?'''
* '''Find the process/processes that has high %CPU.'''
** Through RTMT
*** Server -> Process -> Sort by %CPU.
*** Find the processId (PID) of the top most entries.
** Through CLI
*** Run 'show process using-most cpu'.
*** Find the processId (PID) of the processes listed.
PCPU PID CPU NICE STATE CPUTIME ARGS
4.1 24087 - 0 S 05:47:41 LRMServer -s CadSplkStd -l CADLRMServer
'''74.8''' '''23884''' - 0 S 4-08:02:12 /opt/cisco/uccx/bin/UCCX_Engine /opt/cisco/uccx/conf/UCCX_EngineCfg.xml

* '''Find the thread/threads in the process that has high %CPU.'''
** Through CLI
*** Run 'show process pid <pid extracted above>'.
**** Find the entry (thread) that has high %CPU.
***** Entries are not sorted based upon %CPU. Need to figure out manually.
**** Extract the thread id (TID) of the thread.
** Through root account (remote account)
*** Run 'top -b -n1 -H -p <pid extracted above>'.
**** Find the entry (thread) that has high %CPU - listed at the top.
**** Extract the thread id (PID) of the thread.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
'''32010''' uccxuser 16 0 680m 438m 16m S '''15.4''' 5.4 814:41.05 UCCX_Engine
1305 uccxuser 16 0 680m 438m 16m S 13.5 5.4 733:38.05 UCCX_Engine
1341 uccxuser 16 0 680m 438m 16m S 13.5 5.4 623:30.13 UCCX_Engine
942 uccxuser 16 0 680m 438m 16m S 3.9 5.4 326:40.52 UCCX_Engine

** Convert the thread id (decimal) to the corresponding hexadecimal value.
*** Use decimal to hexadecimal converters.
*** Thread dumps have threads listed with hexadecimal thread ids. So this mapping is required.
* '''Generate a thread dump.'''
** If the process is a Java process (most of the UCCX processes are Java processes - UCCX_Engine, UCCX_Cvd, tomcat, etc)
*** Login into CCX Serviceability Admin. Browse 'Tools -> Performance Configuration and Logging -> Select Server and Service -> Dump Thread trace'.
*** The thread dump gets generated in JVM.log of the respective service log directory.
**** For tomcat, it's logged in catalina.out.
* '''Open the thread dump, and search for the TID (hexadecimal value) extracted above.'''
MIVR_ICD_CTI_CLIENTPOOL-72-451-client_thread_452; daemon prio=10 tid=0xa01a4000 '''nid=0x7d0a''' runnable [0x9e6c6000]
java.lang.Thread.State: RUNNABLE
* Debug the application logs and the code to find out why this thread is taking more CPU cycles.

How to debug OutOfMemoryError

2010-09-24T06:09:36Z

Sdandu: New page: JVM can crash with OOM Error for multiple reasons. * Where to check ** Check in the application logs for OutOfMemoryError Exception in thread “xxx" '''java.lang.OutOfMemoryError''':...

JVM can crash with OOM Error for multiple reasons.
* Where to check
** Check in the application logs for OutOfMemoryError
Exception in thread “xxx" '''java.lang.OutOfMemoryError''': Java heap space
* OutOfMemoryError types
** Java heap space
** PermGen space
** Out of swap space
** unable to create new native thread
** Requested array size exceeds VM limit
** GC overhead limit exceeded

* How to debug different OutOfMemory errors
** '''Java heap space'''
** '''GC overhead limit exceeded'''
*** Happens due to a memory leak.
*** Heap dumps are automatically generated at 70/80/90/98% thresholds of the heap size.
**** file name looks like heapDump_MIVR_70%Threshold.hprof.
*** When the application goes OOM, JVM generates a heap dump during the crash.
**** file name looks like java_pid2665.hprof.
**** Due to a bug CSCte90486, this file can be collected only using remote account.
*** Collect the heap dumps (present in the service log directory. i.e. MIVR directory for engine heap dumps).
*** Collect the thread dump present in JVM.log file.
*** Refer [[ How to analyze heap dumps ]] for the procedure to analyze a heap dump.

** '''PermGen space'''
*** Happens most probably due to ClassLoader leak.
*** When the application goes OOM, JVM generates a heap dump during the crash.
**** file name looks like java_pid2665.hprof.
**** Due to a bug CSCte90486, this file can be collected only using remote account.
*** Collect the heap dumps.
*** Collect the thread dump present in the JVM.log file.

** '''Out of swap space'''
*** Happens due to a memory leak in the JNI code, or in some native applications.
*** System is low on memory.
*** Collect the Perfmon counters of all the services running in the system.
*** Plot a graph of memory usage and find which process is the culprit.

** '''unable to create new native thread'''
*** Happens when OS runs out of system resources, or the application has run out of its address space due to excessive threads.
*** Collect the thread dump present in the JVM.log file.
*** Debug the perfmon logs to find the faulty application.

** '''Requested array size exceeds VM limit'''
*** Happens due to an application bug or a memory leak.
*** Debug application logs and heap dumps if present.

File:Duplicate Classes.JPG

2010-09-24T06:06:38Z

Sdandu:

File:List Object Path to GC Roots.JPG

2010-09-24T06:05:43Z

Sdandu:

File:Dominator tree show objects by class.JPG

2010-09-24T06:04:55Z

Sdandu:

File:Dominator Tree Group by class.JPG

2010-09-24T06:04:05Z

Sdandu:

File:Dominator tree.gif

2010-09-24T06:02:42Z

Sdandu:

File:Leak Suspects.JPG

2010-09-24T06:01:37Z

Sdandu:

How to analyze heap dumps

2010-09-24T05:59:15Z

Sdandu: New page: '''Eclipse Memory Analyzer''' is a very good tool for analyzing the heap dumps (.hprof files). Following is a guideline for the initial analysis of the heap dump. * How to get the Eclipse ...

'''Eclipse Memory Analyzer''' is a very good tool for analyzing the heap dumps (.hprof files). Following is a guideline for the initial analysis of the heap dump.
* How to get the Eclipse Memory Analyzer Tool (MAT)?
** Download it from http://www.eclipse.org/mat/ and install it on your desktop.
* How to open the dump?
** Start Memory Analyzer and select “Open Heap Dump” from the File menu.
*** MAT parses the heap dump and opens a few default reports.
*** The pie in the overview pane shows the distribution of retained memory on a per-object basis. It shows the biggest objects in memory (objects that have high Retained memory - memory accumulated by it and the objects that it references).
* Leak Suspects? - '''Find the memory leak'''
** If the leak is induced by a single component, it would show up in the pie graph in the overview pane itself.
*** In the following instance, historical call data was not written to DB and they were accumulated in memory, thereby leading to OOM.
*** [[Image:Leak_Suspects.JPG]]
*** It is always easy to find and fix such problems :-)
** What if the leak is not induced by a single/few objects
*** Leak suspect may not show up in the overview reports, or in the top of the dominator tree.
*** The leak could be induced by a class of objects (i.e. leak induced per call).
**** Open the dominator tree from the toolbar using the button [[Image:Dominator tree.gif]].
**** Select "Group result by..." from the toolbar and select "Group by class".
**** It is more likely to show better view than the per object view.
**** [[Image:Dominator Tree Group by class.JPG]]
**** Once a suspect class is found, list its objects and find the objects holding reference to it.
***** Select List objects –> with incoming references from the context menu. As a result we will get an object reference graph.
***** Drill down the graph to find the references.
***** If we are analyzing the references of a single class, it is better to use "Show objects by class -> with incoming references".
***** [[Image:Dominator tree show objects by class.JPG]]
***** Use "Immediate Dominators" to get the list of objects referencing it. You can exclude the packages that you are not interested in.
**** The easiest way to find out what keeps these leaking objects alive is to check the path to the garbage collection roots.
***** Select an object and select "Path to GC Roots"
***** [[Image:List Object Path to GC Roots.JPG]]
***** To find it at the class level, use the "Merge Shortest Path to GC Roots" query. The Merged Paths to GC Roots view shows the shortest paths from the GC roots to each instance of the selected class.
****** The thread that is holding the reference is shown. Drill down to find the top level object referencing our suspect class, and drill down further to find our suspect class and its immediate references.
***** Both the per object and class level path to GC roots gives us an idea of bottom-up and top-down references to the suspect class.
** '''Use OQL for any custom queries'''.
*** In one of the customer setup, most of the heap was occupied by Workflow steps. It could be either due to uploading too many scripts, or due to excessive usage of steps per workflow.
**** Following is an OQL query to list the scripts uploaded, no. of steps per script, and the memory occupied by each script.
***** ''SELECT toString(wf.script.name) AS Workflow_Script_Name, wf.steps.nSize AS Workflow_Steps_Count, wf.@retainedHeapSize AS Workflow_Mem_Occupied FROM com.cisco.wfframework.obj.WFWorkflow wf''
**** Following is an OQL query to find which scripts did the customer configure as sub-flows, and their corresponding main script.
***** ''SELECT toString(sf.subflowExpr.text) AS SubFlow_Name, toString(sf.container.script.name) AS MainFlow_Name FROM com.cisco.wfframework.steps.core.StepCallSubflow sf''
**** Similar queries can be devised by going through the object graph and using the Inspector window to look at the values of the object attributes.
** '''Check for Finalizers'''
*** Finalizable objects are reclaimed only after a few GC cycles since when the object is discovered as unreachable.
**** If GC discovers that a finalizable object is unreachable, it adds it to finalization queue, from which the finalizer thread retrieves the objects to finalize. The object gets reclaimed only during the GC later to the finalizer thread marking the object as finalized.
*** Ideally a component should implement finalize() only when it is really required. Also it should be small and should not reference large objects, otherwise the referenced objects too will not get GCed until this object is finalized.
*** Select "Java Basics -> References -> Finalizer Reference Statistics" to get an overview of which objects implemented finalization, their retained heap, objects ready for finalization, etc.
** '''PermGen issues - ClassLoader leak'''
*** Container like Tomcat uses a separate Classloader for loading each webapp. If this classloader does not get GCed when the webapp is stopped/restarted, it is a classloader leak.
*** Select "Java Basics -> Class Loader Explorer" to get an overview of the list of classloaders, the classes that they defined, and the no. of objects loaded.
**** To check the references to this Classloader, right click on the classloader and select "ClassLoader -> Path to GC Roots".
**** If the references are only held by a thread's contextClassLoader, it could be due to the improper setting of the thread's contextClassLoader.
*** Select "Java Basics -> Duplicate Classes" to get a list of classes that were loaded by multiple Classloaders.
**** [[Image:Duplicate Classes.JPG]]
**** If you notice that a class should have been loaded only once, you can identify the leaking Classloaders by checking the "Path to GC Roots" of the Classloaders that loaded the duplicate class.
* '''Performance issues'''
** Select "Java Basics -> Thread Overview" to get the list of the threads that were running at the time of dump generation, and the objects that they are referencing.
** Use "Java Collection -> *" to get a report on what is the size of each collection, fill ratio of each collection, collision ratio of the maps, objects referencing each collection, etc.
*** This will give an idea on whether the collection used is an appropriate one, whether it's initial size is properly set, whether the objects implemented proper hashcode, etc.

Checking if the current node is a VOS publisher

2010-02-08T05:37:51Z

Sdandu: New page: == Checking if the current node is a VOS publisher == {| border="1" |- ! '''Problem Summary''' | How to check if the current node is a VOS publisher? |- ! '''Error Message''' | NA |- ! ''...

== Checking if the current node is a VOS publisher ==

{| border="1"
|-
! '''Problem Summary'''
| How to check if the current node is a VOS publisher?
|-
! '''Error Message'''
| NA
|-
! '''Possible Cause'''
| NA
|-
! '''Recommended Action'''
| Use the UCCX CLI command 'utils service list'. Look for 'Primary Node =true' at the end of the command's output. On publisher node the value is true, and on subscriber it is false.
|-
! '''Release'''
| Release 8.0(1)
|-
! '''Associated CDETS #'''
| NA
|-
|}

[[Category:Unified CCX, Release 8.0]]

CAD services visible in CLI 'utils service list' command

2010-02-08T05:20:31Z

Sdandu: New page: == CAD services visible in CLI 'utils service list' command. == {| border="1" |- ! '''Problem Summary''' | CAD services are visible in the output of CLI commands though they should not b...

== CAD services visible in CLI 'utils service list' command. ==

{| border="1"
|-
! '''Problem Summary'''
| CAD services are visible in the output of CLI commands though they should not be in an IVR setup.
|-
! '''Error Message'''
| Cisco Desktop * [STOPPED] Service Not Activated
|-
! '''Possible Cause'''
| This is not an issue. The setup being IVR licensed, CAD services wont be running. Only the services that are licensed gets activated.
|-
! '''Recommended Action'''
| Ignore.
|-
! '''Release'''
| Release 8.0(1)
|-
! '''Associated CDETS #'''
| None.
|-
|}

[[Category:Unified CCX, Release 8.0]]