Crash dump anaylsis

Here is how to analyze a kernel crash dump in CentOs

First, to install vmlinux with debugging symbols

strings /var/crash/127.0.0.1-2013-06-22-18\:15\:01/vmcore | less

Look for the kernel version. In this case:

OSRELEASE=2.6.32-220.el6.x86_64

Go to

http://debuginfo.centos.org/6/x86_64/ and download
kernel-debuginfo-2.6.32-220.el6.x86_64.rpm  and
kernel-debuginfo-common-x86_64-2.6.32-220.el6.x86_64.rpm

Install on target machine with

rpm -ivh kernel-debuginfo-common-x86_64-2.6.32-220.el6.x86_64.rpm
rpm -ivh kernel-debuginfo-2.6.32-220.el6.x86_64.rpm

Finally, the crash dump analysis:

  crash /usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64/vmlinux ./vmcore

Use the “bt” command to pull up a backtrace. It will tell you what program was running and what happened to cause the crash.