Core Dump Analysis with mdb/dbx

We had a Java application core dump. pstack and jstack will show all threads, but does not reveal which thread's guilty. Finding this one thread is simple:

# mdb /usr/bin/java /var/core/core_hostname_java_8081_8081_1276625241_1986
mdb: core file data for mapping at ffb80000 not saved: Bad address
Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 ld.so.1 ]
> $C
4d2fe5f8 libc.so.1`_lwp_kill+8(6, 0, 20f04, ff36932c, ff38a000, ff38abdc)
4d2fe658 libumem.so.1`umem_do_abort+0x1c(3c, 4d2fe5a8, 6, 20e40, ff376ad8, 0)
4d2fe6b8 libumem.so.1`umem_err_recoverable+0x7c(ff377b54, a, 20d38, 656ebd84, ff38d0e8, ff377b5f)
4d2fe718 libumem.so.1`process_free+0x114(59c2008, 1, 0, 3e3a1000, 1ec08, 656d3e9c)
4d2fe778 libxy_xyzclient_native.so.solaris`XYZfree+0x1b8(59c2008, 65725b48, 15b, 4, 45a, 4d2fe9f8)
4d2fe810 libxy_xyzclient_native.so.solaris`XYZ_XYZ_FreeUser+0x28(59c2008, 4d2fea64, 4d2fe9e0, ffffff80, 80000000, 0)
4d2fe880 libxy_xyzclient_native.so.solaris`Java_net_xyz_xyzserver_XYZUser_free+0x24(68204c4, 4d2fea64, 0, 4, 45a, 4d2fe9f8)
...

Or using dbx (included in Oracle Solaris Studio):

# dbx /usr/bin/java /var/core/core_hostname_java_8081_8081_1276625241_1986
 For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc
Reading java
core file header read successfully
Reading ld.so.1
Reading libumem.so.1
... (omitted) ...
Reading libpthread.so.1
Reading libcmd.so.1
Reading libaio.so.1
WARNING!!
A loadobject was found with an unexpected checksum value.
See `help core mismatch' for details, and run `proc -map'
to see what checksum values were expected and found.
dbx: warning: Some symbolic information might be incorrect.
t@null (l@79) terminated by signal ABRT (Abort)
0xff2c642c: _postfork1_child+0x00ac:    add      %i4, 212, %o1
Current function is XYZfree
dbx: warning: can't find file "/tmp/builds/XYZ-UI-8.0.031/XYZ-UI-8.0.031-source/CommonLibraries/nplexlib/unix/../src/mdebug.c"
dbx: warning: see `help finding-files'
(dbx) lwps                                                                 
  l@1 LWP suspended in door_create_server()
  l@2 LWP suspended in _postfork1_child()
... (omitted) ...
  l@77 LWP suspended in _postfork1_child()
  l@78 LWP suspended in _postfork1_child()
o>l@79 signal SIGABRT in _postfork1_child()
  l@80 LWP suspended in libc_init()
  l@81 LWP suspended in _postfork1_child()
  l@82 LWP suspended in door_create_server()
... (omitted) ...
(dbx) select l@79
(dbx) where
  [1] _postfork1_child(0x6, 0x0, 0x20f04, 0xff36932c, 0xff38a000, 0xff38abdc), at 0xff2c642c
  [2] umem_do_abort(0x3e, 0x68f7e2b8, 0x6, 0x20e40, 0xff376ad8, 0x0), at 0xff369188
  [3] umem_err_recoverable(0xff377b54, 0xa, 0x20d38, 0x652ebd84, 0xff38d0e8, 0xff377b5f), at 0xff36932c
  [4] process_free(0x7262008, 0x1, 0x0, 0x3e3a1000, 0x1ec08, 0x652d3e9c), at 0xff36b504
=>[5] XYZfree(ptr = 0x7262008, f = 0x65325b48 "../src/xy_xyz_user.c", l = 347), line 311 in "mdebug.c" 
... (omitted) ...

Now off to the SW vendor... :-)

Comments

Post a Comment

Popular posts from this blog

Heating up the Data Pipeline (Part 1)

Image
Pre-Processing Data I often hear the question from our customers, how data can be transformed prior to indexing in Splunk. Damien from  Baboonbones  has done a tremendous job in creating add-ons   providing custom inputs for Splunk. Most of his custom inputs provide the means to pre-process data by allowing custom event handlers to be written. Sometimes you still want to pre-process data that gets collected from Splunk's standard input types, like file monitors, Windows EventLogs, scripted inputs etc. Also, not everyone is capable of writing custom event handlers. A requirement these customers have, is that they have rolled out a large number of Splunk Universal Forwarders and they do not want to install another agent. To summarize, the solution capable of pre-processing data, should be easy to use, be easily integrated and be build on top of their existing architecture. How to plumb Splunk Pipelines Splunk has its own fittings to connect a Universal Forwar
Read more

Heating up the Data Pipeline (Part 3)

Image
Welcome back to the "Heating up the Data Pipeline" blog series. In part 1 we talked about how to route data from Splunk to a 3rd party system. In part 2 walked through a simple data flow that passes data collected from Splunk Forwarders through Apache NiFi back to Splunk over the HTTP Event Collector. In this part, we will look at a more complex use case, where we route events to an index, based on the sending host's classification. The classification will be looked up from a CSV file. We will make use of Apache NiFi's new Record-Oriented  data handling capabilities, which will look initially a bit more complicated, but once you grasp it, it will make further Use Cases easier and faster to build. High-Level Dataflow We will again start with our ListentTCP input, but this time we will send the data to another Processor Group. The Processor Group will again emit events suitable for Splunk HEC. Note that we have increased the Max Batch Size from 1
Read more

Heating up the Data Pipeline (Part 2)

Image
In Part 1 we went through how to route events from Splunk to a 3rd party system without losing metadata. Now I'll show you how events can be transformed using Apache NiFi and be sent back to Splunk into the  HTTP Event Collector . Note: The following is not a step-by-step documentation. To learn how to use Apache NiFi you should read the G etting Started Guide . Simple Pass-Through Flow As a first exercise we will create a simple flow, that only passes data through NiFi, without applying any complex transformations. The following picture shows a high-level NiFi flow, that receives events in our custom uncooked event format with a TCP listener, then sends the data further into a transformation "black box" (aka Processor Group), which emits events in a format, that can be ingested into a Splunk HTTP Event Collector input. Apache Nifi currently provides a rapidly growing number of processors (currently 266), which can be used for data ingestion, transfo
Read more