Showing posts from September, 2010

DTrace saving the planet again....

...maybe not the planet, but DTrace was again a big help in finding another performance problem.

During peak-hours and when we had a spam-wave coming in, we saw a spike on our T5120 systems.

Where we usually see a very low load of ~ 1-2, the load jumped up to 400 within a few minutes.

Something did'nt seem right.

Like with the previous DTrace Problem, we saw that our userland processes had a high system time, so most of the things it was doing (or maybe not) was in the kernel.

It was time to use one of the secret weapons that Solars Admins have: lockstat (uses DTrace).

Sorry for the lengthy output...

# lockstat -i 133 -I sleep 5
Profiling interrupt: 43328 events in 5.088 seconds (8516 events/sec)
Count indv cuml rcnt     nsec CPU+PIL                Caller                  -------------------------------------------------------------------------------   155   0%   0% 0.00     2347 cpu[42]                rdccr_delay+0x8           152   0%   1% 0.00     2184 cpu[53]                rdccr_…