DTrace to the rescue!

We recently migrated data to some older servers and ran into a performance problem during peakhours. Analysis On our busiest days, we've seen the load go up to over 300. Luckily this is a SPARC system, and it was still somewhat responsive. Not bad for an old 2 Socket system... Looking at the CPU utilization using vmstat, showed a high percentage of time spent in SYS. You can see this in the two performance graphs (the graphs were made on a Sunday, with a lower load). After looking at the processes with prstat -Lm (microstate accounting) and seeing the high SYS-percentage, we decided to look at what the kernel is doing. Solaris has a nifty tool called lockstat. This is what it revealed: # lockstat -i 133 -I sleep 5 Profiling interrupt: 1394 events in 5.241 seconds (266 events/sec) Count indv cuml rcnt nsec CPU+PIL Caller ...