DTrace to the rescue!
We recently migrated data to some older servers and ran into a performance problem during peakhours.   Analysis    On our busiest days, we've seen the load go up to over 300. Luckily this is a SPARC system, and it was still somewhat responsive. Not bad for an old 2 Socket system...   Looking at the CPU utilization using vmstat, showed a high percentage of time spent in SYS.    You can see this in the two performance graphs (the graphs were made on a Sunday, with a lower load).                    After looking at the processes with prstat -Lm  (microstate accounting) and seeing the high SYS-percentage, we decided to look at what the kernel is doing.   Solaris has a nifty tool called lockstat. This is what it revealed:   # lockstat -i 133 -I sleep 5  Profiling interrupt: 1394 events in 5.241 seconds (266 events/sec) Count indv cuml rcnt     nsec CPU+PIL                Caller   ...