My 2nd Head

I had the situation, where I wanted to see if read/write operations take too much time. It was something that I thought could be done easily with DTrace. Unfortunately, my DTrace skills are a bit rusty, so I contacted my personal DTrace guru Javier, who gave me a script. Here is the script for read operations: slow_read.d: #!/usr/sbin/dtrace -s #pragma D option quiet #pragma D option switchrate=10hz syscall::*read:entry {         this->filistp = curthread->t_procp->p_user.u_finfo.fi_list;         this->ufentryp = (uf_entry_t *)((uint64_t)this->filistp + (uint64_t)arg0 * (uint64_t)sizeof (uf_entry_t));         this->filep = this->ufentryp->uf_file;         self->offset = this->filep->f_offset;         this->vnodep = this->filep != 0 ? this->filep->f_vnode : 0; ...

We recently migrated data to some older servers and ran into a performance problem during peakhours. Analysis On our busiest days, we've seen the load go up to over 300. Luckily this is a SPARC system, and it was still somewhat responsive. Not bad for an old 2 Socket system... Looking at the CPU utilization using vmstat, showed a high percentage of time spent in SYS. You can see this in the two performance graphs (the graphs were made on a Sunday, with a lower load). After looking at the processes with prstat -Lm (microstate accounting) and seeing the high SYS-percentage, we decided to look at what the kernel is doing. Solaris has a nifty tool called lockstat. This is what it revealed: # lockstat -i 133 -I sleep 5 Profiling interrupt: 1394 events in 5.241 seconds (266 events/sec) Count indv cuml rcnt     nsec CPU+PIL                Caller   ...