Wednesday, October 12, 2011

Splunking Oracle's ZFS Appliance Part II

In my first part I wrote about storing long term analytics data in Splunk.

Wouldn't it be nice to also have storage capacity tracked with Splunk?

This is how it's done:

1. Get pool properties


#!/bin/ksh


# list capacity all Pools in a System to ${outputdir}/${poolname}.pools.log


# Example: listPools.ksh /tmp 10.16.5.14


typeset outputdir=$1
typeset ipname=$2
typeset debug=$3
typeset user=monitor


if [ -z "$1" -o -z "$2" ]; then
  printf "\nUsage: $0 <output Dir≷ <ZFSSA ipname> [ debug ]\n\n"
  exit 1
fi


mkdir -p ${outputdir}
dat=$(date +'%y-%m-%d %H:%M:%S')


ssh -T ${user}@${ipname} << --EOF-- > ${outputdir}/${ipname}.pools.log
script
run('status');
run('storage');
var poollist=list();


printf("Time,pool,avail,compression,used,space_percentage\\n");


for(var k=0; k&lt:poollist.length; k++) {
  run('select ' + poollist[k]);


  var space_used=get('used')/1024/1024/1024;
  var space_avail=get('avail')/1024/1024/1024;
  var compression=get('compression');
  var space_percentage=space_used/(space_used + space_avail)*100;


  printf("$dat,%s,%0.0f,%0.2f,%0.0f,%0.0f\\n",poollist[k],space_avail,compression,space_used,space_percentage);


  run('done');
}
run('done');


--EOF--


exit $?

2. Write Splunk props.conf

As an exercise...


3. Enjoy Splunk Dashboards:

















4. Repeat for projects and shares.































PS: If I find the time, I will eventually package this into a Splunk App.

Friday, October 7, 2011

Splunking Oracle's ZFS Appliance

We have a bunch of Oracle ZFS Appliances. What I really like is their integrated dtrace based analytics feature.

However, some things are missing or causing problems:

-Storing long-term analytics data on the appliances produces a lot of data on the internal disks. This can fill up your appliance and in the worst case slow down the appliance software

-Scaling the timeline out too much, makes peaks invisible. This is probably a problem of the rendering software used on the appliance (JavaScript)

-Comparing all our appliances is not possible. There is no central analytics console.

As we are a heavy Splunk user, I sat together with our friendly storage consultant from Oracle and we brought these two great products closer together:

This is how we did it:

1. Setting up analytics worksheets

First we had to create the analytics worksheets. This is best done using the CLI interface, as the order of drilldowns should be always the same. Otherwise fields in the generated csv file might be in a different order on every appliance. Doing this in the BUI is possible, but hard...

I would also recommend to store the worksheet under a separate Appliance User.

Sample CLI commands:

analytics
worksheets
create Monitor
select worksheet-???


dataset
set name=io.ops[op]
set drilldowns=read,write
set seconds=3600
commit


dataset
set name=nfs4.ops
set seconds=3600
commit
...


2. Fetch Analytics Data

Script Excerpt:

ssh -T ${user}@${ipname} << --EOF--
${outputdir}/${wsname}.${ipname}.out
script
run('analytics');
run('worksheets');
var ws=list();
printf("Worksheets:%d\\n",ws.length);
printf("%s\\n",ws);
for(var i=0; i<ws.length; i++)="" p="" {<=""></ws.length;>
  run('select ' + ws[i]);
  var wsname=get('name');
  printf("Worksheet Name:%s\\n",wsname);
  if ( wsname == "$wsname" ) {
    var ds=list();
    for(var j=0; j<ds.length; j++)="" p="" {<=""></ds.length;>
      run('select ' + ds[j]);
      var dsname=get('name');
      printf("zfssa_%s\\n",dsname);
      dump(run('csv'));
      run('done');
    }
  }
  run('done');
}
run('done');


--EOF--

3. Configure Splunk Inputs

4. Create Splunk Dashboards

5. Enjoy Analytics Data Under Splunk













Happy Spelunking...