My 2nd Head

Efficient metadata caching

2009-12-10T23:06:00.006+01:00

In my last post I was talking about 1 Million mailboxes. Each of them is a directory with several subdirectories, like Trash, Sent Items etc.

The mailbox directory itself lies 4 directories below the root node (like /a/b/c/mailbox). The hierarchy is managed by our mail-store application.

I don't know the average number of files/directory, but let's assume, each mailbox consists in average of 20 files/directories, we would currently have about 20'000'000 inodes.

Mailbox access is mostly random. We don't know when a mail is coming in, we also don't know about when a user is reading his mails. What we now from experience is, that a lot of time is spend in looking up metadata.

With mostly random access (we measured it as ~ 55% write / 45% read a while ago), and the amount of data, the chance to identify a data working set is quite low. Ok, maybe recently received emails could be part of a "working-set".

But wouldn't it be great if we could cache as much metadata as possible?

Roch Bourbonnais wrote a blog entry a while ago about inodes on zfs. This is by no means a scientific analysis, but let's take his numbers:

"23.8M files consuming 27GB of data. Basically less than 1.2K of used disk space per KB of files"

Let's say, each mail/directory uses 0.2K, and we have 20'000'000 of them, we would currently have 3.8GB of inode data. No problem to cache that. I certainly have to investigate a little bit more what kind of metadata the ARC Cache is caching.

Thanks to analytics, I can at least do a bit of sanity checking, it currently shows me that around 11G inside ARC are used for metadata caching.

If we take another view, we can see that not only do we have metadata cached, it is also heavily used. In this picture I have colored all cache hits.

Lessons learned today:

-ZFS does not waste space for inodes and therefore not cache.
-ARC is very efficient

Questions to be answered:

-What does "metadata" include?

The noise of 1'000'000 inactive mailboxes

2009-12-02T10:22:00.008+01:00

We have now migrated all inactive mailboxes (some may obviously be active again) to one 7410 Cluster.

What you can see is the IO generated by these boxes. Even if the mailboxes are abandoned they receive mails (spam, newsletters etc.)

Storage2/HDD4 and storage2/HDD8 are again the mirrored SLOG devices. As we can see here, they don't have any problems at all with the write load. If you look at all the other HDDs you see the low IOPS numbers

Looking at how many bytes per seconds are going through the disks we can see that the SLOGs are busy collecting all synchronous bits and bytes.

The slow 1TB disks get about ~700k of data per second. Looking at e.g. HDD11 we see a low number of IOPS. I would guess the average IO size is about 60-70kB. As a reference, an email is around 4k to 8k.

What this means: We get larger IOs to the disk thanks to the slog.

Thanks, mighty Logzilla :-)

SLOG Latency

2009-10-18T19:04:00.009+02:00

After reading Brendan's newest blog entry, I was curious about what kind of slog latency we can see for our data migration load.

To remind you, only synchronous writes go into the slog SSD devices.

As this is a migration running, we can see mostly NFS write operations:

In our configuration the SSD slog is mirrored (HDD4 and HDD8). Hence the same number of IOPS:

The next picture shows us the latency for our SDD SLOG Device HDD 4. We can see here that latencies start at 79 us and are mostly under 200 us. There are some outliers, but approx. 95% are under 500 us:

This matches quite well with the values Brendan blogged about (137-181 us), which includes NFSv3 latency. For reference (no picture here), we can see latencies of about 170-500 us mostly for NFSv4.

By the way. SLOG Devices are mostly one way devices, as shown here. Only if things go really bad, they are read from...

Resilvering progress...found!

2009-10-06T20:48:00.002+02:00

Found the CLI menu for resilvering progress :-)

file1:configuration storage (pool-1)> show
Pools:

POOL OWNER DATA PROFILE LOG PROFILE STATUS
pool = pool-1 file1 mirror_nspf log_mirror_nspf degraded

Properties:
pool = pool-1
status = degraded
owner = file1
profile = mirror_nspf
log_profile = log_mirror
cache_profile = cache_stripe
scrub = resilver in progress for 4h46m, 83.85% done, 0h55m to go

High IOPS for 1TB Disks

2009-10-01T21:45:00.003+02:00

Wondering how 1TB disk can have such a high IOPS.

HDD8 and HDD4 are Log-Devices (SSD)

UPDATE: For the moment I assume, these are writes to the disk cache.

Being bored....

2009-10-01T07:01:00.004+02:00

...while doing 22536 NFS ops per second, and doing gzip-2 at the same time.

HSP really works. NFS synchronous operations get "eaten" by the SSD devices (1133 IOPS).

Trekkie Stuff...

2009-09-01T20:41:00.008+02:00

Enterprise Warp Core:

Borg Cube:

PS: We all know who won in the end ;-)

Opensolaris, Huawei E220, Swisscom and Sunrise

2008-11-03T22:25:00.002+01:00

I was able to open a 3G connection to Swisscom and Sunrise using a Huawei E220 3G Modem under Opensolaris.

Here are the config files (If PIN is enabled on the SIM, add OK "AT+CPIN=????" to the chat script)

Swisscom:

/etc/ppp/chat-swisscom:


ABORT BUSY
ABORT 'NO CARRIER'
ABORT ERROR
REPORT CONNECT
TIMEOUT 120
"" "AT&F"
OK "ATZ"
OK "ATQ0 V1 E1 S0=0 &C1 &D2"
#OK "AT+CPIN=????"
OK 'AT+CGDCONT=1,"IP","gprs.swisscom.ch"'
SAY "Calling Swisscom"
OK "ATDT*99***1#"
TIMEOUT 120
CONNECT ''

/etc/ppp/peers/swisscom:


term/1
115200
connect "/usr/bin/chat -V -t15 -f /etc/ppp/chat-swisscom"
idle 7200
lock
crtscts
noipdefault
modem
user "gprs"
password "gprs"
noauth
passive
usepeerdns
defaultroute
connect-delay 6000
novj
nodetach

Sunrise:

/etc/ppp/chat-sunrise:


ABORT BUSY
ABORT 'NO CARRIER'
ABORT ERROR
REPORT CONNECT
TIMEOUT 120
"" "AT&F"
OK "ATZ"
OK 'ATS7=60'
#OK "AT+CPIN=????"
OK "AT+COPS?"
OK "AT&V"
OK "AT+CSQ"
OK 'AT+CFUN=?'
OK 'AT+CGDCONT=?'
OK 'AT+CPBR=?'
OK 'AT+CPBS=?'
OK 'AT+CGDCONT=1,"IP","Internet"'
SAY "Calling Sunrise"
OK 'ATDT*99***1#'
TIMEOUT 120
CONNECT ''

/etc/ppp/peers:


term/1
115200
connect "/usr/bin/chat -V -t15 -f /etc/ppp/chat-sunrise"
idle 7200
lock
crtscts
noipdefault
modem
user ""
password ""
noauth
passive
usepeerdns
defaultroute
connect-delay 6000
novj
nodetach

Don't forget to adjust the term/? line to reflect your /dev/term/? device.

After this, just fire up "pppd call swisscom" or "pppd call sunrise" and adjust your resolv.conf. The DNS Servers are in the output. For debugging just add -d to the pppd command.

There are a few problems with the Huawei E220, as it is recognized as a storage device first, and not as a serial device.

See the Opensolaris Forum

Most of the time the modem is recognized correctly when the Modem is plugged in before booting, and doing a reconfiguration boot with grub. ( press "e" -> append "-rv" -> -> press "b" )

There is also a patch for the Eee PC's Serial USB Port driver. See Masafumi's Blog.

Good luck!

Connecting Opensolaris to a Cisco VPN

2008-11-02T22:47:00.007+01:00

I finally got my Opensolaris box hooked up to a Cisco VPN:

1. Get the tun/tap driver from Kazuyoshi. This driver is used for creating a using between the client and the Cisco VPN router.

2. Compile and install the driver:


pkg install sunstudioexpress
export CC=/opt/SunStudioExpress/bin/cc
./configure
make
make install

3. Download the vpnc client. This program handels the vpn connection. I've used the version 0.5.1.

4. Get gmake


pkg install SUNWgmake

5. Adjust installation paths (my personal preferences):


Makefile:

PREFIX=/opt/vpnc
ETCDIR=/opt/vpnc/etc

config.c (hardcoded locations):

Line 250: return "/opt/vpnc/etc/ssl/certs";
Line 270: return "/opt/vpnc/etc/vpnc-script";
Line 275: return "/var/run/vpnc.pid";
Line 541: ... : /opt/vpnc/etc/", ...
Line 760: ... "/opt/vpnc/etc/default.conf" ...
Line 761: ... "/opt/vpnc/etc/vpnc.conf" ...

gmake
gmake install

6. Convert the pcf file into vpnc format using /opt/vpnc/pcf2vpnc. Store the file in /opt/vpnc/etc

7. Now comes the tricky part. Create the /opt/vpnc/etc/vpnc-script -script. The script opens the connection and sets up the appropriate routing.

My target was to just have a couple of connections go through the tunnel instead of all.

Attention! If you want to have a default route set into the tunnel, be sure to add some direct routes to your vpn gateway before the default route points to the tunnel. Otherwise the vpn gateway will become unreachable and the tunnel goes down. Your remote gateway can be found in your conf file ("IPSec gateway" line).

Depending on what you want to achieve, you have to configure your DNS servers in /etc/resolv.conf manually.

Here is my sample /opt/vpnc/etc/vpnc-script :


#!/bin/sh
#* reason                       -- why this script was called, one of: pre-init connect disconnect
#* VPNGATEWAY                   -- vpn gateway address (always present)
#* TUNDEV                       -- tunnel device (always present)
#* INTERNAL_IP4_ADDRESS         -- address (always present)
#* INTERNAL_IP4_NETMASK         -- netmask (often unset)
#* INTERNAL_IP4_DNS             -- list of dns serverss
#* INTERNAL_IP4_NBNS            -- list of wins servers
#* CISCO_DEF_DOMAIN             -- default domain name
#* CISCO_BANNER                 -- banner from server
#* CISCO_SPLIT_INC              -- number of networks in split-network-list
#* CISCO_SPLIT_INC_%d_ADDR      -- network address
#* CISCO_SPLIT_INC_%d_MASK      -- subnet mask (for example: 255.255.255.0)
#* CISCO_SPLIT_INC_%d_MASKLEN   -- subnet masklen (for example: 24)
#* CISCO_SPLIT_INC_%d_PROTOCOL  -- protocol (often just 0)
#* CISCO_SPLIT_INC_%d_SPORT     -- source port (often just 0)
#* CISCO_SPLIT_INC_%d_DPORT     -- destination port (often just 0)

# =========== script (variable) setup ====================================

PATH=/sbin:/usr/sbin:$PATH

FULL_SCRIPTNAME=/usr/local/sbin/vpnc
SCRIPTNAME=`basename $FULL_SCRIPTNAME`

# =========== tunnel interface handling ====================================

do_ifconfig() {

       ifconfig "$TUNDEV" inet "$INTERNAL_IP4_ADDRESS" "$INTERNAL_IP4_ADDRESS" netmask 255.255.255.255 mtu 1412 up
}

# =========== route handling ====================================

get_default_gw() {
       # isn't -n supposed to give --numeric output?
               # apperently not...
               # Get rid of lines containing IPv6 addresses (':')
               netstat -r -n | sed 's/default/0.0.0.0/' | sed 's/^.*:.*$//' | grep '^0.0.0.0' | awk '{print $2}'
}
      
do_pre_init() {
echo "do_pre_init"
}

do_connect() {
       if [ -n "$CISCO_BANNER" ]; then
               echo "Connect Banner:"
               echo "$CISCO_BANNER" | while read LINE ; do echo "|" "$LINE" ; done
               echo
       fi
      
       do_ifconfig
       if [ -n "$CISCO_SPLIT_INC" ]; then
               i=0
               while [ $i -lt $CISCO_SPLIT_INC ] ; do
                       eval NETWORK="\${CISCO_SPLIT_INC_${i}_ADDR}"
                       eval NETMASK="\${CISCO_SPLIT_INC_${i}_MASK}"
                       eval NETMASKLEN="\${CISCO_SPLIT_INC_${i}_MASKLEN}"
                       i=`expr $i + 1`
               done

       fi
      
       # Hosts 1&2 & 3
       add_host_route "xx.xx.xx.xx"
       add_host_route "xx.xx.xx.xy"
       add_host_route "xx.xx.xx.xz"

       for i in $INTERNAL_IP4_DNS ; do
               add_host_route "$i"
       done
}

do_disconnect() {
       if [ -n "$CISCO_SPLIT_INC" ]; then
               i=0
               while [ $i -lt $CISCO_SPLIT_INC ] ; do
                       eval NETWORK="\${CISCO_SPLIT_INC_${i}_ADDR}"
                       eval NETMASK="\${CISCO_SPLIT_INC_${i}_MASK}"
                       eval NETMASKLEN="\${CISCO_SPLIT_INC_${i}_MASKLEN}"
                       i=`expr $i + 1`
               done
       fi
      
       # Delete all routes again
       del_host_route "xx.xx.xx.xx"
       del_host_route "xx.xx.xx.xy"
       del_host_route "xx.xx.xx.xy"


       for i in $INTERNAL_IP4_DNS ; do
               del_host_route "$i"
       done
}

add_host_route() {
       HOST="$1"
       route add "$HOST" "$INTERNAL_IP4_ADDRESS" -interface
       }

del_host_route() {
       HOST="$1"
       route delete "$HOST" "$INTERNAL_IP4_ADDRESS" -interface
       }


#### Main

if [ -z "$reason" ]; then
       echo "this script must be called from vpnc" 1>&2
       exit 1
fi

case "$reason" in
       pre-init)
               do_pre_init
               ;;
       connect)
               do_connect
               ;;
       disconnect)
               do_disconnect
               ;;
       *)
               echo "unknown reason '$reason'. Maybe vpnc-script is out of date" 1>&2
               exit 1
               ;;
esac

exit 0

8. Now you should be able to connect to your vpn:


/opt/vpnc/sbin/vpnc myvpn.conf

9.The output should look like this:


 /usr/local/sbin/vpnc cia.conf
Enter password for tzhbomi5@bwpir.bluewin.ch: 
do_pre_init
add host xx.xx.xx.xx: gateway zz.zz.zz.zz
add host xx.xx.xx.xy: gateway zz.zz.zz.zz
add host xx.xx.xx.xz: gateway zz.zz.zz.zz
VPNC started in background (pid: 3971)...

10. The interfaces look like this, where zz.zz.zz.zz is the IP Address that got assigned from the other side of the tunnel. The tun0 device was automatically created by vpnc·


ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
 inet 127.0.0.1 netmask ff000000 
rge0: flags=201004843 mtu 1500 index 2
 inet 192.168.1.35 netmask ffffff00 broadcast 192.168.1.255
 ether 0:22:15:5e:61:2b 
tun0: flags=10010008d1 mtu 1412 index 4
 inet zz.zz.zz.zz --> zz.zz.zz.zz netmask ffffffff 
 ether 2:0:0:0:0:0

OpenSolaris 2008.05, BrandZ and Zattoo

2008-05-11T23:35:00.002+02:00

I've been using OpenSolaris 2008.05 for about a week, and despite some small bugs, I really fell in love.

Until now I did not have the time to play around with Nevada releases, but OpenSolaris 2008.05 could really be my future #1 desktop (beside my Sunray...).

Things that worked so far:
-Getting Windows inside Virtualbox (yeah there are some apps still left running only on windows..)
-CentOS in a Branded Linux Zone (experimental Linux 2.6 support)
-Skype in a Branded Linux Zone(without sound yet)
-Zattoo in a Branded Linux Zone (without sound yet)

What does not work yet:
-TUN for Cisco VPNs
-ACPI (there are some drivers...)
-Sun Secure Global Desktop Client (at least the automatic client download...)

What's left to say?

Thanks to the OpenSolaris Community for all efforts.

OpenSolaris Rocks!!!

Using wget for downloads.oracle.com

2008-04-14T09:20:00.003+02:00

Downloading from oracle.com using wget is simple.

-First go to downloads.oracle.com and select the software to be downloaded

-Log in with your account and accecpt the license

-Copy the download link to your clipboard

-Find your cookies.txt in your $HOME

-Use wget:

$ wget --load-cookies=$HOME/.mozilla/default/.../cookies.txt -c [your url]

-Now you should be able to download the software

-This should also work for other sites.

Debugging sendmail

2007-11-27T13:50:00.000+01:00

To debug outgoing mail use following command

# /usr/lib/sendmail -d -f root@hostname -ODeliveryMode=i recipient@hostname

I've got a green brain

2007-08-13T22:27:00.000+02:00

Your Brain is Green

Of all the brain types, yours has the most balance.
You are able to see all sides to most problems and are a good problem solver.
You need time to work out your thoughts, but you don't get stuck in bad thinking patterns.

You tend to spend a lot of time thinking about the future, philosophy, and relationships (both personal and intellectual).

What Color Is Your Brain?

ZFS Rocks

2007-07-22T10:23:00.000+02:00

Jeff Bonwick writes about ZFS license announcements and how it rocks.

ZFS Take-Off

2007-07-21T15:03:00.001+02:00

Robin Haris (zdnet) and Jörg (c0t0d0s0) are writing about RAID6 respectivly ZFS.

RAID6 will surely be a marketing success as most people do not know about ZFS, or are thinking that making the move from RAID5 to RAID6 will solve all their problems. Those companies who have heard about ZFS have certainly had a look at it.

Most companies are rather conservative when implementing new technology. This is not a bad thing, especially when trusting your precious data to a new filesystem.

Paradoxically, without proper checksumming like ZFS does, your data could be at higher risk, even if ZFS is a rather new technology.

While there have been some problems with ZFS, none of them have affected the on-disk data. This is certainly the result of thoroughly testing ZFS like no other filesystem (leaving real-world "testing" beside).

There are still some issues, that may prevent ZFS to be deployed in a broader area:

-Performance issues on Storage Array with stable storage (not ignoring cache flush)
-No dynamic LUN resizing (not really a ZFS issue)
-Database performance may not be at UFS DirectIO level (work is on the way)
-No long-term database performance experience available
-Booting from ZFS not yet integrated
-3rd Party support missing (e.g. Backup solutions not yet there)

If Sun is working on these technical issues, and I know they are, my guess is that ZFS will really take off in a timeframe of 2 years. Compared with the age of UFS this is a short chapter.

Again ZFS performance improvements for databases

2007-07-19T22:07:00.000+02:00

There is a new fix to improve performance for databases on ZFS.

Can't wait to see OLTP benchmarks where ZFS is being faster than UFS with DirectIO for.

Oh, and did you know there is some work going on for getting the ZIL onto seperate devices (NVRAM or Solid State Disks)

(For those who don't know what the ZIL is, look here)

Xen code drop to OpenSolaris

2007-07-19T21:41:00.000+02:00

After almost waiting a year, the Lego bricks are falling neatly into place. This means, that it is finally possible to run Windows under Solaris (where needed :-).

Splunk 3.0 with Access-Control

2007-07-18T22:37:00.000+02:00

Yeaah!!!

Just found out that Splunk 3.0 will support access control. This means e.g., that developers can debug production problems without logging in to that host. They will see only logfiles relevant to finding the problem...

This is a huge step forward, as logs often contain classified data.

Where should security happen?

2007-07-18T22:04:00.000+02:00

As close to the data as possible!

Yesterday's post from Erik reminded me about a paradigm, that came up when I implemented the SSH Tectia Suite. It's the question where to do security properly. The answer is as close to the data as possible.

The problem with today's corporate networks is, that the "enemy" is already inside. There are so many people (internal/external) connecting to a company's network, which makes firewalls almost irrelevant. This effect is called deperimeterization.

Firewalls have a false reputation, that they protect everything. But because there are so many people who have access to both sides of firewalls, this doesn't make it very secure.

What could be a new approach? The Jericho Forum (a security focused group) says: "Individual Hosts should be able to defend themselves".

I certainly agree with that. Most operating systems contain integrated firewalls waiting for activation. Many applications provide extended authentication features and encryption (e.g. TLS/SSL). Not to forget the SSH protocol for managing the operating system instead of telnet.

While in theory you could take all firewalls away, and rely on host security, in practice you wouldn't do that of course. As an analogy to real life, you would certainly lock the gate to your stately home...

Paul Murphy clarifies the Sun Ray "difference"

2007-07-17T22:20:00.000+02:00

In today's blog, Paul Murphy lists advantages of the Sun Ray technology. This as an answer to a comment of Erik Engbrecht (a regular visitor to Murph's blog).

Of course, Erik has already answered in his blog.

While Erik certainly makes some valid points, I personally believe that having the desktop processing happening in the datacenter would be for a lot of companies the best approach.

I worked for several banks, and you wouldn't believe me, the investments done or planned to make desktops/laptops secure (harddisk encryption, USB-port-locking software, reverse-firewalls, virus-scanners, security audit tools, etc.)

Most of these security activities just aren't needed with a stateless device.

Another point was made, that Sun Rays are dependent on a working network connection. As more and more vital information is kept on servers that require online access, you already are dependent of a connection to your company's network.

Offline work usually requires documents to be carried with in paper form (bad!) or on harddisk. The later would again require harddisk encryption.

Do you trust harddisk encryption made in China?

Links for 2007-07-17

2007-07-17T21:32:00.000+02:00

Comparison of open source configuration management software
(Wikipedia)
Virtual Desktop Talk (Podcast on desktop virtualization)
Sun Desktop Virtualization Solution Blueprint (with VMWare)
Win4Solaris (Running Windows on Solaris)

SunRay Stuff

2007-07-16T22:30:00.001+02:00

ThinGuy has a few interesting blog entries...

First of all there are some Youtube movies about the Sun/Mitel partnership. This sounds like a great deal. The deal consists of two parts. A Multi-Instance Call Server (I guess this is something like a phone switch), and a unified solution for SunRays and the Mitel IP phone. This allows to hotdesk not only between SunRays, but also between IP phone. You can find more information here.

Another entry in ThinGuy's blog sounds also promising. He will talk in august about the upcoming SunRay Software 4.2. He promises some "trendy" new features for desktop virtualisation. As always, these kind of product release events are never around the corner...

I really believe that in the next 1-2 years desktop virtualization will be the next big management buzzword (not in the negative meaning).

The only thing Sun desperately needs if they want SunRays to take off, are salesmen who really want to sell thin clients. Selling Sunrays is certainly harder than selling big boxes, but hey, take it as a challenge!

But maybe Sun salesmen just don't understand the real-life problems of todays desktops and how thin-clients are a solution for most of these problems...

Links for 2007-07-15

2007-07-15T21:53:00.000+02:00

Seattle Conference on Scalability Videos (from Storagemojo)
IT Jungle about Oracle 11g
IT Jungle about AIX 6.1 Beta
Hitachi Data Systems, Upcoming Webinars
storagearchitect.blogspot.com

Configuration Engines for Unix

2007-07-15T10:06:00.000+02:00

As a system administrator, there is one problem that is persistent. Standardizing and keeping track of configuration changes.

Standardizing begins with the installation of a system. All major unix brands have their own installation methods. As mainly a Solaris administrator, I'm very familiar with the Jumpstart framework.

Using plain vanilla jumpstart is ok, if no customization in addition to the OS is needed (special configurations/application installation)

For advanced customization, Sun Professional Services UK developed the JET framework. JET is an addition to Jumpstart. The advantage of JET is in its usage of template files. All information about a client to install is kept in one file. The framework provides a simple way to add additional software, make additional software changes.

With JET it is possible to get a host running with all its settings and applications. But,
as soon as there are changes on a specific host, which could affect standardization, those changes have to be propagated towards JET.

A couple of years ago, when we used our own framework on top of Jumpstart, we already had this discussion, about getting changes back to Jumpstart. We never found a real solution to this problem. Sometimes, changes were forgot by human error (lazyness?), or some changes would not fit into the framework.

Fortunatly there are others, that have had the same problem. After googling some time, I found three configuration engines:

While I haven't looked at these tools in detail, all of them are managing configuration files from a central host. Local configuration changes are always initiated from a central host. The new configuration will either be pushed from the central host or pulled from the local host. If somebody changes configuration files locally they will be overwritten. The local configurations can be generated dynamically by rules.

The more I think about it, the right way seems to be to install a plain-vanilla OS, then install additional software packages (the configuration engines support this too), and finally push the configuration to the host.

This would make it simple to reinstall or upgrading the OS on a host. Jumpstart could be used in a plain-vanilla configuration, with only needing the configuration engine to be installed.

I will certainly further investigate the tools above.

Using ZFS clones with several Application instances

2007-07-12T22:06:00.000+02:00

This was an interesting blog entry about E25Ks, DTrace and ZFS.

One thing I was also thinking about, was the performance gains with ZFS clones, when running several instances of Applications using mostly the same data.

Quote:

"Then came a flash of inspiration. Using clones of a ZFS snapshot of the data together with Zones it was possible to partition multiple instances of the application. But the really cool bit is that ZFS snapshots are almost instant and virtually free.

ZFS clones are implemented using copy-on-write relative to a snapshot. This means that most of the storage blocks on disk and filesystem cache in RAM can be shared across all instances. Although snapshots and partitioning are possible on other systems, they are not instant, and they are unable to share RAM."

I had this idea, when I was thinking about using Solaris as a Xen Dom0 and running several identical (at least in the beginning) MS-Windows instances on Xen DomUs. The cloned MS-Windows images would be located on ZFS clones.

Most of the blocks would then reside in the memory of Dom0. I guess this would certainly improve performance.

The OS images could of course be also served over e.g. ZFS iSCSI Target devices, but the effect would be the same...