I was reading Ric Wheelers "One Billion Files: Scalability Limits in Linux Filesystems". As a ZFS user, I was wondering how many files we store on one of our mail storage systems in a single zpool.

My colleague was so kind to start a find on the system. Four days later we got

# find /export -type f | wc -l 

Interesting. We are already close to one billion files in production.

Next step was to look at the average file size. Currently, 12.5TB are referenced, compression ratio is 1.77x. This results in a average size of ~30kB.

How long will it take to reach one billion files? 

In average, 60 mails get delivered per second to the storage system (over NFSv4!). Therefore, to get the missing 190 million files, we only need to wait a little bit more than a month.


Very cool indeed! Is this on a Unified storage system? What hardware config. do you have? Latest firmware & SAS disks?

I'm asking because hadn't had much success with our 7410 system. I've heard the SATA disks we have (1st gen system) were the biggest cause of problems.

Keep us posted when you reach 1B files!
Mika said…
Yes, this is on the 7410, also 1st gen.

It's important to get the newest Firmware for the BIOS/SP, Disks and SIM Modules...

We just upgraded to the Q3 Release. Things are starting to look really good, with most of the annoyances solved.

There are still some deferred zpool updates pending, which should improve snapshot-performance.

Will keep you updated...

Popular posts from this blog

Opensolaris, Huawei E220, Swisscom and Sunrise

Adjusting ZFS resilvering speed

Heating up the Data Pipeline (Part 1)