Setting Up a NAS on NexentaCP
06-14-09
A while back I decided to take advantage of some deals on 1TB disks and bought myself 5TB of storage with the intent to build out a NAS that could serve files and streaming media using RAID5 with hot spare. The adventure took a turn as I did more research after having purchased the disk and RAID controller.
My research turned up many warnings about RAID5 on such large volumes and a fair amount of SUN propaganda for ZFS. With RAID5 rebuild times of many hours and the risk of a second drive failure during the rebuild in my mind as I read through the ZFS information, I was sold that ZFS was the way to go. No rebuild times, a RAID implementation that understands the filesystem, multiple CRC checks guaranteeing clean data. It sounded like paradise…well, for my data at least 
Having chosen the ZFS route, I next had to choose a platform to run it on. OpenSolaris would be the defacto “standard”, but I’m not a huge Solaris fan, coming from a BSD background, I find it really difficult to administer Solaris and most unix software these days is written on BSD or Linux, so compiling it on Solaris can often be a challenge. NexentaCP is the base for NexentaStor, a commercial NAS operating system. It’s based on the OpenSolaris kernel (and bits of userland) and wraps it all with an Ubuntu package infrastructure. Lastly, I had FreeBSD. ZFS, was brought into FreeBSD in the 7.x timeframe, but is still considered experimental and the mailing lists are filled with people having stability issues with it (although to some extent that has quieted down in recent months).
Initially I chose NexentaCP, I was familiar with Ubuntu and it seemed like I’d have the most luck. Within a week or so of the install I decided it wasn’t for me. It’s a small community, very little documentation, much of it accurate only for the 1.0 install (I was trying a beta at the time…my mistake). I didn’t know what I was doing and couldn’t find the usual plethora of resources I could on other platforms. Later I realized that most of the information that I needed is for OpenSolaris (more on that in a bit). NexentaCP did come with apt-clone, a rocking way to do ZFS based transactional upgrades or package installs. apt-clone sets up a ZFS snapshot of your syspool (system files) filesystem and installs or upgrades the OS depending on what action you chose to perform.
I quickly looked at FreeNAS which is FreeBSD based, but quickly opted against it. The only versions that supported ZFS were the 0.7 nightlies which were FreeBSD 7 based and I’d seen enough horror stories for ZFS on FreeBSD 7 that I didn’t feel it was an option. Regardless, only the amd64 build would boot on my machine and at the time there was no recent nightly build of that. The i386 builds would just hang on boot. FreeBSD itself didn’t have a way to native install onto a ZFS root, nor could it boot it and using GEOM mirror for root wasn’t something I felt like dealing with, so again, I moved on.
OpenSolaris lasted for a couple weeks. I was mostly happy with it, the install was pretty straightforward, disk setup wasn’t horrible and I had a ZFS mirror boot setup and a raidz2 for my storage volume with the 5 1TB drives. I hated ipkg, it didn’t have anything I was looking for, which pushed me to hand compiling stuff. Now, I’m not against compiling applications, or even attempting to fix them to get them compiled, I do know what I’m doing. But if everything you want to do requires ./configure && make && make install, you may as well not even have a package system. The memory of compile hell in linux in 1995 was enough to make me ditch OpenSolaris.
At first glance, it would seem that I’m out of options, but no, FreeBSD 8 which, while still under development, is actually quite stable and has numerous ZFS fixes in it is next in my merry road trip. It’s actually lasted the longest on my NAS out of all the choices thus-far. However, dismal NFS performance (around 6MB/sec) and sub-par iSCSI performance (14MB/sec) basically made me stop using it as a NAS for anything but serving up ISOs to my ESX box. The ports system however rocked hard enough that I stayed on FreeBSD for two months before finally deciding that enough was enough. Before moving on to my current install, I do want to talk briefly about the install process for ZFS root on FreeBSD 8. Currently sysinstall (the FreeBSD installer) doesn’t support ZFS so the install is a real pain in the rear. First you have to install to USB drive, rebuild the system with all the ZFS bits enabled (don’t forget ZFSLOADER support!) and then you basically get to manually fdisk, install bootloaders, and cpdup files to the new file system. This process took me the better part of two days and at least a half dozen failed attempts to boot the system before I got a working install. Hopefully the process becomes much better in the future – as of today, it’s impossible for a normal person to get ZFS on FreeBSD.
Fast forward to present day. Yesterday I spent the day reinstalling the NAS one more time – hopefully this will truly be the last. I’m back on NexentaCP, this time the 2.0 release (although there’s a small amount of confusion about whether it’s 2.0RC3 or 2.0 – it sounds like RC3 became the release, but I’m not positive). Not really much has changed with Nexenta since I last looked at it, documentation is still primarily for the 1.0 release, apt-clone still rocks though. I haven’t gotten to far on this install, I still need to install something to serve up my media files, but the package repo is full of software and I’ve already got Transmission installed so I can start up the torrents again. During my travels through the OS’s, I started thinking that it’d be nice to use Xen on this box, those dreams were dashed however as NexentaCP 2.0 doesn’t appear to ship with the bits needed to run Xen dom0. I’ll have to make due with Solaris Zones in the meantime, I’ll write up a post on that when I get around to trying them. In the meantime, I’ve got more throughput testing to do, but preliminary tests last night had me running 60MB/sec NFS transfers, a 10x speed improvement over FreeBSD.
To summarize, NexentaCP seems to be the best setup out there today if you insist on running ZFS and can’t stomach the thought of OpenSolaris. It’ll require some amount of self sufficiency, but any seasoned unix admin should be able to pick it up and figure it out without too much difficulty. And it’s FAST.
