NAS (follow-up)

Last Febuary I built a NAS and I learned two things:

  1. Overbuild.
  2. Get a good case.

1. Overbuild

When I chose the Supermicro A1SRi-2558F, I wanted it because it was a robust board, low wattage, and an all Intel SATA chipset. It only has 6 SATA ports but I figured that would be plenty. Also the runner-up was a commonly used ASRock motherboard with 12 SATA ports but some of those SATA ports used a Marvell controller and at the time there were several forum posts about Marvell controllers losing the drives that were attached. So I figured I’d go the safe route and chose all Intel.

Six-months later, iX Systems announced their new FreeNAS Mini appliance and it looks like it uses that same ASRock motherboard. Maybe the bugs were worked out?

Hindsight

It’s too early to tell, but in a year or so I may be kicking myself for limiting the SATA ports. If I want to expand storage I’ll need to buy a SATA pci-e card and that will burn up the only slot the A1SRi has, and I’d like to do 10G networking too.

2. Get a Good Case

Initially, I reused an old case I had lying around. It was a Lian-Li PC60 and it has served me well over the past 10+ years. However the airflow wasn’t as good as some of the more modern products and the fans were just tiny. At a friends recommendation I purchased a Fractal Design – Define R5 and it’s the best case I have ever seen. This case is amazing, and it will definitely be worth every penny. Especially if I ever have to replace a drive.

Hindsight

Don’t reuse a case that’s 10+ years old. The modern cases have much better designed cable management, expansion, and airflow.

Over all I think I did fairly well for my first NAS. Up next is to redo my ESXi server that’s running Plex and owncloud. I also have a computer rack all setup for it now.

IMG_0127.JPG

IMG_0125.JPG

NAS

I finally built a NAS server. I’ve been thinking about it for a long time and finally just bit the bullet and built it. I was debating for quite a while whether I should DIY or buy a purpose built machine like a QNAP or Synology. I finally decided to go DIY and dove in.

Of course, after I decided to build my own I had to decide what to do with my ESX server. It’s currently running on my old Q9450 rig. It’s doing nicely except for decoding h265 video. Doing that will bring the Q9450 to its knees. Plus, it still chokes on higher bitrate h264 too. However the question was: do I build a box to replace both or keep them separate? I decided to keep them separate and built a single purpose NAS.

CASE: Lian Li PC-60

I re-purposed an old Lian Li PC-60. It’s not perfect for a NAS build as the hdd cage isn’t ideally rotated or easy to work with for a NAS, but I already had it lying around and that saved me $150 over my very close second choice, the Silverstone DS380. An extra $150 means one more hdd for storage too.

CPU/MOTHERBOARD: A1SRi-2558F-O ($296/Amazon)

Since this will be on all the time I wanted to get as low wattage as possible without worrying about being cpu limited. The FreeNAS Mini runs on a ASRock motherboard with the 8-core Atom. But I read about possible issues with the Marvell controller on the FreeNAS forums. Since I won’t have any support to call I ended up getting the Supermicro A1SRi. It has the 4-core Intel C2558 and Intel SATA controllers and Intel NICs.

If I was building an all-in-one unit having FreeNAS host my Plex server and other services, then I might have gone with the 8-core C2758. I’ll have to do more tests and see what happens to the 4-core cpu, but I think I’ll be happy with cpu perf if I’m just using this box as a NAS.

IPMI is fantastic. All future builds will have some sort of out-of-band management.

RAM: Kingston ValueRAM 8GB 1600MHz DDR3L PC3-12800 ECC (2x$56/Amazon)

After reading more on the FreeNAS forums, I’ve found out that Kingston isn’t a favorite and not recommended. (I think it was because they changed models and kept the same model names, so users didn’t know what they were getting.) Whoops, I already bought the memory. So, I ran memtest for several hours (not for days like some recommend) and everything seemed fine so I decided to keep it.

The motherboard will hold 64GB and the rule of thumb for ZFS is 1GB of memory for each TB of storage. I may fill out the 4 DIMM slots with 8GB modules for 32GB total. Tho’ that might be a bit overkill.

I have read now that the recommendation is 1GB of memory per TB of disk, not per TB of usable space. I have 24TB of disk (~14 usable in RAID-Z2), so I will be upgrading to 32GB of memory.

DRIVES: Six 4TB Toshiba drives ($109 each/Microcenter)

magical_hdd-y_goodness
All the magic. But it will have to be shutdown to replace a drive.

The biggest expense of any NAS build should be the drives, and I decided the 28% cheaper Toshiba drives were worth the chance over the common choice of WD Red drives. I guess I’ll find out how well they hold out in the long run. So far, they are fast, but noisy. However, this server is going in the basement so I don’t care too much about noise.

Six 4TB drives with RAID-Z2 nets me ~14TB of usable space.

OS: FreeNAS (9.3)

I was tempted to try Unraid, but I decided to go with FreeNAS as the price was better and I am already going down the path of ECC RAM so I might as well go with the extra checksumming and protection. Using a USB drive to install FreeNAS to is no longer recommended because of all the issues associated with USB drives. Now the only official recommendation is to install FreeNAS onto a SATA DOM drive. However, there are only six SATA ports on the motherboard and I didn’t want to burn up one with the FreeNAS installation drive, so I decided to forgo the recommendation and installed FreeNAS to a USB drive. (SanDisk Ultra Fit CZ43 16GB $8). Unfortunately, FreeNAS didn’t like the USB 3.0 port on the inside of the motherboard. Which is too bad, I really like the idea of having nothing but the network cable coming out of the case. I ended up using a USB 2.0 port on the back of the mobo.

PSU: Sparkle 500W 80 PLUS PLATINUM ($56/Newegg)

I probably went over board with this choice, but I’ve been burned on bad PSUs before. In hindsight, a fully modular PSU would be nicer for cable management. And more SATA power connectors would also be nice.

TOTAL COST: $1132

I did have a small ‘want’ when building this. I didn’t want it to consume a lot of power. Currently, the whole rig draws 55W at idle and 62W under load. I’d call that a win in the low power category.

The drives were about 70% of the cost so I think I did fairly well on the whole build in terms of cost. I almost had a refurbished motherboard saving me another $100, but that deal fell through. Maybe that’s for the best though.

PERFORMANCE

This setup will easily saturate a 1Gbps link. LACP won’t get me more throughput to my desktop, so 10G will have to be the next step, but it’s just too spendy for me right now. I may implement LACP someday, to separate out my ESX VM traffic from my desktop’s traffic, but it won’t be necessary in the near future. I’ll stick with my constant 90MB/s transfer speeds over iSCSI. I’m happy with that for right now.

Up Next: A new ESXi build to take over my current, aging, Q9450 machine.

Update: My six month follow up.