I think that coming up with the BOM pretty much handles most of the design decisions. For the final choice of distribution, I usually leave this as an exercise to the reader.
This time, as discussed, I intend to use ZFS, once again. This filesystem allocates new sparse blocks when a file is overwritten, new sets of metablocks that write the list of pointers to the blocks, the file inodes and the directories that contain those file inodes etc. So, at any point in the time, the old data was never half written nor removed, and it should be recoverable. This forms the basis of ZFS snapshots. Efficient, Data Preserving and Recoverable. This used to be the sort of technology that powered WAFL, the NetApp filesystem. But now, it’s free, open source and cross platform.
Any filesystem that does all this, and is cross platform — sign me up for it. The last time I set up ZFS, it wasn’t on Solaris, wasn’t on FreeBSD; I was using ZoL (ZFS On Linux), and I set it up with DKMS and Debian Squeeze (6.0). Currently, this is now one unified codebase for FreeBSD and Linux, and became the OpenZFS project.
Distribution Choice: TrueNAS Core
This time I decided to install the TrueNAS Core 12.0-U4 (previously FreeNAS) distribution on this NAS box. I wanted something that end users would want, something that actually took care of what a NAS appliance should do, only reliably — and allowed customizations.
Hardware Setup Part 1: iLO+RAM Upgrades+Disks
Here is a YouTube tutorial on open the box and add hard disks, iLO Card to it. You may wish to upgrade the RAM if you like.
Hardware Setup Part 2: NVMe PCIe Drive + 4G LACP Option
I chose to add a NVMe PCIe Drive to the free PCIe slot. I decided to stick to my design, I’m connecting all of the network ports and running them in LACP, 4Gbps is sufficient for my Home Office Needs. This is the option I chose.
Other Design Choice: 10G NIC Customization Option
If you instead wanted faster external networking, reduced number of RAID disks and slightly slower (due to SATA) log writes, you would swap one slot’s HDD for a 2.5″ Flash Drive (for L2ARC/ZLOG) and and then you add a 10GBE SFP+ card. You’d also need to buy a tray for the 2.5″ drive.
Hardware Setup Part 3: Boot Drive
I added the two USB drives to the splitter, and plugged them into my internal port. One of them was flashed with the TrueNAS Installation ISO. It will eventually be repurposed as a root volume mirror post the installation.
Hardware Setup Part 4: Network
Currently, for the purposes of boot, I’ve just connected one network cable to the iLO and the second one to the first of the four 1 GBE ports on the MicroServer. You can remove the iLO port connection once you are satisfied with the setup.
Software Setup Part 1: iLO Login and Boot Order
You can run the whole installation headless, and you can access the display through the network via iLO. Once the system is connected to power (but not turned on yet). and the iLO is connected to the network, you should be able to access it via https://IP-of-iLO
I set up the boot order as follows:
You can always HTTP/PXE boot this machine, use the iLO interface to mount an ISO, do a bunch of things to install the distribution. However, having done this exercise a few times and aware of the installation time, I would highly recommend you already flash one USB Flash Disk ahead of time and it will save you a ton of time setting up the machine.
Once you set the boot priority like this, you should adjust it accordingly after the installation.
Software Setup Part 2: TrueNAS Core
The Boot screen will look like this:
You basically want to select da0 (or da1) as the root volume. Install, and adjust the boot order, and reboot.
Once you reboot, you should be able to see this on the iLO:
and then you should be able to open the TrueNAS UI on your browser. If you did not enter a root password during installation, you should be prompted to create one. Do that and log in.
Software Setup Part 3: Mirror Boot Volume
Go to Storage, Disks, click on the USB Installation Disk, wipe it.
Go to System > Boot > Actions > Boot Pool Status and attach that disk to the Boot Volume. You now have a RAID1/2 Way mirror for this boot volume.
Software Setup Part 4: Setup Data Volume
Click on Storage, Disks, select the non NVMe/non Flash Disks (ada* but without the Flash Drive) and create a pool. I call it datapool. It would look like this:
Ignore the nvd0 drive (or other SATA flash drives) for now, only create the volume with Data VDEVs.
We will partition this flash drive into two parts and then manually add them as Cache and Log VDEVs.
Software Setup Part 5: Partition and Log+Cache Volume Devices
First, I used gpart in the Shell and manually added a Log Volume in nvd0; This became the nvd0p1 partition. An equivalent command in the Linux space would either be cfdisk or again gpart. The other partition I created was the Cache Volume in the nvd0p2 partition.
Once you add the Log volume to your Data Pool, it would show like this:
Then you add the Cache volume to your Data Pool:
Of course, TrueNAS is optimized for larger setups, but I wish they had set up better partitioning and slicing of flash for smaller setups.
Software Setup Part 6: Set FS options
Unix systems record the access time of files when people try to open them, which is not a behavior I’d recommend to most users unless they need some sort of auditing on when their file was last opened. So this is the first thing I did:
zfs set atime=off datapool
zfs set relatime=off datapool
Software Setup Part 7: Provision Unix Users
I went to Accounts > Users, created regular (non-root) users on this system. I pointed their home directory to the Data Pool, such as /home/datapool/john
Software Setup Part 7: Storage Access
I went to Services > SMB and enabled it. The non-root users should now be able to access their home directories via the network. Just open \TRUENAS , log in and it works beautifully.
At this point, I would say that the basic setup is complete. You can set up other services such as NFS and AFP. As far as MinIO goes, I recommend you run this from a jail.
Software Setup Part 7: Customization Options
I set up jails for some of the services I wanted to run, such as Plex Media Server, MinIO and Gitlab. You may also want to run Linux/other VMs on it if you wish;
You could set up sub volumes and an iSCSI service so that other machines can use this as a network hard disk. This post talks about how to create this setup. People often use these for ESXi or other cloud volume setups.
For the volumes, I enabled dedup. Not everyone needs this option though, your access pattern may vary.
I decided to connect all my 4 NIC ports too, and enabled Link Aggregation/LACP. This is a fun — and a great thing to learn, irrespective of how big a single NIC link is.
Complete: Final Words
That’s it. This is my setup. Not every setup is about wrangling on the command line, although we did a bit for the flash partitions.
I am truly happy that TrueNAS Core has come a long way and has made it absolutely easy to set up a very reliable storage box at home. Their attention to root volumes, recover-ability, boot disk replacements, logging on data volumes only — is simply amazing, I wish more distributions followed a similar approach.
Reach out to me through Email or Twitter if you get stuck in the process.
This DIY NAS Setup is about learning a few things on your own, so I may have omitted a few details in the process, expecting you to read more on the options available. Feedback/Comments on more options are always welcome.