A Gentle Introduction to ZFS, Part 3: Network File Systems and iperf3 Testing

12 min readDec 28, 2020

If you haven’t yet setup a basic ZFS storage pool yet, please do so first by following the article, A Gentle Introduction to ZFS, Part 2: Storage Pools and Hard Disk Drives on the Jetson TX2.

In this article, we’ll be sharing our ZFS storage pool with our home network and also measuring the performance of the network channel. We’ll be developing this solution on Jetpack R32.4.4 (Ubuntu 18.04, aarch64).

What are Network File Systems?

The Network File System (NFS) is a standard for accessing server network storage as a local file system on a client. The first version was developed within Sun Microsystems (Oracle), but later released as version 2 in RFC 1094 on March 1989 (if you’ve never read a Request For Comment (RFC) before, think of it as a white paper for the internet). Version 3 added 64-bit addressing and asynchronous writes in RFC 1813 on June 1995. Finally, NFSv4, over time, added in numerous performance improvements, security, and an improved stateful protocol in RFC 7530 by March 2015. The takeaway from this is that NFS represents a well-polished network file system built on over 30 years of research (there are a lot of ITs still rocking NFSv3). We’ll also be using NFSv3 to share our ZFS file system with the rest of our home network, both for compatibility with Windows (only supports NFSv3) and also because only the NFSv3 (arm64) appears to advertise on RPC. For some reason, NFSv4 does not want to start up on TX2.

Install NFS on the Server

Download the packages dependencies:

$ sudo apt install build-essential make autogen autoconf libtool gawk alien fakeroot curl wget flex bison dkms zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev parted lsscsi ksh libssl-dev libelf-dev nfs-kernel-server open-iscsi watchdog xattr samba samba-client acl smartmontools mailutils nfs4-acl-tools

Setup the NFS kernel server configuration file at /etc/default/nfs-kernel-server:

# Number of servers to start up
RPCNFSDCOUNT=8# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids"# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=no# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""

I’m setting up NFS with no security. It’s up to you if you’d like to enable Kerberos or not. Security is disabled by default, anyway.

Make sure the Nobody-User and Nobody-Group match up on both the server and client side in the /etc/idmapd.conf file:

...[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup...

Restart and enable the NFS server:

$ sudo systemctl start nfs-server
$ sudo systemctl enable rpcbind nfs-server

Verify that NFS is running:

$ rpcinfo -p | grep nfs
    100003    3 tcp   2049  nfs
    100003    3 udp   2049  nfs

Note that version3 is shown here, but version 4 appears to be missing(?). I don’t have an explanation for that.

Configure NFS Exports on the Server

Create the export directories for NFS and configure permissions. Replace each dataset# name with your own dataset names:

$ sudo mkdir -p /export/<dataset1>
$ sudo mkdir -p /export/<dataset2>
$ sudo mkdir -p /export/<dataset3>
$ sudo chown -R nobody:nogroup /export
$ sudo chmod -R 0777 /export

Be sure you know what you’re doing when removing all restrictions from a directory (i.e. 0777). This may not be feasible in a production environment. M’kay?

Mr. Mackey says you might not want to give 0777 permissions in a production environment.

Mount and bind the export directories to their respective ZFS dataset directories. The bind here lets us remount part of a file hierarchy at a different location while it is still available at the original location using the following commands:

$ sudo mount --bind /tank/<dataset1> /export/<dataset1>
$ sudo mount --bind /tank/<dataset2> /export/<dataset2>
$ sudo mount --bind /tank/<dataset3> /export/<dataset3>

Note: replace tank with the name of your ZFS storage pool.

Make this bind persistent by editing your /etc/fstab file to export each ZFS dataset. The fstab file will automatically mount the listed filesystems on boot.

# /etc/fstab: static file system information. 
# 
# These are the filesystems that are always mounted on boot, you can 
# override any of these by copying the appropriate line from this file into 
# /etc/fstab and tweaking it as you see fit.  See fstab(5). 
# 
# <file system>         <mount point>           <type>  <options>                                                       <dump>  <pass> 
/dev/root               /                       ext4    defaults                                                        0       1 
/tank/dataset1       /export/dataset1          none    bind,defaults,nofail       0       0 
/tank/dataset2       /export/dataset2         none    bind,defaults,nofail       0       0 
/tank/dataset3       /export/dataset3           none    bind,defaults,nofail       0       0

tank is your ZFS storage pool name.
dataset# is the name of the dataset that you will substitute with your own.
bind is functionally the same asmount --bind from above.
defaults applies default mount settings (equivalent to rw,suid,dev,exec,auto,nouser,async).
nofail allows the boot sequence to continue even if the drive fails to mount.

Now that we have our export directories bound, we need to advertise the local filesystems to NFS clients through the /etc/exports file. exportfs will read this table to determine what filesystems to export to NFS clients. Each line contains an export point and a whitespace-separated list of clients allowed to mount the file system at that point. Each listed client may be immediately followed by a parenthesized, comma-separated list of export options for that client. No whitespace is permitted between a client and its option list.

# /etc/exports: the access control list for filesystems which may be exported 
#               to NFS clients.  See exports(5). 
# 
# Example for NFSv2 and NFSv3: 
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) 
# 
# Example for NFSv4: 
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) 
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check) 
 
/export                 192.168.50.0/24(ro,all_squash,anonuid=65534,anongid=65534,no_subtree_check,fsid=0,crossmnt) 
/export/dataset1          192.168.50.0/24(rw,async,all_squash,anonuid=65534,anongid=65534,no_subtree_check) 
/export/dataset2        192.168.50.0/24(rw,async,all_squash,anonuid=65534,anongid=65534,no_subtree_check) 
/export/dataset3           192.168.50.0/24(rw,async,all_squash,anonuid=65534,anongid=65534,no_subtree_check)

/export/dataset# is the NFS partial mount path.
192.168.50.0/24 is the CIDR subnet mask for my network. For your network, record your own IP on the network and replace the last octet with “0/24”.
ro/rw mean read-only/read-write, respectively.
async allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage. This is a huge speedup for HDDs with mechanical heads. The only downside is that a write during a system crash probably will result in corrupt data.
all_squash maps requests from any uid/gid to the anonymous uid/gid (hence the squash). If this kind of access is unpleasant to you, feel free to use root_squash instead, but beware that root may not be the same uid/gid on every system.
no_subtree_check disables subtree checking, which can cause problems with accessing files that are renamed while a client has them open (though in many simple cases it will still work). Generally, a home directory filesystem, which is normally exported at the root and may see lots of file renames, should be exported with subtree checking disabled.
anonuid=65534,anongid=65534 are the uid/gid of the anonymous user (nobody:nogroup on the TX2). This may be different on your system. cat the /etc/passwd and /etc/group for the anonymous user name.
fsid=0 means that this export is the root of all exported filesystems, when using NFSv4.
crossmnt makes it possible for clients to move from the filesystem marked with crossmnt to exported filesystems mounted on it. This is useful for exports that are mounted as the root of all exported filesystems.

Save the file(s), and restart the nfs-server:

$ sudo systemctl restart nfs-server

Mount the Share on the Client in Linux

Again, make sure the Nobody-User and Nobody-Group match up on both the server and client side in the /etc/idmapd.conf file:

...[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup...

Create a directory mount point. Replace tank with your ZFS storage pool name.

$ sudo mkdir -p /mnt/tank

Make sure that the mount points were exported:

$ showmount -e <server>
Export list for <server>: 
/export/<dataset1>   192.168.50.0/24 
/export/<dataset2>   192.168.50.0/24 
/export/<dataset3>   192.168.50.0/24 
/export              192.168.50.0/24

Mount the NFS share:

$ sudo mount -t nfs -o proto=tcp,port=2049 <nfs-server-IP>:/export /mnt/tank

List the /mnt/tank directory to see if your datasets are visible, and that you can write a file to the NFS. You should see something like this:

$ ls /mnt/tank
dataset1  dataset2  dataset3
$ touch /mnt/tank/test
touch: cannot touch '/mnt/tank/test': Read-only file system
$ touch /mnt/tank/dataset1/test
touch: cannot touch '/mnt/tank/dataset1/test': Permission denied

Uh oh! What happened here? Remember that we configured the base directory as read-only. Furthermore, we haven’t yet mounted dataset1, so there is no read-write mounted file system. Let’s mount the other datasets:

$ sudo mount -t nfs -o proto=tcp,port=2049 <nfs-server-IP>:/export/dataset1 /mnt/tank/dataset1
$ sudo mount -t nfs -o proto=tcp,port=2049 <nfs-server-IP>:/export/dataset2 /mnt/tank/dataset2
$ sudo mount -t nfs -o proto=tcp,port=2049 <nfs-server-IP>:/export/dataset3 /mnt/tank/dataset3
$ touch /mnt/tank/dataset1/test   (success!)

If you’d like, feel free to add the following mount entries to your /etc/fstab file to automount on boot:

<nfs-server-IP>:/export /mnt/tank nfs auto,user,_netdev 0 0
<nfs-server-IP>:/export/dataset1 /mnt/tank/dataset1 nfs auto,user,_netdev 0 0
<nfs-server-IP>:/export/dataset2 /mnt/tank/dataset2 nfs auto,user,_netdev 0 0
<nfs-server-IP>:/export/dataset3 /mnt/tank/dataset3 nfs auto,user,_netdev 0 0

auto means mount on boot.
user allows any user to mount (and not just root).
_netdev waits for network access before allowing the mount to proceed.

Mount from /etc/fstab and list the mounted directory to see if your expected datasets are present:

$ sudo mount -a
$ ls /mnt/tank
dataset1  dataset2  dataset3

Mount the Share on the Client in Windows

Install Client for NFS in Windows Features (Source: GraspingTech)

Install Client for NFS in Windows Features. Mount the NFS share to the drive letter of your choice:

mount -o anon <nfs-server-ip>:/export/tank X:

Try to list X:, it may take a long time to access the first time.

Windows 10 supports NFS mounting with NFSv3, so there’s little reason to enable Samba for Windows machines in the ZFS configuration. If you’d still like to try, see the Further Resourcessection.

iPerf3 Testing

iperf3 is a bandwidth testing tool for layer 3 networks (IP-based). We will be using this tool to measure the NAS’s actual bandwidth. Theoretically, a gigabit ethernet link should be able to support a 1Gbps bandwidth, but due to network inefficiencies, this is never the case. Let’s measure what the actual bandwidth is.

iPerf3 Server Setup

$ sudo apt install iperf3

Start up the iperf3 server:

$ iperf3 --server

--server indicates that the NAS will run the iperf3 server

Record the IP address or hostname of your NAS for use in the next section. You can discover the IP address with this command:

$ ip addr | grep eth

iperf3 as a SystemD Service (optional)

Feel free to also make this a service, so that the iperf3 server starts on boot. Do this by putting the following into /etc/systemd/system/iperf.service:

[Unit] 
Description=iperf3 service 
 
[Service] 
Type=simple 
ExecStart=/usr/bin/iperf3 --server > /dev/null & 
 
[Install] 
WantedBy=multi-user.target

As a side note, you might be tempted to call this file iperf3.service. Do not do this, as SystemD will think it’s an incompatible file and refer it to the older SysV, which will fail.

Finally, enable and start the service, by inputting the following commands:

$ sudo systemctl enable iperf
$ sudo systemctl start iperf

iPerf3 Client Setup

Next, on the Ubuntu client side, also make sure that iperf3 is installed:

$ sudo apt install iperf3

If you are on another Linux distribution, Windows, Mac or mobile, get iperf3 binaries from Iperf’s website or your favorite package manager. The following instructions assume an Ubuntu-flavor of Linux, but the command line should be similar.

Launch the client and setup a bandwidth test:

$ iperf3 --client <server IP address> --bytes 2G

--client <server IP address> specifies that iperf3 is running in client mode and is to connect to the <server IP address>
--format 2G instructs iperf3 to finish after transferring 2GB of data

What bits look like travelling down copper… sure, something like that.

After executing the client-side command, you may see some output like this:

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.50.222, port 49916
[  7] local 192.168.50.201 port 5201 connected to 192.168.50.222 port 49918
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd 
[  5]   0.00-1.00   sec   113 MBytes   944 Mbits/sec    0    409 KBytes        
[  5]   1.00-2.00   sec   112 MBytes   939 Mbits/sec    0    426 KBytes        
[  5]   2.00-3.00   sec   111 MBytes   932 Mbits/sec    0    445 KBytes        
[  5]   3.00-4.00   sec   112 MBytes   937 Mbits/sec    0    445 KBytes        
[  5]   4.00-5.00   sec   111 MBytes   930 Mbits/sec    0    467 KBytes        
[  5]   5.00-6.00   sec   111 MBytes   932 Mbits/sec    0    467 KBytes        
[  5]   6.00-7.00   sec   111 MBytes   934 Mbits/sec    0    488 KBytes        
[  5]   7.00-8.00   sec   111 MBytes   935 Mbits/sec    0    488 KBytes        
[  5]   8.00-9.00   sec   111 MBytes   935 Mbits/sec    0    488 KBytes        
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec    0    488 KBytes        
[  5]  10.00-11.00  sec   111 MBytes   933 Mbits/sec    0    488 KBytes        
[  5]  11.00-12.00  sec   112 MBytes   942 Mbits/sec    0    691 KBytes        
[  5]  12.00-13.00  sec   111 MBytes   933 Mbits/sec    0    691 KBytes        
[  5]  13.00-14.00  sec   111 MBytes   933 Mbits/sec    0    691 KBytes        
[  5]  14.00-15.00  sec   111 MBytes   933 Mbits/sec    0    691 KBytes        
[  5]  15.00-16.00  sec   111 MBytes   932 Mbits/sec    0    691 KBytes        
[  5]  16.00-17.00  sec   111 MBytes   934 Mbits/sec    0    691 KBytes        
[  5]  17.00-18.00  sec   111 MBytes   933 Mbits/sec    0    691 KBytes        
[  5]  18.00-18.37  sec  42.5 MBytes   953 Mbits/sec    0    691 KBytes        
- - - - - - - - - - - - - - - - - - - - - - - - - 
[ ID] Interval           Transfer     Bitrate         Retr 
[  5]   0.00-18.37  sec  2.00 GBytes   935 Mbits/sec    0             sender 
[  5]   0.00-18.37  sec  2.00 GBytes   934 Mbits/sec                  receiver 
 
iperf Done.

My client is on wired gigabit ethernet, but it doesn’t have to be (try wifi also). This measurement shows that the sustained bandwidth at the receiver is 934 Mbps or 117 MB/s. This will be the actual, measured upper bound of our NAS bandwidth instead of the theoretical 1000 Mbps.

Known Issues

Client Reports a Different Disk Capacity than the Server

$ sudo df -hH
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   30G   17G   12G  61% /
none            4.1G     0  4.1G   0% /dev
tmpfs           4.2G  4.1k  4.2G   1% /dev/shm
tmpfs           4.2G   51M  4.1G   2% /run
tmpfs           5.3M  4.1k  5.3M   1% /run/lock
tmpfs           4.2G     0  4.2G   0% /sys/fs/cgroup
tmpfs           825M   13k  825M   1% /run/user/120
tank            21T   525k   21T   1% /tank
tmpfs           825M     0  825M   0% /run/user/1000

The NFS server is reporting 21T of disk space available, however, the client is reporting a much different number:

$ sudo df -hH
Filesystem               Size  Used Avail Use% Mounted on
...
nfssrv:/export           30G   17G   12G  61% /mnt/himalayan
nfssrv:/export/dataset1  30G   17G   12G  61% /mnt/himalayan/photos
nfssrv:/export/dataset2  30G   17G   12G  61% /mnt/himalayan/shinobi
nfssrv:/export/dataset3  30G   17G   12G  61% /mnt/himalayan/video
...

It doesn’t seem like I’ve run out of inodes either:

$ df -i /tank 
Filesystem   Inodes        IUsed   IFree         IUse%   Mounted on 
tank         40301147711   6       40301147705   1%      /tank

Not only is the capacity much larger (30G), but also the the used capacity is 12G, compared with 525K on the server. The reason this is happening is because the /export directory is reporting the / filesystem disk space instead of the bound /tank storage pool. If that’s the case, consider exporting the storage pool’s mount point (i.e. /tank) instead of a bound /export directory instead.

This is running into some permissions issues:

$ sudo mount -v -t nfs -o rw,nfsvers=3 server:/tank/dataset1 /mnt/tank/dataset1
mount.nfs: timeout set for Wed Jan 20 08:50:55 2021
mount.nfs: trying text-based options 'nfsvers=3,addr=192.168.51.200'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.51.200 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.51.200 prog 100005 vers 3 prot UDP port 39972
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting server:/tank/dataset1

If using autofs, it’s helpful to run it in the foreground and see any error output:

$ sudo systemctl stop autofs
$ sudo automount -f -v

Summary

We setup the TX2 to bind the ZFS mount to an NFS partial mount and give full permissions to the anonymous group.
We exported our datasets through NFSv3 and optimized NFS to asynchronously write to the ZFS storage pool.
We mounted the NFS share on the client side in both Linux and Windows and setup a recurring mount in the fstab file in Linux.
We setup an iperf3 server on the TX2 and measured the channel bandwidth (934Mbps).

In the next article, we’ll be designing our storage pool and optimizing it for network access. Don’t miss this critical step!