FreeNAS® is © 2011-2016 iXsystems

FreeNAS® and the FreeNAS® logo are registered trademarks of iXsystems.

FreeBSD® is a registered trademark of the FreeBSD Foundation

Written by users of the FreeNAS® network-attached storage operating system.

Version 9.10.1-STABLE

Copyright © 2011-2016 iXsystems

This Guide covers the installation and use of FreeNAS® 9.10.1-STABLE.

The FreeNAS® Users Guide is a work in progress and relies on the contributions of many individuals. If you are interested in helping us to improve the Guide, read the instructions in the README. IRC Freenode users are welcome to join the #freenas channel where you will find other FreeNAS® users.

The FreeNAS® Users Guide is freely available for sharing and redistribution under the terms of the Creative Commons Attribution License. This means that you have permission to copy, distribute, translate, and adapt the work as long as you attribute iXsystems as the original source of the Guide.

FreeNAS® and the FreeNAS® logo are registered trademarks of iXsystems.

Active Directory® is a registered trademark or trademark of Microsoft Corporation in the United States and/or other countries.

Apple, Mac and Mac OS are trademarks of Apple Inc., registered in the U.S. and other countries.

Chelsio® is a registered trademark of Chelsio Communications.

Cisco® is a registered trademark or trademark of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

Django® is a registered trademark of Django Software Foundation.

Facebook® is a registered trademark of Facebook Inc.

FreeBSD and the FreeBSD logo are registered trademarks of the FreeBSD Foundation.

Fusion-io is a trademark or registered trademark of Fusion-io, Inc.

Intel, the Intel logo, Pentium Inside, and Pentium are trademarks of Intel Corporation in the U.S. and/or other countries.

LinkedIn® is a registered trademark of LinkedIn Corporation.

Linux® is a registered trademark of Linus Torvalds.

Marvell® is a registered trademark of Marvell or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates.

Twitter is a trademark of Twitter, Inc. in the United States and other countries.

UNIX® is a registered trademark of The Open Group.

VirtualBox® is a registered trademark of Oracle.

VMware® is a registered trademark of VMware, Inc.

Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.

Windows® is a registered trademark of Microsoft Corporation in the United States and other countries.

Typographic Conventions

The FreeNAS® 9.10.1-STABLE Users Guide uses the following typographic conventions:

  • Names of graphical elements such as buttons, icons, fields, columns, and boxes are enclosed within quotes. For example: click the “Import CA” button.
  • Menu selections are italicized and separated by arrows. For example: System ‣ Information.
  • Commands that are mentioned within text are highlighted in bold text. Command examples and command output are contained in green code blocks.
  • Volume, dataset, and file names are enclosed in a blue box /like/this.
  • Keystrokes are formatted in a blue box. For example: press Enter.
  • bold text: used to emphasize an important point.
  • italic text: used to represent device names or text that is input into a GUI field.

1. Introduction

FreeNAS® is an embedded open source network-attached storage (NAS) operating system based on FreeBSD and released under a 2-clause BSD license. A NAS has an operating system optimized for file storage and sharing.

FreeNAS® provides a browser-based, graphical configuration interface. The built-in networking protocols provide storage access to multiple operating systems. A plugin system is provided for extending the built-in features by installing additional software.

1.1. What’s New in 9.10.1

FreeNAS® uses a “rolling release” model instead of point releases. The Update mechanism makes it easy to keep up-to-date with the latest security fixes, bug fixes, and new features. Some updates affect the user interface, so this section lists any functional changes that have occurred since 9.10.1 was released.

Note

The screenshots in this documentation assume that your system is fully updated to the latest STABLE version of FreeNAS® 9.10.1. If a screen on your system looks different than the documentation, make sure that the system is fully up-to-date. If is is not, apply any outstanding updates.

  • Samba has been updated to 4.3.11.
  • The “Update Server” URL in System ‣ Update has changed to update.ixsystems.com. Due to the way the updater works, it will take two update cycles for existing FreeNAS® installations to reflect this change.
  • When security certificates or SSH keys are generated, the fingerprints are logged in /var/log/messages, var/log/debug.log, and the console.
  • Dashes have been added to the characters allowed in jail names.
  • The GUI prevents duplicate MAC addresses from being entered for jails.
  • A new alert is shown when attempting to bind NFS to a non-existent IP address.

1.2. Hardware Recommendations

Since FreeNAS® 9.10.1-STABLE is based on FreeBSD 10.3, it supports the same hardware found in the FreeBSD Hardware Compatibility List. Supported processors are listed in section 2.1 amd64. FreeNAS® is only available for 64-bit (also known as amd64) processors.

Note

FreeNAS® boots from a GPT partition. This means that the system BIOS must be able to boot using either the legacy BIOS firmware interface or EFI.

Actual hardware requirements vary depending on the usage of the FreeNAS® system. This section provides some starter guidelines. See the FreeNAS® Hardware Forum for performance tips from other FreeNAS® users or to post questions regarding the hardware best suited to meet your requirements. This forum post provides some specific recommendations for those planning on purchasing hardware. Refer to Building, Burn-In, and Testing your FreeNAS system for detailed instructions on how to test new hardware.

1.2.1. RAM

The best way to get the most out of a FreeNAS® system is to install as much RAM as possible. The recommended minimum is 8 GB of RAM. The more RAM, the better the performance, and the FreeNAS® Forums provide anecdotal evidence from users on how much performance is gained by adding more RAM.

Depending upon the use case, your system may require more RAM. Here are some general rules of thumb:

  • To use ZFS deduplication, ensure the system has at least 5 GB of RAM per TB of storage to be deduplicated.
  • To use Active Directory with many users, add an additional 2 GB of RAM for winbind’s internal cache.
  • When Using the phpVirtualBox Template, increase the minimum RAM size by the amount of virtual memory configured for use in virtual machines. For example, if there will be two virtual machines, each with 4 GB of virtual memory, the system needs at least 16 GB of RAM.
  • To use iSCSI, install at least 16 GB of RAM if performance is not critical, or at least 32 GB of RAM if good performance is a requirement.
  • When installing FreeNAS® on a headless system, disable the shared memory settings for the video card in the BIOS.

If the hardware supports it and the budget allows for it, install ECC RAM. While more expensive, ECC RAM is highly recommended as it prevents in-flight corruption of data before the error-correcting properties of ZFS come into play, thus providing consistency for the checksumming and parity calculations performed by ZFS. If you consider your data important, use ECC RAM. This Case Study describes the risks associated with memory corruption.

Unless the system has at least 8 GB of RAM, consider adding RAM before using FreeNAS® to store data. Many users expect FreeNAS® to function with less memory, just at reduced performance. The bottom line is that these minimums are based on feedback from many users. Requests for help in the forums or IRC are sometimes ignored when the installed system does not have at least 8 GB of RAM because of the abundance of information that FreeNAS® may not behave properly with less memory.

1.2.2. Compact or USB Flash

The FreeNAS® operating system is installed to at least one device that is separate from the storage disks. The device can be a USB stick, compact flash, or SSD. Technically, it can also be installed onto a hard drive, but this is discouraged as that drive will then become unavailable for data storage.

Note

to write the installation file to a USB stick, two USB ports are needed, each with an inserted USB device. One USB stick contains the installer. The other USB stick is the destination for the FreeNAS® installation. Take care to select the correct USB device for the FreeNAS® installation. It is not possible to install FreeNAS® onto the same USB stick containing the installer. After installation, remove the installer USB stick. It might also be necessary to adjust the BIOS configuration to boot from the new FreeNAS® USB stick.

When determining the type and size of the target device where FreeNAS® will be installed, keep these points in mind:

  • the bare minimum size is 8 GB. This provides room for the operating system and several boot environments. Since each update creates a boot environment, this is the recommended minimum. 16 GB provides room for more boot environments.
  • if you plan to make your own boot environments, budget about 1 GB of storage per boot environment. Consider deleting older boot environments after making sure they are no longer needed. Boot environments can be created and deleted using System ‣ Boot.
  • when using a USB stick, it is recommended to use a quality, name brand USB stick as ZFS will quickly reveal errors on cheap, poorly made sticks.
  • for a more reliable boot disk, use two identical devices and select them both during the installation. This will create a mirrored boot device.

1.2.3. Storage Disks and Controllers

The Disk section of the FreeBSD Hardware List lists the supported disk controllers. In addition, support for 3ware 6 Gbps RAID controllers has been added along with the CLI utility tw_cli for managing 3ware RAID controllers.

FreeNAS® supports hot pluggable drives. Using this feature requires enabling AHCI in the BIOS.

Reliable disk alerting and immediate reporting of a failed drive can be obtained by using an HBA such as an Avago MegaRAID controller or a 3Ware twa-compatible controller.

Suggestions for testing disks before adding them to a RAID array can be found in this forum post.

This article provides a good overview of hard drives which are well suited for a NAS.

If the budget allows optimization of the disk subsystem, consider the read/write needs and RAID requirements:

  • For steady, non-contiguous writes, use disks with low seek times. Examples are 10K or 15K SAS drives which cost about $1/GB. An example configuration would be six 600 GB 15K SAS drives in a RAID 10 which would yield 1.8 TB of usable space, or eight 600 GB 15K SAS drives in a RAID 10 which would yield 2.4 TB of usable space.
  • 7200 RPM SATA disks are designed for single-user sequential I/O and are not a good choice for multi-user writes.

When high performance is a key requirement and budget permits, consider a Fusion-I/O card which is optimized for massive random access. These cards are expensive and are suited for high-end systems that demand performance. A Fusion-I/O card can be formatted with a filesystem and used as direct storage; when used this way, it does not have the write issues typically associated with a flash device. A Fusion-I/O card can also be used as a cache device when your ZFS dataset size is bigger than your RAM. Due to the increased throughput, systems running these cards typically use multiple 10 GigE network interfaces.

For ZFS, Disk Space Requirements for ZFS Storage Pools recommends a minimum of 16 GB of disk space. Due to the way that ZFS creates swap, it is not possible to format less than 3 GB of space with ZFS. However, on a drive that is below the minimum recommended size, a fair amount of storage space is lost to swap: for example, on a 4 GB drive, 2 GB will be reserved for swap.

Users new to ZFS who are purchasing hardware should read through ZFS Storage Pools Recommendations first.

ZFS vdevs, groups of disks that act like a single device, can be created using disks of different sizes. However, the capacity available on each disk is limited to the same capacity as the smallest disk in the group. For example, a vdev with one 2 TB and two 4 TB disks will only be able to use 2 TB of space on each disk. In general, use disks that are the same size for the best space usage and performance.

1.2.4. Network Interfaces

The Ethernet section of the FreeBSD Hardware Notes indicates which interfaces are supported by each driver. While many interfaces are supported, FreeNAS® users have seen the best performance from Intel and Chelsio interfaces, so consider these brands when purchasing a new NIC. Realtek cards often perform poorly under CPU load as interfaces with these chipsets do not provide their own processors.

At a minimum, a GigE interface is recommended. While GigE interfaces and switches are affordable for home use, modern disks can easily saturate their 110 MB/s throughput. For higher network throughput, multiple GigE cards can be bonded together using the LACP type of Link Aggregations. The Ethernet switch must support LACP, which means a more expensive managed switch is required.

When network performance is a requirement and there is some money to spend, use 10 GigE interfaces and a managed switch. Managed switches with support for LACP and jumbo frames are preferred, as both can be used to increase network throughput. Refer to the 10 Gig Networking Primer for more information.

Note

At present, these are not supported: InfiniBand, FibreChannel over Ethernet, or wireless interfaces.

Both hardware and the type of shares can affect network performance. On the same hardware, CIFS is slower than FTP or NFS because Samba is single-threaded. So a fast CPU can help with CIFS performance.

Wake on LAN (WOL) support depends on the FreeBSD driver for the interface. If the driver supports WOL, it can be enabled using ifconfig(8). To determine if WOL is supported on a particular interface, use the interface name with the following command. In this example, the capabilities line indicates that WOL is supported for the re0 interface:

ifconfig -m re0
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
capabilities=5399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST, WOL_MAGIC,VLAN_HWFILTER,VLAN_H WTSO>

If you find that WOL support is indicated but not working for a particular interface, create a bug report using the instructions in Support.

orphan:

1.3. ZFS Primer

ZFS is an advanced, modern filesystem that was specifically designed to provide features not available in traditional UNIX filesystems. It was originally developed at Sun with the intent to open source the filesystem so that it could be ported to other operating systems. After the Oracle acquisition of Sun, some of the original ZFS engineers founded OpenZFS in order to provided continued, collaborative development of the open source version. To differentiate itself from Oracle ZFS version numbers, OpenZFS uses feature flags. Feature flags are used to tag features with unique names in order to provide portability between OpenZFS implementations running on different platforms, as long as all of the feature flags enabled on the ZFS pool are supported by both platforms. FreeNAS® uses OpenZFS and each new version of FreeNAS® keeps up-to-date with the latest feature flags and OpenZFS bug fixes.

Here is an overview of the features provided by ZFS:

ZFS is a transactional, Copy-On-Write (COW) filesystem. For each write request, a copy is made of the associated disk block(s) and all changes are made to the copy rather than to the original block(s). Once the write is complete, all block pointers are changed to point to the new copy. This means that ZFS always writes to free space and most writes will be sequential. When ZFS has direct access to disks, it will bundle multiple read and write requests into transactions; most filesystems can not do this as they only have access to disk blocks. A transaction either completes or fails, meaning there will never be a write-hole and a filesystem checker utility is not necessary. Because of the transactional design, as additional storage capacity is added it becomes immediately available for writes; to rebalance the data, one can copy it to re-write the existing data across all available disks. As a 128-bit filesystem, the maximum filesystem or file size is 16 exabytes.

ZFS was designed to be a self-healing filesystem. As ZFS writes data, it creates a checksum for each disk block it writes. As ZFS reads data, it validates the checksum for each disk block it reads. If ZFS identifies a disk block checksum error on a pool that is mirrored or uses RAIDZ*, ZFS will fix the corrupted data with the correct data. Since some disk blocks are rarely read, regular scrubs should be scheduled so that ZFS can read all of the data blocks in order to validate their checksums and correct any corrupted blocks. While multiple disks are required in order to provide redundancy and data correction, ZFS will still provide data corruption detection to a system with one disk. FreeNAS® automatically schedules a monthly scrub for each ZFS pool and the results of the scrub will be displayed in View Volumes. Reading the scrub results can provide an early indication of possible disk failure.

Unlike traditional UNIX filesystems, you do not need to define partition sizes at filesystem creation time. Instead, you feed a certain number of disk(s) at a time (known as a vdev) to a ZFS pool and create filesystems from the pool as needed. As more capacity is needed, identical vdevs can be striped into the pool. In FreeNAS®, Volume Manager can be used to create or extend ZFS pools. Once a pool is created, it can be divided into dynamically-sized datasets or fixed-size zvols as needed. Datasets can be used to optimize storage for the type of data being stored as permissions and properties such as quotas and compression can be set on a per-dataset level. A zvol is essentially a raw, virtual block device which can be used for applications that need raw-device semantics such as iSCSI device extents.

ZFS supports real-time data compression. Compression happens when a block is written to disk, but only if the written data will benefit from compression. When a compressed block is accessed, it is automatically decompressed. Since compression happens at the block level, not the file level, it is transparent to any applications accessing the compressed data. By default, ZFS pools made using FreeNAS® version 9.2.1 or later will use the recommended LZ4 compression algorithm.

ZFS provides low-cost, instantaneous snapshots of the specified pool, dataset, or zvol. Due to COW, the initial size of a snapshot is 0 bytes and the size of the snapshot increases over time as changes to the files in the snapshot are written to disk. Snapshots can be used to provide a copy of data at the point in time the snapshot was created. When a file is deleted, its disk blocks are added to the free list; however, the blocks for that file in any existing snapshots are not added to the free list until all referencing snapshots are removed. This means that snapshots provide a clever way of keeping a history of files, should you need to recover an older copy of a file or a deleted file. For this reason, many administrators take snapshots often (e.g. every 15 minutes), store them for a period of time (e.g. for a month), and store them on another system. Such a strategy allows the administrator to roll the system back to a specific time or, if there is a catastrophic loss, an off-site snapshot can restore the system up to the last snapshot interval (e.g. within 15 minutes of the data loss). Snapshots are stored locally but can also be replicated to a remote ZFS pool. During replication, ZFS does not do a byte-for-byte copy but instead converts a snapshot into a stream of data. This design means that the ZFS pool on the receiving end does not need to be identical and can use a different RAIDZ level, volume size, compression settings, etc.

ZFS boot environments provide a method for recovering from a failed upgrade. In FreeNAS®, a snapshot of the dataset the operating system resides on is automatically taken before an upgrade or a system update. This saved boot environment is automatically added to the GRUB boot loader. Should the upgrade or configuration change fail, simply reboot and select the previous boot environment from the boot menu. Users can also create their own boot environments in System ‣ Boot as needed, for example before making configuration changes. This way, the system can be rebooted into a snapshot of the system that did not include the new configuration changes.

ZFS provides a write cache in RAM as well as a ZFS Intent Log (ZIL). The ZIL is a temporary storage area for synchronous writes until they are written asynchronously to the ZFS pool. If the system has many synchronous writes where the integrity of the write matters, such as from a database server or when using NFS over ESXi, performance can be increased by adding a dedicated log device, or slog, using Volume Manager. More detailed explanations can be found in this forum post and in this blog post. A dedicated log device will have no effect on CIFS, AFP, or iSCSI as these protocols rarely use synchronous writes. When creating a dedicated log device, it is recommended to use a fast SSD with a supercapacitor or a bank of capacitors that can handle writing the contents of the SSD’s RAM to the SSD. The zilstat utility can be run from Shell to help determine if the system would benefit from a dedicated ZIL device. See this website for usage information. If you decide to create a dedicated log device to speed up NFS writes, the SSD can be half the size of system RAM as anything larger than that is unused capacity. The log device does not need to be mirrored on a pool running ZFSv28 or feature flags as the system will revert to using the ZIL if the log device fails and only the data in the device which had not been written to the pool will be lost (typically the last few seconds of writes). You can replace the lost log device in the View Volumes ‣ Volume Status screen. Note that a dedicated log device can not be shared between ZFS pools and that the same device cannot hold both a log and a cache device.

ZFS provides a read cache in RAM, known as the ARC, to reduce read latency. FreeNAS® adds ARC stats to top(1) and includes the arc_summary.py and arcstat.py tools for monitoring the efficiency of the ARC. If an SSD is dedicated as a cache device, it is known as an L2ARC and ZFS uses it to store more reads which can increase random read performance. However, adding an L2ARC is not a substitute for insufficient RAM as L2ARC needs RAM in order to function. If you do not have enough RAM for a good sized ARC, you will not be increasing performance, and in most cases you will actually hurt performance and could potentially cause system instability. RAM is always faster than disks, so always add as much RAM as possible before determining if the system would benefit from a L2ARC device. If you have a lot of applications that do large amounts of random reads, on a dataset small enough to fit into the L2ARC, read performance may be increased by adding a dedicated cache device using Volume Manager. SSD cache devices only help if your active data is larger than system RAM, but small enough that a significant percentage of it will fit on the SSD. As a general rule of thumb, an L2ARC should not be added to a system with less than 64 GB of RAM and the size of an L2ARC should not exceed 5x the amount of RAM. In some cases, it may be more efficient to have two separate pools: one on SSDs for active data and another on hard drives for rarely used content. After adding an L2ARC, monitor its effectiveness using tools such as arcstat. If you need to increase the size of an existing L2ARC, you can stripe another cache device using Volume Manager. The GUI will always stripe L2ARC, not mirror it, as the contents of L2ARC are recreated at boot. Losing an L2ARC device will not affect the integrity of the pool, but may have an impact on read performance, depending upon the workload and the ratio of dataset size to cache size. Note that a dedicated L2ARC device can not be shared between ZFS pools.

ZFS was designed to provide redundancy while addressing some of the inherent limitations of hardware RAID such as the write-hole and corrupt data written over time before the hardware controller provides an alert. ZFS provides three levels of redundancy, known as RAIDZ*, where the number after the RAIDZ indicates how many disks per vdev can be lost without losing data. ZFS also supports mirrors, with no restrictions on the number of disks in the mirror. ZFS was designed for commodity disks so no RAID controller is needed. While ZFS can also be used with a RAID controller, it is recommended that the controller be put into JBOD mode so that ZFS has full control of the disks. When determining the type of ZFS redundancy to use, consider whether your goal is to maximize disk space or performance:

  • RAIDZ1 maximizes disk space and generally performs well when data is written and read in large chunks (128K or more).
  • RAIDZ2 offers better data availability and significantly better mean time to data loss (MTTDL) than RAIDZ1.
  • A mirror consumes more disk space but generally performs better with small random reads. For better performance, a mirror is strongly favored over any RAIDZ, particularly for large, uncacheable, random read loads.
  • Using more than 12 disks per vdev is not recommended. The recommended number of disks per vdev is between 3 and 9. If you have more disks, use multiple vdevs.
  • Some older ZFS documentation recommends that a certain number of disks is needed for each type of RAIDZ in order to achieve optimal performance. On systems using LZ4 compression, which is the default for FreeNAS® 9.2.1 and higher, this is no longer true. See ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and Love RAIDZ for details.

The following resources can also help you determine the RAID configuration best suited to your storage needs:

Warning

NO RAID SOLUTION PROVIDES A REPLACEMENT FOR A RELIABLE BACKUP STRATEGY. BAD STUFF CAN STILL HAPPEN AND YOU WILL BE GLAD THAT YOU BACKED UP YOUR DATA WHEN IT DOES. See Periodic Snapshot Tasks and Replication Tasks if you would like to use replicated ZFS snapshots as part of your backup strategy.

While ZFS provides many benefits, there are some caveats to be aware of:

  • At 90% capacity, ZFS switches from performance- to space-based optimization, which has massive performance implications. For maximum write performance and to prevent problems with drive replacement, add more capacity before a pool reaches 80%. If you are using iSCSI, it is recommended to not let the pool go over 50% capacity to prevent fragmentation issues.
  • When considering the number of disks to use per vdev, consider the size of the disks and the amount of time required for resilvering, which is the process of rebuilding the vdev. The larger the size of the vdev, the longer the resilvering time. When replacing a disk in a RAIDZ*, it is possible that another disk will fail before the resilvering process completes. If the number of failed disks exceeds the number allowed per vdev for the type of RAIDZ, the data in the pool will be lost. For this reason, RAIDZ1 is not recommended for drives over 1 TB in size.
  • It is recommended to use drives of equal sizes when creating a vdev. While ZFS can create a vdev using disks of differing sizes, its capacity will be limited by the size of the smallest disk.

If you are new to ZFS, the Wikipedia entry on ZFS provides an excellent starting point to learn more about its features. These resources are also useful to bookmark and refer to as needed: