7. Storage¶

7.1.1. Volume Manager¶

The Volume Manager is used to add disks to a ZFS pool. Any old data on added disks is overwritten, so save it elsewhere before reusing a disk. Please see the ZFS Primer for information on ZFS redundancy with multiple disks before using Volume Manager. It is important to realize that different layouts of virtual devices (vdevs) affect which operations can be performed on that volume later. For example, drives can be added to a mirror to increase redundancy, but that is not possible with RAIDZ arrays.

Selecting Storage → Volumes → Volume Manager opens a screen like the example shown in Figure 7.1.1.

Fig. 7.1.1 Creating a ZFS Pool Using Volume Manager

Table 7.1.1 summarizes the configuration options of this screen.

Drag the slider to select the desired number of disks. Volume Manager displays the resulting storage capacity, taking reserved swap space into account. To change the layout or the number of disks, drag the slider to the desired volume layout. The Volume layout drop-down menu can also be clicked if a different level of redundancy is required.

Volume Manager only allows choosing a configuration if enough disks have been selected to create that configuration. These layouts are supported:

• Stripe: requires at least one disk
• Mirror: requires at least two disks
• RAIDZ1: requires at least three disks
• RAIDZ2: requires at least four disks
• RAIDZ3: requires at least five disks
• log device: requires at least one dedicated device, a fast, low-latency, power-protected SSD is recommended
• cache device: requires at least one dedicated device, SSD is recommended

When more than five disks are used, consideration must be given to the optimal layout for the best performance and scalability. An overview of the recommended disk group sizes as well as more information about log and cache devices can be found in the ZFS Primer.

The Add Volume button warns that existing data will be cleared. In other words, creating a new volume reformats the selected disks. To preserve existing data, click the Cancel button and refer to Import Disk and Import Volume to see if the existing format is supported. If so, perform that action instead. If the current storage format is not supported, it is necessary to back up the data to external media, format the disks, then restore the data to the new volume.

Depending on the size and number of disks, the type of controller, and whether encryption is selected, creating the volume may take some time. After the volume is created, the screen refreshes and the new volume is listed in the tree under Storage → Volumes. Click the + next to the volume name to access Change Permissions, Create Dataset, and Create zvol options for that volume.

7.1.1.2. Manual Setup¶

The Manual Setup button shown in Figure 7.1.1 can be used to create a ZFS volume manually. While this is not recommended, it can, for example, be used to create a non-optimal volume containing disks of different sizes.

Note

The usable space of each disk in a volume is limited to the size of the smallest disk in the volume. Because of this, creating volumes with disks of the same size through the Volume Manager is recommended.

Figure 7.1.2 shows the Manual Setup screen. Table 7.1.2 shows the available options.

Fig. 7.1.2 Manually Creating a ZFS Volume

Note

Because of the disadvantages of creating volumes with disks of different sizes, the displayed list of disks is sorted by size.

Table 7.1.2 Manual Setup Options¶

Setting	Value	Description
Volume name	string	ZFS volumes must conform to these naming conventions; choose a name that will stand out in the logs (e.g. not data or freenas)
Encryption	checkbox	see the warnings in Encryption before using encryption
Member disks	list	highlight desired number of disks from list of available disks
Deduplication	drop-down menu	do not change this setting unless instructed to do so by an iXsystems support engineer
ZFS Extra	bullet selection	specify disk usage: storage (None), a log device, a cache device, or a spare

7.1.2. Change Permissions¶

Setting permissions is an important aspect of configuring volumes. The graphical administrative interface is meant to set the initial permissions for a volume or dataset in order to make it available as a share. Once a share is available, the client operating system should be used to fine-tune the permissions of the files and directories that are created by the client.

The chapter on Sharing contains configuration examples for several types of permission scenarios. This section provides an overview of the screen that is used to set permissions.

After a volume or dataset is created, it is listed by its mount point name in Storage → Volumes. Clicking the Change Permissions icon for a specific volume/dataset displays the screen shown in Figure 7.1.3. Table 7.1.3 summarizes the options in this screen.

Fig. 7.1.3 Changing Permissions on a Volume or Dataset

The Windows Permission Type is used for SMB shares or when the TrueNAS^® system is a member of an Active Directory domain. This adds ACLs to traditional Unix permissions. When the Windows Permission Type is set, ACLs are set to Windows defaults for new files and directories. A Windows client can be used to further fine-tune permissions as needed.

The Unix Permission Type is usually used with NFS shares. These permissions are compatible with most network clients and generally work well with a mix of operating systems or clients. However, Unix permissions do not support Windows ACLs and should not be used with SMB shares.

The Mac Permission Type is used with AFP shares.

After a volume or dataset has been set to Windows, it cannot be changed to Unix permissions because that would remove extended permissions provided by Windows ACLs.

7.1.3. Create Dataset¶

An existing ZFS volume can be divided into datasets. Permissions, compression, deduplication, and quotas can be set on a per-dataset basis, allowing more granular control over access to storage data. Like a folder or directory, permissions can be set on dataset. Datasets are also similar to filesystems in that properties such as quotas and compression can be set, and snapshots created.

Selecting an existing ZFS volume in the tree and clicking Create Dataset shows the screen in Figure 7.1.4.

Fig. 7.1.4 Creating a ZFS Dataset

Table 7.1.4 summarizes the options available when creating a ZFS dataset. Some settings are only available in Advanced Mode. To see these settings, either click the Advanced Mode button, or configure the system to always display these settings by checking the box Show advanced fields by default in System → Advanced. Most attributes, except for the Dataset Name, Case Sensitivity, and Record Size, can be changed after dataset creation by highlighting the dataset name and clicking its Edit Options button in Storage → Volumes.

After a dataset is created, you can click on that dataset and select Create Dataset, thus creating a nested dataset, or a dataset within a dataset. A zvol can also be created within a dataset. When creating datasets, double-check that you are using the Create Dataset option for the intended volume or dataset. If you get confused when creating a dataset on a volume, click all existing datasets to close them–the remaining Create Dataset will be for the volume.

7.1.4. Create zvol¶

A zvol is a feature of ZFS that creates a raw block device over ZFS. This allows you to use a zvol as an iSCSI device extent.

To create a zvol, select an existing ZFS volume or dataset from the tree then click Create zvol to open the screen shown in Figure 7.1.5.

Fig. 7.1.5 Creating a Zvol

The configuration options are described in Table 7.1.5. Some settings are only available in Advanced Mode. To see these settings, either click the Advanced Mode button or configure the system to always display these settings by checking Show advanced fields by default in System → Advanced.

7.1.5. Import Disk¶

The Volume → Import Disk screen, shown in Figure 7.1.6, is used to import a single disk that has been formatted with the UFS, NTFS, MSDOS, or EXT2 filesystem. The import is meant to be a temporary measure to copy the data from a disk to an existing ZFS dataset. Only one disk can be imported at a time.

Fig. 7.1.6 Importing a Disk

Use the drop-down menu to select the disk to import, select the type of filesystem on the disk, and browse to the ZFS dataset that will hold the copied data. When you click Import Volume, the disk is mounted, its contents are copied to the specified ZFS dataset, and the disk is unmounted after the copy operation completes.

7.1.6. Import Volume¶

If you click Storage → Volumes → Import Volume, you can configure TrueNAS^® to use an existing ZFS pool. This action is typically performed when an existing TrueNAS^® system is re-installed. Since the operating system is separate from the storage disks, a new installation does not affect the data on the disks. However, the new operating system needs to be configured to use the existing volume.

Figure 7.1.7 shows the initial pop-up window that appears when you import a volume.

Fig. 7.1.7 Initial Import Volume Screen

If you are importing an unencrypted ZFS pool, select No: Skip to import to open the screen shown in Figure 7.1.8.

Fig. 7.1.8 Importing a Non-Encrypted Volume

Existing volumes should be available for selection from the drop-down menu. In the example shown in Figure 7.1.8, the TrueNAS^® system has an existing, unencrypted ZFS pool. Once the volume is selected, click the OK button to import the volume.

If an existing ZFS pool does not show in the drop-down menu, run zpool import from Shell to import the pool.

If you plan to physically install ZFS formatted disks from another system, be sure to export the drives on that system to prevent an “in use by another machine” error during the import.

7.1.6.1. Importing an Encrypted Pool¶

If you are importing an existing GELI-encrypted ZFS pool, you must decrypt the disks before importing the pool. In Figure 7.1.7, select Yes: Decrypt disks to access the screen shown in Figure 7.1.9.

Fig. 7.1.9 Decrypting Disks Before Importing a ZFS Pool

Select the disks in the encrypted pool, browse to the location of the saved encryption key, input the passphrase associated with the key, then click OK to decrypt the disks.

Note

The encryption key is required to decrypt the pool. If the pool cannot be decrypted, it cannot be re-imported after a failed upgrade or lost configuration. This means that it is very important to save a copy of the key and to remember the passphrase that was configured for the key. Refer to Managing Encrypted Volumes for instructions on how to manage the keys for encrypted volumes.

Once the pool is decrypted, it will appear in the drop-down menu of Figure 7.1.8. Click the OK button to finish the volume import.

7.1.7. View Disks¶

Storage → Volumes → View Disks shows all of the disks recognized by the TrueNAS^® system. An example is shown in Figure 7.1.10.

Fig. 7.1.10 Viewing Disks

The current configuration of each device is displayed. Click a disk entry and the Edit button to change its configuration. The configurable options are described in Table 7.1.6.

The Wipe function is provided for when an unused disk is to be discarded.

Clicking Wipe offers several choices. Quick erases only the partitioning information on a disk, making it easy to reuse but without clearing other old data. For more security, Full with zeros overwrites the entire disk with zeros, while Full with random data overwrites the entire disk with random binary data.

Quick wipes take only a few seconds. A Full with zeros wipe of a large disk can take several hours, and a Full with random data takes longer. A progress bar is displayed during the wipe to track status.

7.1.8. View Enclosure¶

Click Storage → Volumes → View Enclosure to receive a status summary of the appliance’s disks and hardware. An example is shown in Figure 7.1.11.

Fig. 7.1.11 View Enclosure

This screen is divided into the following sections:

Array Device Slot: has an entry for each slot in the storage array, indicating the disk’s current status and FreeBSD device name. To blink the status light for that disk as a visual indicator, click its Identify button.

Cooling: has an entry for each fan, its status, and its RPM.

Enclosure: shows the status of the enclosure.

Power Supply: shows the status of each power supply.

SAS Expander: shows the status of the expander.

Temperature Sensor: shows the current temperature of each expander and the disk chassis.

Voltage Sensor: shows the current voltage for each sensor, VCCP, and VCC.

7.1.9. Volumes¶

Storage → Volumes is used to view and further configure existing ZFS pools, datasets, and zvols. The example shown in Figure 7.1.12 shows one ZFS pool (volume1) with two datasets (the one automatically created with the pool, volume1, and dataset1) and one zvol (zvol1).

Note that in this example, there are two datasets named volume1. The first represents the ZFS pool and its Used and Available entries reflect the total size of the pool, including disk parity. The second represents the implicit or root dataset and its Used and Available entries indicate the amount of disk space available for storage.

Buttons are provided for quick access to Volume Manager, Import Disk, Import Volume, and View Disks. If the system has multipath-capable hardware, an extra button will be added, View Multipaths. For each entry, the columns indicate the Name, how much disk space is Used, how much disk space is Available, the type of Compression, the Compression Ratio, the Status, whether it is mounted as read-only, and any Comments entered for the volume.

Fig. 7.1.12 Viewing Volumes

Clicking the entry for a pool causes several buttons to appear at the bottom of the screen. The buttons perform these actions:

Detach Volume: allows you to either export the pool or to delete the contents of the pool, depending upon the choice you make in the screen shown in Figure 7.1.13. The Detach Volume screen displays the current used space and indicates if there are any shares, provides checkboxes to Mark the disks as new (destroy data) and to Also delete the share’s configuration, asks if you are sure that you want to do this, and the browser will turn red to alert you that you are about to do something that will make the data inaccessible. If you do not check the box to mark the disks as new, the volume will be exported. This means that the data is not destroyed and the volume can be re-imported at a later time. If you will be moving a ZFS pool from one system to another, perform this export action first as it flushes any unwritten data to disk, writes data to the disk indicating that the export was done, and removes all knowledge of the pool from the system. If you do check the box to mark the disks as new, the pool and all the data in its datasets, zvols, and shares will be destroyed and the underlying disks will be returned to their raw state.

Fig. 7.1.13 Detach or Delete a Volume

Scrub Volume: scrubs and scheduling them are described in more detail in Scrubs. This button allows manually initiating a scrub. Scrubs are I/O intensive and can negatively impact performance. Avoid initiating a scrub when the system is busy.

A Cancel button is provided to cancel a scrub. When a scrub is cancelled, it is abandoned. The next scrub to run starts from the beginning, not where the cancelled scrub left off.

The status of a running scrub or the statistics from the last completed scrub can be seen by clicking the Volume Status button.

Volume Status: as shown in the example in Figure 7.1.14, this screen shows the device name and status of each disk in the ZFS pool as well as any read, write, or checksum errors. It also indicates the status of the latest ZFS scrub. Clicking the entry for a device causes buttons to appear to edit the device’s options (shown in Figure 7.1.15), offline or online the device, or replace the device (as described in Replacing a Failed Drive).

Upgrade: used to upgrade the pool to the latest ZFS features, as described in Upgrading a ZFS Pool. This button does not appear if the pool is running the latest version of feature flags.

Fig. 7.1.14 Volume Status

Selecting a disk in Volume Status and clicking its Edit Disk button shows the screen in Figure 7.1.15. Table 7.1.6 summarizes the configurable options.

Fig. 7.1.15 Editing a Disk

Clicking a dataset in Storage → Volumes causes buttons to appear at the bottom of the screen, providing these options:

Change Permissions: edit the dataset’s permissions as described in Change Permissions.

Create Snapshot: create a one-time snapshot. To schedule the regular creation of snapshots, instead use Periodic Snapshot Tasks.

Promote Dataset: only applies to clones. When a clone is promoted, the origin filesystem becomes a clone of the clone making it possible to destroy the filesystem that the clone was created from. Otherwise, a clone can not be destroyed while its origin filesystem exists.

Destroy Dataset: clicking the Destroy Dataset button causes the browser window to turn red to indicate that this is a destructive action. The Destroy Dataset screen forces you to check the box I’m aware this will destroy all child datasets and snapshots within this dataset before it will perform this action.

Edit Options: edit the volume’s properties described in Table 7.1.4. Note that it will not allow changing the dataset’s name.

Create Dataset: used to create a child dataset within this dataset.

Create zvol: create a child zvol within this dataset.

Clicking a zvol in Storage → Volumes causes icons to appear at the bottom of the screen: Create Snapshot, Edit zvol, and Destroy zvol. Similar to datasets, a zvol’s name cannot be changed, and destroying a zvol requires confirmation.

7.1.9.1. Managing Encrypted Volumes¶

If the Encryption box is checked during the creation of a pool, additional buttons appear in the entry for the volume in Storage → Volumes. An example is shown in Figure 7.1.16.

Fig. 7.1.16 Encryption Icons Associated with an Encrypted Volume

These additional encryption buttons are used to:

Create/Change Passphrase: set and confirm a passphrase associated with the GELI encryption key. The desired passphrase is entered and repeated for verification. A red warning is a reminder to Remember to add a new recovery key as this action invalidates the previous recovery key. Unlike a password, a passphrase can contain spaces and is typically a series of words. A good passphrase is easy to remember (like the line to a song or piece of literature) but hard to guess (people who know you should not be able to guess the passphrase). Remember this passphrase. An encrypted volume cannot be reimported without it. In other words, if the passphrase is forgotten, the data on the volume can become inaccessible if it becomes necessary to reimport the pool. Protect this passphrase, as anyone who knows it could reimport the encrypted volume, thwarting the reason for encrypting the disks in the first place.

Fig. 7.1.17 Add or Change a Passphrase to an Encrypted Volume

After the passphrase is set, the name of this button changes to Change Passphrase. After setting or changing the passphrase, it is important to immediately create a new recovery key by clicking the Add recovery key button. This way, if the passphrase is forgotten, the associated recovery key can be used instead.

Encrypted volumes with a passphrase display an additional lock button:

Fig. 7.1.18 Lock Button

These encrypted volumes can be locked. The data is not accessible until the volume is unlocked by suppying the passphrase or encryption key, and the button changes to an unlock button:

Fig. 7.1.19 Unlock Button

To unlock the volume, click the unlock button to display the Unlock dialog:

Fig. 7.1.20 Unlock Locked Volume

Unlock the volume by entering a passphrase or using the Browse button to load the recovery key. If both a passphrase and a recovery key are entered, only the passphrase is used. By default, the services listed will restart when the volume is unlocked. This allows them to see the new volume and share or access data on it. Individual services can be prevented from restarting by unchecking them. However, a service that is not restarted might not be able to access the unlocked volume.

Download Key: download a backup copy of the GELI encryption key. The encryption key is saved to the client system, not on the TrueNAS^® system. The TrueNAS^® administrative password must be entered, then the directory in which to store the key is chosen. Since the GELI encryption key is separate from the TrueNAS^® configuration database, it is highly recommended to make a backup of the key. If the key is ever lost or destroyed and there is no backup key, the data on the disks is inaccessible.

Encryption Re-key: generate a new GELI encryption key. Typically this is only performed when the administrator suspects that the current key may be compromised. This action also removes the current passphrase.

Note

A re-key is not allowed if Failover (High Availability) has been enabled and the standby node is down.

Add recovery key: generate a new recovery key. This screen prompts for the TrueNAS^® administrative password and then the directory in which to save the key. Note that the recovery key is saved to the client system, not on the TrueNAS^® system. This recovery key can be used if the passphrase is forgotten. Always immediately add a recovery key whenever the passphrase is changed.

Remove recovery key: Typically this is only performed when the administrator suspects that the current recovery key may be compromised. Immediately create a new passphrase and recovery key.

Note

The passphrase, recovery key, and encryption key must be protected. Do not reveal the passphrase to others. On the system containing the downloaded keys, take care that the system and its backups are protected. Anyone who has the keys has the ability to re-import the disks if they are discarded or stolen.

Warning

If a re-key fails on a multi-disk system, an alert is generated. Do not ignore this alert as doing so may result in the loss of data.

7.1.10. View Multipaths¶

TrueNAS^® uses gmultipath(8) to provide multipath I/O support on systems containing hardware that is capable of multipath. An example would be a dual SAS expander backplane in the chassis or an external JBOD.

Multipath hardware adds fault tolerance to a NAS as the data is still available even if one disk I/O path has a failure.

TrueNAS^® automatically detects active/active and active/passive multipath-capable hardware. Any multipath-capable devices that are detected will be placed in multipath units with the parent devices hidden. The configuration will be displayed in Storage → Volumes → View Multipaths. Note that this option is not be displayed in the Storage → Volumes tree on systems that do not contain multipath-capable hardware.

7.1.11. Replacing a Failed Drive¶

Replace failed drives as soon as possible to repair the degraded state of the RAID.

Before physically removing the failed device, go to Storage → Volumes. Select the volume’s name. At the bottom of the interface are several icons, one of which is Volume Status. Click the Volume Status icon and locate the failed disk. Then perform these steps:

1. Click the disk’s entry, then its Offline button to change that disk’s status to OFFLINE. This step is needed to properly remove the device from the ZFS pool and to prevent swap issues. Click the disk’s Offline button and pull the disk. If there is no Offline button but only a Replace button, the disk is already offlined and this step can be skipped.

   Note

   If the process of changing the disk’s status to OFFLINE fails with a “disk offline failed - no valid replicas” message, the ZFS volume must be scrubbed first with the Scrub Volume button in Storage → Volumes. After the scrub completes, try to Offline the disk again before proceeding.

2. After the disk has been replaced and is showing as OFFLINE, click the disk again and then click its Replace button. Select the replacement disk from the drop-down menu and click the Replace Disk button. After clicking the Replace Disk button, the ZFS pool begins resilvering.

3. After the drive replacement process is complete, re-add the replaced disk in the S.M.A.R.T. Tests screen.

In the example shown in Figure 7.1.21, a failed disk is being replaced by disk ada5 in the volume named volume1.

Fig. 7.1.21 Replacing a Failed Disk

After the resilver is complete, Volume Status shows a Completed resilver status and indicates any errors. Figure 7.1.22 indicates that the disk replacement was successful in this example.

Fig. 7.1.22 Disk Replacement is Complete

7.1.12. Replacing Drives to Grow a ZFS Pool¶

The recommended method for expanding the size of a ZFS pool is to pre-plan the number of disks in a vdev and to stripe additional vdevs using Volume Manager as additional capacity is needed.

However, this is not an option if there are no open drive ports and a SAS/SATA HBA card cannot be added. In this case, one disk at a time can be replaced with a larger disk, waiting for the resilvering process to incorporate the new disk into the pool, then repeating with another disk until all of the original disks have been replaced.

The safest way to perform this is to use a spare drive port or an eSATA port and a hard drive dock. The process follows these steps:

1. Shut down the system.
2. Install one new disk.
3. Start up the system.
4. Go to Storage → Volumes, select the pool to expand and click the Volume Status button. Select a disk and click the Replace button. Choose the new disk as the replacement.
5. The status of the resilver process can be viewed by running zpool status. When the new disk has resilvered, the old one will be automatically offlined. The system is then shut down to physically remove the replaced disk. One advantage of this approach is that there is no loss of redundancy during the resilver.

If a spare drive port is not available, a drive can be replaced with a larger one using the instructions in Replacing a Failed Drive. This process is slow and places the system in a degraded state. Since a failure at this point could be disastrous, do not attempt this method unless the system has a reliable backup. Replace one drive at a time and wait for the resilver process to complete on the replaced drive before replacing the next drive. After all the drives are replaced and the final resilver completes, the added space will appear in the pool.

7.1.13. Hot Spares¶

ZFS provides the ability to have “hot” spares. These are drives that are connected to a volume, but not in use. If the volume experiences the failure of a data drive, the system uses the hot spare as a temporary replacement. If the failed drive is replaced with a new drive, the hot spare drive is no longer needed and reverts to being a hot spare. If the failed drive is instead removed from the volume, the spare is promoted to a full member of the volume.

Hot spares can be added to a volume during or after creation. On TrueNAS^®, hot spare actions are implemented by zfsd(8).

7.3.1. Examples: Common Configuration¶

The examples shown here use the same setup of source and destination computers.

7.3.1.1. Alpha (Source)¶

Alpha is the source computer with the data to be replicated. It is at IP address 10.0.0.102. A volume named alphavol has already been created, and a dataset named alphadata has been created on that volume. This dataset contains the files which will be snapshotted and replicated onto Beta.

This new dataset has been created for this example, but a new dataset is not required. Most users will already have datasets containing the data they wish to replicate.

Create a periodic snapshot of the source dataset by selecting Storage → Periodic Snapshot Tasks. Click the alphavol/alphadata dataset to highlight it. Create a periodic snapshot of it by clicking Periodic Snapshot Tasks, then Add Periodic Snapshot as shown in Figure 7.3.1.

This example creates a snapshot of the alphavol/alphadata dataset every two hours from Monday through Friday between the hours of 9:00 and 18:00 (6:00 PM). Snapshots are automatically deleted after their chosen lifetime of two weeks expires.

Fig. 7.3.1 Create a Periodic Snapshot for Replication

7.3.2. Example: TrueNAS^® to TrueNAS^® Semi-Automatic Setup¶

TrueNAS^® offers a special semi-automatic setup mode that simplifies setting up replication. Create the replication task on Alpha by clicking Replication Tasks and Add Replication. alphavol/alphadata is selected as the dataset to replicate. betavol is the destination volume where alphadata snapshots are replicated. The Setup mode dropdown is set to Semi-automatic as shown in Figure 7.3.2. The IP address of Beta is entered in the Remote hostname field. A hostname can be entered here if local DNS resolves for that hostname.

Fig. 7.3.2 Add Replication Dialog, Semi-Automatic

The Remote Auth Token field expects a special token from the Beta computer. On Beta, choose Storage → Replication Tasks, then click Temporary Auth Token. A dialog showing the temporary authorization token is shown as in Figure 7.3.3.

Highlight the temporary authorization token string with the mouse and copy it.

Fig. 7.3.3 Temporary Authentication Token on Destination

On the Alpha system, paste the copied temporary authorization token string into the Remote Auth Token field as shown in Figure 7.3.4.

Fig. 7.3.4 Temporary Authentication Token Pasted to Source

Finally, click the OK button to create the replication task. After each periodic snapshot is created, a replication task will copy it to the destination system. See Limiting Replication Times for information about restricting when replication is allowed to run.

7.3.3. Example: TrueNAS^® to TrueNAS^® Dedicated User Replication¶

A dedicated user can be used for replications rather than the root user. This example shows the process using the semi-automatic replication setup between two TrueNAS^® systems with a dedicated user named repluser. SSH key authentication is used to allow the user to log in remotely without a password.

In this example, the periodic snapshot task has not been created yet. If the periodic snapshot shown in the example configuration has already been created, go to Storage → Periodic Snapshot Tasks, click on the task to select it, and click Delete to remove it before continuing.

On Alpha, select Account → Users. Click the Add User. Enter repluser for Username, enter /mnt/alphavol/repluser in the Create Home Directory In field, enter Replication Dedicated User for the Full Name, and set the Disable password login checkbox. Leave the other fields at their default values, but note the User ID number. Click OK to create the user.

On Beta, the same dedicated user must be created as was created on the sending computer. Select Account → Users. Click the Add User. Enter the User ID number from Alpha, repluser for Username, enter /mnt/betavol/repluser in the Create Home Directory In field, enter Replication Dedicated User for the Full Name, and set the Disable password login checkbox. Leave the other fields at their default values. Click OK to create the user.

A dataset with the same name as the original must be created on the destination computer, Beta. Select Storage → Volumes, click on betavol, then click the Create Dataset icon at the bottom. Enter alphadata as the Dataset Name, then click Add Dataset.

The replication user must be given permissions to the destination dataset. Still on Beta, open a Shell and enter this command:

zfs allow -ldu repluser create,destroy,diff,mount,readonly,receive,release,send,userprop betavol/alphadata

The destination dataset must also be set to read-only. Enter this command in the Shell:

zfs set readonly=on betavol/alphadata

Close the Shell by typing exit and pressing Enter.

The replication user must also be able to mount datasets. Still on Beta, go to System → Tunables. Click Add Tunable. Enter vfs.usermount for the Variable, 1 for the Value, and choose Sysctl from the Type drop-down. Click OK to save the tunable settings.

Back on Alpha, create a periodic snapshot of the source dataset by selecting Storage → Periodic Snapshot Tasks. Click the alphavol/alphadata dataset to highlight it. Create a periodic snapshot of it by clicking Periodic Snapshot Tasks, then Add Periodic Snapshot as shown in Figure 7.3.1.

Still on Alpha, create the replication task by clicking Replication Tasks and Add Replication. alphavol/alphadata is selected as the dataset to replicate. betavol/alphadata is the destination volume and dataset where alphadata snapshots are replicated.

The Setup mode dropdown is set to Semi-automatic as shown in Figure 7.3.2. The IP address of Beta is entered in the Remote hostname field. A hostname can be entered here if local DNS resolves for that hostname.

The Remote Auth Token field expects a special token from the Beta computer. On Beta, choose Storage → Replication Tasks, then click Temporary Auth Token. A dialog showing the temporary authorization token is shown as in Figure 7.3.3.

Highlight the temporary authorization token string with the mouse and copy it.

On the Alpha system, paste the copied temporary authorization token string into the Remote Auth Token field as shown in Figure 7.3.4.

Set the Dedicated User checkbox. Choose repluser in the Dedicated User drop-down.

Click the OK button to create the replication task.

Replication will begin when the periodic snapshot task runs.

Additional replications can use the same dedicated user that has already been set up. The permissions and read only settings made through the Shell must be set on each new destination dataset.

7.3.4. Example: TrueNAS^® to TrueNAS^® or Other Systems, Manual Setup¶

This example uses the same basic configuration of source and destination computers shown above, but the destination computer is not required to be a TrueNAS^® system. Other operating systems can receive the replication if they support SSH, ZFS, and the same features that are in use on the source system. The details of creating volumes and datasets, enabling SSH, and copying encryption keys will vary when the destination computer is not a TrueNAS^® system.

7.3.4.1. Encryption Keys¶

A public encryption key must be copied from Alpha to Beta to allow a secure connection without a password prompt. On Alpha, select Storage → Replication Tasks → View Public Key, producing the window shown in Figure 7.3.5. Use the mouse to highlight the key data shown in the window, then copy it.

Fig. 7.3.5 Copy the Replication Key

On Beta, select Account → Users → View Users. Click the root account to select it, then click Modify User. Paste the copied key into the SSH Public Key field and click OK as shown in Figure 7.3.6.

Fig. 7.3.6 Paste the Replication Key

Back on Alpha, create the replication task by clicking Replication Tasks and Add Replication. alphavol/alphadata is selected as the dataset to replicate. The destination volume is betavol. The alphadata dataset and snapshots are replicated there. The IP address of Beta is entered in the Remote hostname field as shown in Figure 7.3.7. A hostname can be entered here if local DNS resolves for that hostname.

Click the SSH Key Scan button to retrieve the SSH host keys from Beta and fill the Remote hostkey field. Finally, click OK to create the replication task. After each periodic snapshot is created, a replication task will copy it to the destination system. See Limiting Replication Times for information about restricting when replication is allowed to run.

Fig. 7.3.7 Add Replication Dialog

7.3.5. Replication Options¶

Table 7.3.1 describes the options in the replication task dialog.

The replication task runs after a new periodic snapshot is created. The periodic snapshot and any new manual snapshots of the same dataset are replicated onto the destination computer.

When multiple replications have been created, replication tasks run serially, one after another. Completion time depends on the number and size of snapshots and the bandwidth available between the source and destination computers.

The first time a replication runs, it must duplicate data structures from the source to the destination computer. This can take much longer to complete than subsequent replications, which only send differences in data.

Selecting Storage → Replication Tasks displays Figure 7.3.8, the list of replication tasks. The Last snapshot sent to remote side column shows the name of the last snapshot that was successfully replicated, and Status shows the current status of each replication task. The display is updated every five seconds, always showing the latest status.

Fig. 7.3.8 Replication Task List

7.3.6. Replication Encryption¶

The default Encryption Cipher Standard setting provides good security. Fast is less secure than Standard but can give reasonable transfer rates for devices with limited cryptographic speed. For networks where the entire path between source and destination computers is trusted, the Disabled option can be chosen to send replicated data without encryption.

7.3.7. Limiting Replication Times¶

The Begin and End times in a replication task make it possible to restrict when replication is allowed. These times can be set to only allow replication after business hours, or at other times when disk or network activity will not slow down other operations like snapshots or Scrubs. The default settings allow replication to occur at any time.

These times control when replication task are allowed to start, but will not stop a replication task that is already running. Once a replication task has begun, it will run until finished.

7.3.8. Replication Topologies and Scenarios¶

The replication examples shown above are known as simple or A to B replication, where one machine replicates data to one other machine. Replication can also be set up in more sophisticated topologies to suit various purposes and needs.

7.3.9. Troubleshooting Replication¶

Replication depends on SSH, disks, network, compression, and encryption to work. A failure or misconfiguration of any of these can prevent successful replication.

ssh -vv -i /data/ssh/replication 10.0.0.118

No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

zfs send alphavol/alphadata@auto-20161206.1110-2w | ssh -i /data/ssh/replication 10.0.0.118 zfs recv betavol

zfs destroy -R betavol/alphadata@auto-20161206.1110-2w