26. Command Line Utilities — FreeNAS®11.1-U4 User Guide Table of Contents

# 26. Command Line Utilities

Several command line utilities which are provided with FreeNAS® are demonstrated in this section.

The following utilities can be used for benchmarking and performance testing:

- Iperf: used for measuring maximum TCP and UDP bandwidth performance
- Netperf: a tool for measuring network performance
- IOzone: filesystem benchmark utility used to perform a broad filesystem analysis
- arcstat: used to gather ZFS ARC statistics

The following utilities are specific to RAID controllers:

- tw_cli:_used to monitor and maintain 3ware RAID controllers
- MegaCli: used to configure and manage Broadcom MegaRAID SAS family of RAID controllers

This section also describes these utilities:

- freenas-debug: the backend used to dump FreeNAS® debugging information
- tmux: a terminal multiplexer similar to GNU screen
- Dmidecode: reports information about system hardware as described in the system’s BIOS

## 26.1. Iperf

Iperf is a utility for measuring maximum TCP and UDP bandwidth performance. It can be used to chart network throughput over time. For example, you can use it to test the speed of different types of shares to determine which type best performs on your network.

FreeNAS® includes the Iperf server. To perform network testing, you will need to install an Iperf client on a desktop system that has network access to the FreeNAS® system. This section will demonstrate how to use the xjperf GUI client as it works on Windows, Mac OS X, Linux, and BSD systems.

Since this client is Java-based, the appropriate JRE must be installed on the client computer.

Linux and BSD users will need to install the iperf package using their operating system’s package management system.

To start xjperf on Windows: unzip the downloaded file, start Command Prompt in Run as administrator mode, cd to the unzipped folder, and run jperf.bat.

To start xjperf on Mac OS X, Linux, or BSD, unzip the downloaded file, cd to the unzipped directory, type chmod u+x jperf.sh, and run ./jperf.sh.

Once the client is ready, you need to start the Iperf server on FreeNAS®.

Note

Beginning with FreeNAS® version 11.1, both iperf2 and iperf3 are pre-installed. To use iperf2, use iperf. To use iperf3, instead type iperf3. The examples below are for iperf2.

To see the available server options, open Shell and type:

    iperf --help | more

or:

    iperf3 --help | more

For example, to perform a TCP test and start the server in daemon mode (to get the prompt back), type:

    iperf -sD
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 64.0 KByte (default)
    ------------------------------------------------------------
    Running Iperf Server as a daemon
    The Iperf daemon process ID: 4842

Note

If you close Shell, the daemon process will stop. Have your environment set up (e.g. shares configured and started) before starting the iperf process.

From your desktop, open the client. Enter the IP of address of the FreeNAS® system, specify the running time for the test under `Application layer options → Transmit` (the default test time is 10 seconds), and click the Run Iperf! button. Figure 26.1.1 shows an example of the client running on a Windows system while an SFTP transfer is occurring on the network.

Fig. 26.1.1 Viewing Bandwidth Statistics Using xjperf

Depending upon the traffic being tested (e.g. the type of share running on your network), you may need to test UDP instead of TCP. To start the iperf server in UDP mode, use iperf -sDu as the u specifies UDP; the startup message should indicate that the server is listening for UDP datagrams. If you are not sure if the traffic that you wish to test is UDP or TCP, run this command to determine which services are running on the FreeNAS® system:

    sockstat -4 | more
    USER     COMMAND PID     FD PROTO        LOCAL ADDRESS   FOREIGN ADDRESS
    root     iperf   4870    6  udp4         *:5001          *:*
    root     iperf   4842    6  tcp4         *:5001          *:*
    www      nginx   4827    3  tcp4         127.0.0.1:15956 127.0.0.1:9042
    www      nginx   4827    5  tcp4         192.168.2.11:80 192.168.2.26:56964
    www      nginx   4827    7  tcp4         *:80            *:*
    root     sshd    3852    5  tcp4         *:22            *:*
    root     python  2503    5  udp4         *:*             *:*
    root     mountd  2363    7  udp4         *:812           *:*
    root     mountd  2363    8  tcp4         *:812           *:*
    root     rpcbind 2359    9  udp4         *:111           *:*
    root     rpcbind 2359    10 udp4         *:886           *:*
    root     rpcbind 2359    11 tcp4         *:111           *:*
    root     nginx   2044    7  tcp4         *:80            *:*
    root     python  2029    3  udp4         *:*             *:*
    root     python  2029    4  tcp4         127.0.0.1:9042  *:*
    root     python  2029    7  tcp4         127.0.0.1:9042  127.0.0.1:15956
    root     ntpd    1548    20 udp4         *:123           *:*
    root     ntpd    1548    22 udp4         192.168.2.11:123*:*
    root     ntpd    1548    25 udp4         127.0.0.1:123   *:*
    root     syslogd 1089    6  udp4         127.0.0.1:514   *:*

When you are finished testing, either type killall iperf or close Shell to terminate the iperf server process.

## 26.2. Netperf

Netperf is a benchmarking utility that can be used to measure the performance of unidirectional throughput and end-to-end latency.

Before you can use the netperf command, you must start its server process using this command:

    netserver
    Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC

The following command will display the available options for performing tests with the netperf command. The Netperf Manual describes each option in more detail and explains how to perform many types of tests. It is the best reference for understanding how each test works and how to interpret your results. When you are finished with your tests, type killall netserver to stop the server process.

    netperf -h |more
    Usage: netperf [global options] -- [test options]
    Global options:
        -a send,recv       Set the local send,recv buffer alignment
        -A send,recv       Set the remote send,recv buffer alignment
        -B brandstr        Specify a string to be emitted with brief output
        -c [cpu_rate]      Report local CPU usage
        -C [cpu_rate]      Report remote CPU usage
        -d                 Increase debugging output
        -D [secs,units] *  Display interim results at least every secs seconds
                           using units as the initial guess for units per second
        -f G|M|K|g|m|k     Set the output units
        -F fill_file       Pre-fill buffers with data from fill_file
        -h                 Display this text
        -H name|ip,fam *   Specify the target machine and/or local ip and family
        -i max,min         Specify the max and min number of iterations (15,1)
        -I lvl[,intvl]     Specify confidence level (95 or 99) (99)
                           and confidence interval in percentage (10)
        -j                 Keep additional timing statistics
        -l testlen         Specify test duration (>0 secs) (<0 bytes|trans)
        -L name|ip,fam *   Specify the local ip|name and address family
        -o send,recv       Set the local send,recv buffer offsets
        -O send,recv       Set the remote send,recv buffer offset
        -n numcpu          Set the number of processors for CPU util
        -N                 Establish no control connection, do 'send' side only
        -p port,lport*     Specify netserver port number and/or local port
        -P 0|1             Don't/Do display test headers
        -r                 Allow confidence to be hit on result only
        -s seconds         Wait seconds between test setup and test start
        -S                 Set SO_KEEPALIVE on the data connection
        -t testname        Specify test to perform
        -T lcpu,rcpu       Request netperf/netserver be bound to local/remote cpu
        -v verbosity       Specify the verbosity level
        -W send,recv       Set the number of send,recv buffers
        -v level           Set the verbosity level (default 1, min 0)
        -V                 Display the netperf version and exit

For those options taking two parms, at least one must be specified; specifying one value without a comma will set both parms to that value, specifying a value with a leading comma will set just the second parm, a value with a trailing comma will set just the first. To set each parm to unique values, specify both and separate them with a comma.

For these options taking two parms, specifying one value with no comma will only set the first parms and will leave the second at the default value. To set the second value it must be preceded with a comma or be a comma-separated pair. This is to retain previous netperf behaviour.

## 26.3. IOzone

IOzone is a disk and filesystem benchmarking tool. It can be used to test file I/O performance for the following operations: read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, mmap, aio_read, and aio_write.

FreeNAS® ships with IOzone, meaning that it can be run from Shell. When using IOzone on FreeNAS®, cd to a directory in a volume that you have permission to write to, otherwise you will get an error about being unable to write the temporary file.

Before using IOzone, read through the IOzone documentation PDF as it describes the tests, the many command line switches, and how to interpret your results.

If you have never used this tool before, these resources provide good starting points on which tests to run, when to run them, and how to interpret the results:

- How To Measure Linux Filesystem I/O Performance With iozone
- Analyzing NFS Client Performance with IOzone
- 10 iozone Examples for Disk I/O Performance Measurement on Linux

You can receive a summary of the available switches by typing the following command. As you can see from the number of options, IOzone is comprehensive and it may take some time to learn how to use the tests effectively.

Starting with version 9.2.1, FreeNAS® enables compression on newly created ZFS pools by default. Since IOzone creates test data that is compressible, this can skew test results. To configure IOzone to generate incompressible test data, include the options -+w 1 -+y 1 -+C 1.

Alternatively, consider temporarily disabling compression on the ZFS pool or dataset when running IOzone benchmarks.

Note

If you prefer to visualize the collected data, scripts are available to render IOzone’s output in Gnuplot.

    iozone -h | more
    iozone: help mode
    Usage: iozone[-s filesize_Kb] [-r record_size_Kb] [-f [path]filename] [-h]
                 [-i test] [-E] [-p] [-a] [-A] [-z] [-Z] [-m] [-M] [-t children]
                 [-l min_number_procs] [-u max_number_procs] [-v] [-R] [-x] [-o]
                 [-d microseconds] [-F path1 path2...] [-V pattern] [-j stride]
                 [-T] [-C] [-B] [-D] [-G] [-I] [-H depth] [-k depth] [-U mount_point]
                 [-S cache_size] [-O] [-L cacheline_size] [-K] [-g maxfilesize_Kb]
                 [-n minfilesize_Kb] [-N] [-Q] [-P start_cpu] [-e] [-c] [-b Excel.xls]
                 [-J milliseconds] [-X write_telemetry_filename] [-w] [-W]
                 [-Y read_telemetry_filename] [-y minrecsize_Kb] [-q maxrecsize_Kb]
                 [-+u] [-+m cluster_filename] [-+d] [-+x multiplier] [-+p # ]
                 [-+r] [-+t] [-+X] [-+Z] [-+w percent dedupable] [-+y percent_interior_dedup]
                 [-+C percent_dedup_within]
             -a  Auto mode
             -A  Auto2 mode
             -b Filename  Create Excel worksheet file
             -B  Use mmap() files
             -c  Include close in the timing calculations
             -C  Show bytes transferred by each child in throughput testing
             -d #  Microsecond delay out of barrier
             -D  Use msync(MS_ASYNC) on mmap files
             -e  Include flush (fsync,fflush) in the timing calculations
             -E  Run extension tests
             -f  filename to use
             -F  filenames for each process/thread in throughput test
             -g #  Set maximum file size (in Kbytes) for auto mode (or #m or #g)
             -G  Use msync(MS_SYNC) on mmap files
             -h  help
             -H #  Use POSIX async I/O with # async operations
             -i #  Test to run (0=write/rewrite, 1=read/re-read, 2=random-read/write
                   3=Read-backwards, 4=Re-write-record, 5=stride-read, 6=fwrite/re-fwrite
                   7=fread/Re-fread, 8=random_mix, 9=pwrite/Re-pwrite, 10=pread/Re-pread
                   11=pwritev/Re-pwritev, 12=preadv/Re-preadv)
             -I  Use VxFS VX_DIRECT, O_DIRECT,or O_DIRECTIO for all file operations
             -j #  Set stride of file accesses to (# * record size)
             -J #  milliseconds of compute cycle before each I/O operation
             -k #  Use POSIX async I/O (no bcopy) with # async operations
             -K  Create jitter in the access pattern for readers
             -l #  Lower limit on number of processes to run
             -L #  Set processor cache line size to value (in bytes)
             -m  Use multiple buffers
             -M  Report uname -a output
             -n #  Set minimum file size (in Kbytes) for auto mode (or #m or #g)
             -N  Report results in microseconds per operation
             -o  Writes are synch (O_SYNC)
             -O  Give results in ops/sec.
             -p  Purge on
             -P #  Bind processes/threads to processors, starting with this cpu
             -q #  Set maximum record size (in Kbytes) for auto mode (or #m or #g)
             -Q  Create offset/latency files
             -r #  record size in Kb
                or -r #k .. size in Kb
                or -r #m .. size in Mb
                or -r #g .. size in Gb
             -R  Generate Excel report
             -s #  file size in Kb
                or -s #k .. size in Kb
                or -s #m .. size in Mb
                or -s #g .. size in Gb
             -S #  Set processor cache size to value (in Kbytes)
             -t #  Number of threads or processes to use in throughput test
             -T  Use POSIX pthreads for throughput tests
             -u #  Upper limit on number of processes to run
             -U  Mount point to remount between tests
             -v  version information
             -V #  Verify data pattern write/read
             -w  Do not unlink temporary file
             -W  Lock file when reading or writing
             -x  Turn off stone-walling
             -X filename  Write telemetry file. Contains lines with (offset reclen compute_time) in ascii
             -y #  Set minimum record size (in Kbytes) for auto mode (or #m or #g)
             -Y filename  Read telemetry file. Contains lines with (offset reclen compute_time) in ascii
             -z  Used in conjunction with -a to test all possible record sizes
             -Z  Enable mixing of mmap I/O and file I/O
             -+E Use existing non-Iozone file for read-only testing
             -+K Sony special. Manual control of test 8.
             -+m Cluster_filename  Enable Cluster testing
             -+d File I/O diagnostic mode. (To troubleshoot a broken file I/O subsystem)
             -+u Enable CPU utilization output (Experimental)
             -+x # Multiplier to use for incrementing file and record sizes
             -+p # Percentage of mix to be reads
             -+r Enable O_RSYNC|O_SYNC for all testing.
             -+t Enable network performance test. Requires -+m
             -+n No retests selected.
             -+k Use constant aggregate data set size.
             -+q Delay in seconds between tests.
             -+l Enable record locking mode.
             -+L Enable record locking mode, with shared file.
             -+B Sequential mixed workload.
             -+A # Enable madvise. 0 = normal, 1=random, 2=sequential 3=dontneed, 4=willneed
             -+N Do not truncate existing files on sequential writes.
             -+S # Dedup-able data is limited to sharing within each numerically identified file set
             -+V Enable shared file. No locking.
             -+X Enable short circuit mode for filesystem testing ONLY
                 ALL Results are NOT valid in this mode.
             -+Z Enable old data set compatibility mode. WARNING.. Published
                 hacks may invalidate these results and generate bogus, high values for results.
             -+w ## Percent of dedup-able data in buffers.
             -+y ## Percent of dedup-able within & across files in buffers.
             -+C ## Percent of dedup-able within & not across files in buffers.
             -+H Hostname  Hostname of the PIT server.
             -+P Service  Service of the PIT server.
             -+z Enable latency histogram logging.

## 26.4. arcstat

Arcstat is a script that prints out ZFS ARC statistics. Originally it was a perl script created by Sun. That perl script was ported to FreeBSD and was then ported as a Python script for use on FreeNAS®.

Watching ARC hits/misses and percentages will provide an indication of how well your ZFS pool is fetching from the ARC rather than using disk I/O. Ideally, you want as many things fetching from cache as possible. Keep your load in mind as you review the stats. For random reads, expect a miss and having to go to disk to fetch the data. For cached reads, expect it to pull out of the cache and have a hit.

Like all cache systems, the ARC takes time to fill with data. This means that it will have a lot of misses until the pool has been in use for a while. If there continues to be lots of misses and high disk I/O on cached reads, there is cause to investigate further and tune the system.

The FreeBSD ZFS Tuning Guide provides some suggestions for commonly tuned sysctl values. It should be noted that performance tuning is more of an art than a science and that any changes you make will probably require several iterations of tune and test. Be aware that what needs to be tuned will vary depending upon the type of workload and that what works for one person’s network may not benefit yours.

In particular, the value of pre-fetching depends upon the amount of memory and the type of workload, as seen in these two examples:

- Understanding ZFS: Prefetch
- ZFS prefetch algorithm can cause performance drawbacks

FreeNAS® provides two command line scripts which can be manually run from Shell:

- arc_summary.py: provides a summary of the statistics
- arcstat.py: used to watch the statistics in real time

The advantage of these scripts is that they can be used to provide real time (right now) information, whereas the current GUI reporting mechanism is designed to only provide graphs charted over time.

This forum post demonstrates some examples of using these scripts with hints on how to interpret the results.

To view the help for arcstat.py:

    arcstat.py -h
    Usage: arcstat [-hvx] [-f fields] [-o file] [-s string] [interval [count]]
    -h: Print this help message
    -v: List all possible field headers and definitions
    -x: Print extended stats
    -f: Specify specific fields to print (see -v)
    -o: Redirect output to the specified file
    -s: Override default field separator with custom character or string

    Examples:
    arcstat -o /tmp/a.log 2 10
    arcstat -s "," -o /tmp/a.log 2 10
    arcstat -v
    arcstat -f time,hit%,dh%,ph%,mh% 1

To view ARC statistics in real time, specify an interval and a count. This command will display every 1 second for a count of five.

    arcstat.py 1 5
        time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
    06:19:03     7     0      0     0    0     0    0     0    0   153M  6.6G
    06:19:04   257     0      0     0    0     0    0     0    0   153M  6.6G
    06:19:05   193     0      0     0    0     0    0     0    0   153M  6.6G
    06:19:06   193     0      0     0    0     0    0     0    0   153M  6.6G
    06:19:07   255     0      0     0    0     0    0     0    0   153M  6.6G

Table 26.4.1 briefly describes the columns in the output.

Table 26.4.1 arcstat Column Descriptions
Column | Description
--- | ---
read | total ARC accesses/second
miss | ARC misses/second
miss% | ARC miss percentage
dmis | demand data misses/second
dm% | demand data miss percentage
pmis | prefetch misses per second
pm% | prefetch miss percentage
mmis | metadata misses/second
mm% | metadata miss percentage
arcsz | arc size
c | arc target size

To receive a summary of statistics, use:

    arcsummary.py
    System Memory:
           2.36%   93.40   MiB Active,     8.95%   353.43  MiB Inact
           8.38%   330.89  MiB Wired,      0.15%   5.90    MiB Cache
           80.16%  3.09    GiB Free,       0.00%   0       Bytes Gap
           Real Installed:                         4.00    GiB
           Real Available:                 99.31%  3.97    GiB
           Real Managed:                   97.10%  3.86    GiB
           Logical Total:                          4.00    GiB
           Logical Used:                   13.93%  570.77  MiB
           Logical Free:                   86.07%  3.44    GiB
    Kernel Memory:                                 87.62   MiB
           Data:                           69.91%  61.25   MiB
           Text:                           30.09%  26.37   MiB
    Kernel Memory Map:                             3.86    GiB
           Size:                           5.11%   201.70  MiB
           Free:                           94.89%  3.66    GiB
    ARC Summary: (HEALTHY)
           Storage pool Version:                   5000
           Filesystem Version:                     5
           Memory Throttle Count:                  0
    ARC Misc:
           Deleted:                                8
           Mutex Misses:                           0
           Evict Skips:                            0
    ARC Size:                               5.83%   170.45  MiB
           Target Size: (Adaptive)         100.00% 2.86    GiB
           Min Size (Hard Limit):          12.50%  365.69  MiB
           Max Size (High Water):          8:1     2.86    GiB
    ARC Size Breakdown:
           Recently Used Cache Size:       50.00%  1.43    GiB
           Frequently Used Cache Size:     50.00%  1.43    GiB
    ARC Hash Breakdown:
           Elements Max:                           5.90k
           Elements Current:               100.00% 5.90k
           Collisions:                             72
           Chain Max:                              1
           Chains:                                 23
    ARC Total accesses:                                    954.06k
           Cache Hit Ratio:                99.18%  946.25k
           Cache Miss Ratio:               0.82%   7.81k
           Actual Hit Ratio:               98.84%  943.00k
           Data Demand Efficiency:         99.20%  458.77k
           CACHE HITS BY CACHE LIST:
             Anonymously Used:             0.34%   3.25k
             Most Recently Used:           3.73%   35.33k
             Most Frequently Used:         95.92%  907.67k
             Most Recently Used Ghost:     0.00%   0
             Most Frequently Used Ghost:   0.00%   0
           CACHE HITS BY DATA TYPE:
             Demand Data:                  48.10%  455.10k
             Prefetch Data:                0.00%   0
             Demand Metadata:              51.56%  487.90k
             Prefetch Metadata:            0.34%   3.25k
           CACHE MISSES BY DATA TYPE:
             Demand Data:                  46.93%  3.66k
             Prefetch Data:                0.00%   0
             Demand Metadata:              49.76%  3.88k
             Prefetch Metadata:            3.30%   258
    ZFS Tunable (sysctl):
           kern.maxusers                           590
           vm.kmem_size                            4141375488
           vm.kmem_size_scale                      1
           vm.kmem_size_min                        0
           vm.kmem_size_max                        1319413950874
           vfs.zfs.vol.unmap_enabled               1
           vfs.zfs.vol.mode                        2
           vfs.zfs.sync_pass_rewrite               2
           vfs.zfs.sync_pass_dont_compress         5
           vfs.zfs.sync_pass_deferred_free         2
           vfs.zfs.zio.exclude_metadata            0
           vfs.zfs.zio.use_uma                     1
           vfs.zfs.cache_flush_disable             0
           vfs.zfs.zil_replay_disable              0
           vfs.zfs.version.zpl                     5
           vfs.zfs.version.spa                     5000
           vfs.zfs.version.acl                     1
           vfs.zfs.version.ioctl                   5
           vfs.zfs.debug                           0
           vfs.zfs.super_owner                     0
           vfs.zfs.min_auto_ashift                 9
           vfs.zfs.max_auto_ashift                 13
           vfs.zfs.vdev.write_gap_limit            4096
           vfs.zfs.vdev.read_gap_limit             32768
           vfs.zfs.vdev.aggregation_limit          131072
           vfs.zfs.vdev.trim_max_active            64
           vfs.zfs.vdev.trim_min_active            1
           vfs.zfs.vdev.scrub_max_active           2
           vfs.zfs.vdev.scrub_min_active           1
           vfs.zfs.vdev.async_write_max_active     10
           vfs.zfs.vdev.async_write_min_active     1
           vfs.zfs.vdev.async_read_max_active      3
           vfs.zfs.vdev.async_read_min_active      1
           vfs.zfs.vdev.sync_write_max_active      10
           vfs.zfs.vdev.sync_write_min_active      10
           vfs.zfs.vdev.sync_read_max_active       10
           vfs.zfs.vdev.sync_read_min_active       10
           vfs.zfs.vdev.max_active                 1000
           vfs.zfs.vdev.async_write_active_max_dirty_percent60
           vfs.zfs.vdev.async_write_active_min_dirty_percent30
           vfs.zfs.vdev.mirror.non_rotating_seek_inc1
           vfs.zfs.vdev.mirror.non_rotating_inc    0
           vfs.zfs.vdev.mirror.rotating_seek_offset1048576
           vfs.zfs.vdev.mirror.rotating_seek_inc   5
           vfs.zfs.vdev.mirror.rotating_inc        0
           vfs.zfs.vdev.trim_on_init               1
           vfs.zfs.vdev.larger_ashift_minimal      0
           vfs.zfs.vdev.bio_delete_disable         0
           vfs.zfs.vdev.bio_flush_disable          0
           vfs.zfs.vdev.cache.bshift               16
           vfs.zfs.vdev.cache.size                 0
           vfs.zfs.vdev.cache.max                  16384
           vfs.zfs.vdev.metaslabs_per_vdev         200
           vfs.zfs.vdev.trim_max_pending           10000
           vfs.zfs.txg.timeout                     5
           vfs.zfs.trim.enabled                    1
           vfs.zfs.trim.max_interval               1
           vfs.zfs.trim.timeout                    30
           vfs.zfs.trim.txg_delay                  32
           vfs.zfs.space_map_blksz                 4096
           vfs.zfs.spa_slop_shift                  5
           vfs.zfs.spa_asize_inflation             24
           vfs.zfs.deadman_enabled                 1
           vfs.zfs.deadman_checktime_ms            5000
           vfs.zfs.deadman_synctime_ms             1000000
           vfs.zfs.recover                         0
           vfs.zfs.spa_load_verify_data            1
           vfs.zfs.spa_load_verify_metadata        1
           vfs.zfs.spa_load_verify_maxinflight     10000
           vfs.zfs.check_hostid                    1
           vfs.zfs.mg_fragmentation_threshold      85
           vfs.zfs.mg_noalloc_threshold            0
           vfs.zfs.condense_pct                    200
           vfs.zfs.metaslab.bias_enabled           1
           vfs.zfs.metaslab.lba_weighting_enabled  1
           vfs.zfs.metaslab.fragmentation_factor_enabled1
           vfs.zfs.metaslab.preload_enabled        1
           vfs.zfs.metaslab.preload_limit          3
           vfs.zfs.metaslab.unload_delay           8
           vfs.zfs.metaslab.load_pct               50
           vfs.zfs.metaslab.min_alloc_size         33554432
           vfs.zfs.metaslab.df_free_pct            4
           vfs.zfs.metaslab.df_alloc_threshold     131072
           vfs.zfs.metaslab.debug_unload           0
           vfs.zfs.metaslab.debug_load             0
           vfs.zfs.metaslab.fragmentation_threshold70
           vfs.zfs.metaslab.gang_bang              16777217
           vfs.zfs.free_bpobj_enabled              1
           vfs.zfs.free_max_blocks                 18446744073709551615
           vfs.zfs.no_scrub_prefetch               0
           vfs.zfs.no_scrub_io                     0
           vfs.zfs.resilver_min_time_ms            3000
           vfs.zfs.free_min_time_ms                1000
           vfs.zfs.scan_min_time_ms                1000
           vfs.zfs.scan_idle                       50
           vfs.zfs.scrub_delay                     4
           vfs.zfs.resilver_delay                  2
           vfs.zfs.top_maxinflight                 32
           vfs.zfs.delay_scale                     500000
           vfs.zfs.delay_min_dirty_percent         60
           vfs.zfs.dirty_data_sync                 67108864
           vfs.zfs.dirty_data_max_percent          10
           vfs.zfs.dirty_data_max_max              4294967296
           vfs.zfs.dirty_data_max                  426512793
           vfs.zfs.max_recordsize                  1048576
           vfs.zfs.zfetch.array_rd_sz              1048576
           vfs.zfs.zfetch.max_distance             8388608
           vfs.zfs.zfetch.min_sec_reap             2
           vfs.zfs.zfetch.max_streams              8
           vfs.zfs.prefetch_disable                1
           vfs.zfs.mdcomp_disable                  0
           vfs.zfs.nopwrite_enabled                1
           vfs.zfs.dedup.prefetch                  1
           vfs.zfs.l2c_only_size                   0
           vfs.zfs.mfu_ghost_data_lsize            0
           vfs.zfs.mfu_ghost_metadata_lsize        0
           vfs.zfs.mfu_ghost_size                  0
           vfs.zfs.mfu_data_lsize                  26300416
           vfs.zfs.mfu_metadata_lsize              1780736
           vfs.zfs.mfu_size                        29428736
           vfs.zfs.mru_ghost_data_lsize            0
           vfs.zfs.mru_ghost_metadata_lsize        0
           vfs.zfs.mru_ghost_size                  0
           vfs.zfs.mru_data_lsize                  122090496
           vfs.zfs.mru_metadata_lsize              2235904
           vfs.zfs.mru_size                        139389440
           vfs.zfs.anon_data_lsize                 0
           vfs.zfs.anon_metadata_lsize             0
           vfs.zfs.anon_size                       163840
           vfs.zfs.l2arc_norw                      1
           vfs.zfs.l2arc_feed_again                1
           vfs.zfs.l2arc_noprefetch                1
           vfs.zfs.l2arc_feed_min_ms               200
           vfs.zfs.l2arc_feed_secs                 1
           vfs.zfs.l2arc_headroom                  2
           vfs.zfs.l2arc_write_boost               8388608
           vfs.zfs.l2arc_write_max                 8388608
           vfs.zfs.arc_meta_limit                  766908416
           vfs.zfs.arc_free_target                 7062
           vfs.zfs.arc_shrink_shift                7
           vfs.zfs.arc_average_blocksize           8192
           vfs.zfs.arc_min                         383454208
           vfs.zfs.arc_max                         3067633664

When reading the tunable values, 0 means no, 1 typically means yes, and any other number represents a value. To receive a brief description of a “sysctl” value, use sysctl -d. For example:

    sysctl -d vfs.zfs.zio.use_uma
    vfs.zfs.zio.use_uma: Use uma(9) for ZIO allocations

The ZFS tunables require a fair understanding of how ZFS works, meaning that you will be reading man pages and searching for the meaning of acronyms you are unfamiliar with. Do not change a tunable’s value without researching it first. If the tunable takes a numeric value (rather than 0 for no or 1 for yes), do not make one up. Instead, research examples of beneficial values that match your workload.

If you decide to change any of the ZFS tunables, continue to monitor the system to determine the effect of the change. It is recommended that you test your changes first at the command line using sysctl. For example, to disable pre-fetch (i.e. change disable to 1 or yes):

    sysctl vfs.zfs.prefetch_disable=1
    vfs.zfs.prefetch_disable: 0 -> 1

The output will indicate the old value followed by the new value. If the change is not beneficial, change it back to the original value. If the change turns out to be beneficial, you can make it permanent by creating a sysctl using the instructions in Tunables.

## 26.5. tw_cli

FreeNAS® includes the tw_cli command line utility for providing controller, logical unit, and drive management for AMCC/3ware ATA RAID Controllers. The supported models are listed in the man pages for the twe(4) and twa(4) drivers.

Before using this command, read its man page as it describes the terminology and provides some usage examples.

If you type tw_cli in Shell, the prompt will change, indicating that you have entered interactive mode where you can run all sorts of maintenance commands on the controller and its arrays.

Alternately, you can specify one command to run. For example, to view the disks in the array:

    tw_cli /c0 show
    Unit   UnitType        Status  %RCmpl  %V/I/M  Stripe  Size(GB)        Cache   AVrfy
    ------------------------------------------------------------------------------
    u0     RAID-6          OK      -       -       256K    5587.88         RiW     ON
    u1     SPARE           OK      -       -       -       931.505         -       OFF
    u2     RAID-10         OK      -       -       256K    1862.62         RiW     ON

    VPort Status   Unit    Size            Type    Phy Encl-Slot   Model
    ------------------------------------------------------------------------------
    p8     OK      u0      931.51 GB SAS   -       /c0/e0/slt0     SEAGATE ST31000640SS
    p9     OK      u0      931.51 GB SAS   -       /c0/e0/slt1     SEAGATE ST31000640SS
    p10    OK      u0      931.51 GB SAS   -       /c0/e0/slt2     SEAGATE ST31000640SS
    p11    OK      u0      931.51 GB SAS   -       /c0/e0/slt3     SEAGATE ST31000640SS
    p12    OK      u0      931.51 GB SAS   -       /c0/e0/slt4     SEAGATE ST31000640SS
    p13    OK      u0      931.51 GB SAS   -       /c0/e0/slt5     SEAGATE ST31000640SS
    p14    OK      u0      931.51 GB SAS   -       /c0/e0/slt6     SEAGATE ST31000640SS
    p15    OK      u0      931.51 GB SAS   -       /c0/e0/slt7     SEAGATE ST31000640SS
    p16    OK      u1      931.51 GB SAS   -       /c0/e0/slt8     SEAGATE ST31000640SS
    p17    OK      u2      931.51 GB SATA  -       /c0/e0/slt9     ST31000340NS
    p18    OK      u2      931.51 GB SATA  -       /c0/e0/slt10    ST31000340NS
    p19    OK      u2      931.51 GB SATA  -       /c0/e0/slt11    ST31000340NS
    p20    OK      u2      931.51 GB SATA  -       /c0/e0/slt15    ST31000340NS

    Name   OnlineState     BBUReady        Status  Volt    Temp    Hours   LastCapTest
    ---------------------------------------------------------------------------
    bbu    On              Yes             OK      OK      OK      212     03-Jan-2012

Or, to review the event log:

    tw_cli /c0 show events
    Ctl    Date                            Severity        AEN Message
    ------------------------------------------------------------------------------
    c0     [Thu Feb 23 2012 14:01:15]      INFO            Battery charging started
    c0     [Thu Feb 23 2012 14:03:02]      INFO            Battery charging completed
    c0     [Sat Feb 25 2012 00:02:18]      INFO            Verify started: unit=0
    c0     [Sat Feb 25 2012 00:02:18]      INFO            Verify started: unit=2,subunit=0
    c0     [Sat Feb 25 2012 00:02:18]      INFO            Verify started: unit=2,subunit=1
    c0     [Sat Feb 25 2012 03:49:35]      INFO            Verify completed: unit=2,subunit=0
    c0     [Sat Feb 25 2012 03:51:39]      INFO            Verify completed: unit=2,subunit=1
    c0     [Sat Feb 25 2012 21:55:59]      INFO            Verify completed: unit=0
    c0     [Thu Mar 01 2012 13:51:09]      INFO            Battery health check started
    c0     [Thu Mar 01 2012 13:51:09]      INFO            Battery health check completed
    c0     [Thu Mar 01 2012 13:51:09]      INFO            Battery charging started
    c0     [Thu Mar 01 2012 13:53:03]      INFO            Battery charging completed
    c0     [Sat Mar 03 2012 00:01:24]      INFO            Verify started: unit=0
    c0     [Sat Mar 03 2012 00:01:24]      INFO            Verify started: unit=2,subunit=0
    c0     [Sat Mar 03 2012 00:01:24]      INFO            Verify started: unit=2,subunit=1
    c0     [Sat Mar 03 2012 04:04:27]      INFO            Verify completed: unit=2,subunit=0
    c0     [Sat Mar 03 2012 04:06:25]      INFO            Verify completed: unit=2,subunit=1
    c0     [Sat Mar 03 2012 16:22:05]      INFO            Verify completed: unit=0
    c0     [Thu Mar 08 2012 13:41:39]      INFO            Battery charging started
    c0     [Thu Mar 08 2012 13:43:42]      INFO            Battery charging completed
    c0     [Sat Mar 10 2012 00:01:30]      INFO            Verify started: unit=0
    c0     [Sat Mar 10 2012 00:01:30]      INFO            Verify started: unit=2,subunit=0
    c0     [Sat Mar 10 2012 00:01:30]      INFO            Verify started: unit=2,subunit=1
    c0     [Sat Mar 10 2012 05:06:38]      INFO            Verify completed: unit=2,subunit=0
    c0     [Sat Mar 10 2012 05:08:57]      INFO            Verify completed: unit=2,subunit=1
    c0     [Sat Mar 10 2012 15:58:15]      INFO            Verify completed: unit=0

If you add some disks to the array and they are not showing up in the GUI, try running this command:

    tw_cli /c0 rescan

Use the drives to create units and export them to the operating system. When finished, run camcontrol rescan all and they should now be available in the FreeNAS® GUI.

This forum post contains a handy wrapper script that will notify you of errors.

## 26.6. MegaCli

MegaCli is the command line interface for the Broadcom :MegaRAID SAS family of RAID controllers. FreeNAS® also includes the mfiutil(8) utility which can be used to configure and manage connected storage devices.

The MegaCli command is quite complex with several dozen options. The commands demonstrated in the Emergency Cheat Sheet can get you started.

## 26.7. freenas-debug

The FreeNAS® GUI provides an option to save debugging information to a text file using `System → Advanced → Save Debug`. This debugging information is created by the freenas-debug command line utility and a copy of the information is saved to `/var/tmp/fndebug`.

This command can be run manually from Shell to gather specific debugging information. To see a usage explanation listing all options, run the command without any options:

    freenas-debug
    Usage: /usr/local/bin/freenas-debug <options>
    Where options are:
        -e  Email debug log to this comma-delimited list of email addresses
        -A  Dump all debug information

        -a  Dump Active Directory Configuration
        -f  Dump AFP Configuration
        -c  Dump (AD|LDAP) Cache
        -B  Dump System Configuration Database
        -D  Dump Domain Controller Configuration
        -d  Dump DTrace Scripts
        -g  Dump GEOM Configuration
        -G  Dump Grub Configuration
        -h  Dump Hardware Configuration
        -I  Dump IPMI Configuration
        -i  Dump iSCSI Configuration
        -j  Dump Jail Information
        -l  Dump LDAP Configuration
        -T  Loader Configuration Information
        -n  Dump Network Configuration
        -N  Dump NFS Configuration
        -S  Dump SMART Information
        -C  Dump SMB Configuration
        -s  Dump SSL Configuration
        -y  Dump Sysctl Configuration
        -t  Dump System Information
        -v  Dump Boot System File Verification Status and Inconsistencies
        -z  Dump ZFS Configuration

Individual tests can be run alone. For example, when troubleshooting an Active Directory configuration, use:

    freenas-debug -a

To collect the output of every module, use `-A`:

    freenas-debug -A

## 26.8. tmux

tmux is a terminal multiplexer which enables a number of :terminals to be created, accessed, and controlled from a single :screen. tmux is an alternative to GNU screen. Similar to screen, tmux can be detached from a screen and continue running in the background, then later reattached. Unlike Shell, tmux allows you to have access to a command prompt while still providing access to the graphical administration screens.

To start a session, simply type tmux. As seen in Figure 26.8.1, a new session with a single window opens with a status line at the bottom of the screen. This line shows information on the current session and is used to enter interactive commands.

Fig. 26.8.1 tmux Session

To create a second window, press `Ctrl+b` then `"`. To close a window, type exit within the window.

tmux(1) lists all of the key bindings and commands for interacting with tmux windows and sessions.

If you close Shell while tmux is running, it will detach its session. The next time you open Shell, run tmux attach to return to the previous session. To leave the tmux session entirely, type exit. If you have multiple windows running, you will need to exit out of each first.

These resources provide more information about using tmux:

- A tmux Crash Course
- TMUX - The Terminal Multiplexer

## 26.9. Dmidecode

Dmidecode reports hardware information as reported by the system BIOS. Dmidecode does not scan the hardware, it only reports what the BIOS told it to. A sample output can be seen here.

To view the BIOS report, type the command with no arguments:

    dmidecode | more

dmidecode(8) describes the supported strings and types.