My TrueNAS Scale custom configs to maximize ram usage and use more metacaching

[Will]

Overview

Wanted to add a list here of my custom configs that I use to change how TrueNAS Scale uses RAM. Basically linux ram could be as high as 2x usage then what its expected due to fragmentation, so this commit updated RAM usage in Linux ZFS to only 50%. This fragmentation is a worst case scenario, so I normally allocate it to 80%. Especially on high ram systems.

I also really like adding in more metadata in the arc talked about in this post here:

One of the best ZFS tunable parameters out there - ZFS.arc_meta_min

So one of the best tunables for ZFS is setting the arc_meta_min parameter to something like 10 gigs, if you have more than ~32 gigs of RAM. Metadata is incredibly small on ZFS, so my entire 40 TB system has only about 7 gigs of ARC taken up by this, but it means that my random performance when accessing files is huge! ZFS has a tendency to evict metadata in favor for data, pretty easily. Something I wish was not the case because in my experience metadata is almost as valuable as data in the ARC,…

My ZFS parameters

Warning: Do this at your own risk, these are for my 128 gig setup, you will need to change the values according to your setup

Allocate 100 gigs as the max ram:

echo $((100 * 2**30)) >> /sys/module/zfs/parameters/zfs_arc_max

Allocate 30 gigs as the min ram (not really required):

echo $((30 * 2**30)) >> /sys/module/zfs/parameters/zfs_arc_min

Force at least 8 gigs of ARC to be used for metadata (really nice to have)

echo $((8 * 2**30)) >> /sys/module/zfs/parameters/zfs_arc_meta_min

I go over these in an upcoming video here!

My ULTIMATE Custom Video Editing NAS (all SSD, 25GbE, TrueNAS Scale)

[CapitalJD]

Hi Will - I installed the beta version of TrueNAS Cobia (sp?) in my homelab. I’m not seeing zfs_arc_meta_min in the parameters directory, but I am seeing zfs_arc_meta_balance, which openzfs defines as

Balance between metadata and data on ghost hits. Values above 100 increase metadata caching by proportionally reducing effect of ghost data hits on target data/metadata rate.

Do you think they’re changing this parameter in the latest release?

[Will]

Interesting! I have not seen this. Hopefully this means that as a default we will have more metadata in ARC

[Haruki]

After updating to TrueNAS-SCALE-23.10.2, the metadata command doesn’t seem to work anymore. Has this happened to your system?