So one of the best tunables for ZFS is setting the arc_meta_min parameter to something like 10 gigs, if you have more than ~32 gigs of RAM. Metadata is incredibly small on ZFS, so my entire 40 TB system has only about 7 gigs of ARC taken up by this, but it means that my random performance when accessing files is huge! ZFS has a tendency to evict metadata in favor for data, pretty easily. Something I wish was not the case because in my experience metadata is almost as valuable as data in the ARC, but takes up less than .1% the space so its not crazy to leave it all in ARC! Things like rsyncs, Time Machine backups, and file sync tools get insane performance bumps from doing this.
Hi Will - How would a special vdev for metadata figure in? Raw speed wouldn’t be as good as RAM but much better than spinning drives. Would you still recommend allocating more ARC space to metadata even with an NVME metadata vdev?
So if you have your redundant metadata only vdev then this is not as crucial as reads are not nearly as time intensive. But for how ‘cheap’ it is (5 gigs can get most people everything in RAM) its still a good idea to increase!
1 Like