As @Will said in this YouTube video, a Layer 2 ARC might even be detrimental if you already have an SSD pool. So I calculated this, and he’s right. (Just idealized numbers, of course.)
Let’s say you have 7 x 4TB SATA SSDs in RAIDZ1 (RAID5): then you’d get a maximum read speed of 3.92 GB/s (100% read operations). So even with a fast gen3 M.2 NVMe SSD as L2ARC, with the fastest speed on paper at about 3.5 GB/s, your performance would in fact suffer, because it would be faster for ZFS to fetch the data directly from the pool, if it’s not in the ARC.
But what about a gen4 M.2 SSD? On paper, those will deliver up to 7.5 GB/s, which would be more than twice as fast as the maximum theoretical speed of the SSD pool.
So I surmise that using a gen4 M.2 as L2ARC for a SATA SSD pool would in fact boost performance. True? Or am I missing something?
Of course, once you start building your pool with gen3 M.2 or U.2 SSDs, I guess that even a gen5-based L2ARC wouldn’t add anything, and even reduce performance, similar to what a gen3 M.2 would do for a SATA SSD pool.
Addendum: I assume you could always use an M.2 carrier card, which in my case would be x8, since PCIe slot 1 is always occupied by a NIC (x8). Those cards should have a PLX or a switch, so they’d work without mainboard PCIe bifurcation, e.g. the HighPoint Rocket 1204 (4 slots) or the Delock 90305 (2 slots), so you can add two M.2 SSDs as a stripe (RAID0) for the L2ARC. The available L2ARC speed would then again exceed that of a SATA SSD RAIDZ1 pool, at least for pools with up to 12 drives (theoretical maximum read speeds).
So I guess that would be an option for gen3 M.2 SSDs, too.
So L2ARC is really only blow away good for HDD pools, it may help SATA SSD pools (though you would want to maybe only add metadata to it, or other specific use cases).
The other thing is L2ARC’s increase throughput similar to how RAID0 does. So if you did somehow have that bottleneck you could always add in 4x NVMe’s and get 4x the speed.
All that being said: It’s pretty rare to actually have a sequential workflow on a NAS that is hitting that 2+GB/s. While it can happen, it’s going to be pretty rare.
Yes. I wouldn’t buy it right away anyhow… I would however go with a consumer-grade SATA SSD storage pool (RAID5) with at least seven SSDs right out of the gate, whether unRAID (my current favorite), TrueNAS Scale or good old Synology.
(Another option is a DIY build with a pool of seven hotswap gen3 M.2 NVMe SSDs, but that depends on price & budget.)
In the end, if it comes to the L2ARC, I’ll just calculate the speeds against each other (network vs pool vs M.2 cache), and see where that leads me… but a striped L2ARC would probably help, definitely striped gen4. (If the network is fast enough.)
Question is if I’ll need it. I’ll eventually see how the server/NAS performs as part of an audio/music production setup, and only then will I decide.
But yes, metadata (and application data) only would be the only thing I’d use it for.