BTRFS on small DS-models and impact on restoring ActiveBackupforBusiness ABB backups

[Stephan]

Hi folks

I’m sysadmin with about 20 SMB customers running Synology for file servers and backups.
Since I am in area with bad internet, cloud backups are not viable means for off-site backups and therefore I am constantly looking to find the BEST WAY TO GET DATA OFF the synology boxes to rotating media. Unfortunately tape devices are not affordable for SMB customer.

I’m currently fighting a very similar issues at different customers with DS200/220/415/420+ devices.
All systems are running perfectly well (file server, ABB, ABO365, CloudSync) and SMB transfer rates are ok-ish.

But trouble starts when starting Hyperbackup to get data out of the Synology univers.
I recently set up 2 customers ro run Hyperbackup on DS200+ to rotating USB harddrives. As soon as the Hyperbackup jobs start running the NAS becomes so unresponsive it is unuseable.

Customer #1 is Hyperbackup via USB (2 hours just to backup app settings, no sared folders)
Customer #2 is Hyperbackup via 1Gb Ethernet (9 DAYS for initial backupjob, 3Tb with 6Mio files)

I have noted that both system basically crash as soon as hyperbackup compares the content of the local BTFRS volume with the locally attached USB harddrive (ext4) or SMB3 share on secondary identical DS220+. Memory/processor usage is basically the same as always but Disk utilization is maxed out with both systems.

Also, I have noted that restore operations of ActiveBackup jobs from older NAS models (e.g.DA415, DS420 with BTRFS volumes on 4 disk Raid5 systems ) to Windows Hyper-V servers take very long. (e.g. 20+ hours for 125GB virtual machine)

My thoughts on this are:

• While BTRFS is a VERY nice file system, reading big chunks of data or very tiny files has a massive impact on the disks. If your NAS has not enough horse power or only 2-4 disks, a big recovery/copy job takes too long to be practical.
• Snapshot replication is a very nice tool to keep data (and version history) in sync on multiple devices but it still keeps the data within the synology environment
• When working with low-budget NAS plattforms a work around is to have a file sync tool like GOODSYNC to read data from NAS to a HDD connected to local computer.

Considerations for restoring computers:
When you have to recover a crashed server or VMs you need to get the data FROM the NAS to the target device. In my eyes the most efficient way is to have at hand a more powerfull NAS (e.g. 1523+ when restoring 2-4 bay systems). Transfer the discs from 2/4 bay NAS into the bigger System. Insert a off-brand SSD in the 5th slot for an additional fast volume. Also a Synology NVMe drive could be used for a NVMe volume. Install VMM virtual machine manager on the new volume and restore the computers locally in VMM. Like this the server/computer/VM becomes quickly accessible via RDP and the bare metal recovery can be done without pressure.

TOO LONG / DIDN’t READ - When dealing with ActiveBackupForBusness tell the customer that a DS1523+ or bigger is imperative (because of 32GB RAM + 5 drives). Do not discard the old devices.
Especially the old 2-bay systems can than be used as portable rotating off-site cold storage devices via Snapshot replication (the sync jobs automatically restart once the devices are connecteds to the same network as the primary device)

What are your thoughs on this?

[Will]

I absolutely have seen slow restores with ABB. One thing that really helps speed it along is to pin all of the BTRFS metadata to a read write NVMe SSD cache. This can have huge bumps in performance because when its piecing together all of the versions again all of the metadata is already in NVMe.

When it comes to hyper backup I have really not seen horrible slow downs like that, except in cases where the volume is really full, or a dying drive (more on that in a second). Another killer thing is making sure you have at least 2 gigs of ram if you are running DSM 7 or later. It really can use it.

@Stephan

Customer #1 is Hyperbackup via USB (2 hours just to backup app settings, no sared folders)

So if you are backing up app settings like drive, hyper backup will automatically backup and Synology Drive team folders

@Stephan

I have noted that both system basically crash as soon as hyperbackup compares the content of the local BTFRS volume with the locally attached USB harddrive (ext4) or SMB3 share on secondary identical DS220+. Memory/processor usage is basically the same as always but Disk utilization is maxed out with both systems.

In the past when I have seen this, it’s normally due to one of the disks in the RAID volume dying. What I will do is open up activity monitor, and check the disk utilization of all the drives. In a few different cases I have seen that one drive is pined at 95%+, but the rest are at like 20%. Where most likely the drive is dying, but has not actually thrown any errors. Pulling it makes the whole NAS responsive again.

I have seen this with hyperbackup (among other times). I think it’s likely due to an error somewhere on disk that is pretty much never accessed.

@Stephan

When working with low-budget NAS plattforms a work around is to have a file sync tool like GOODSYNC to read data from NAS to a HDD connected to local computer.

You could also just use USB copy to send them over, or do hyper backup (single version) which is just an rsync