Mitigating Corruption with Backup File Health Checks

Overview

A health check for backup files is a periodic task that can be enabled on Veeam backup jobs to ensure that restore points within a backup chain remain usable. During the health check, a CRC check for metadata and a hash check for backup files is performed to verify file integrity.

Why Cloud Connect Backups Should Run Health Checks

While the chances are low, all jobs are potentially susceptible to corruption that could be caused by a number of factors. This goes double for Cloud Connect Backup. Not only are backups being transmitted across the public internet, but there are also several more Veeam components and tasks at play in the background compared to an on-premises backup job. When transmitting backup data to a Veeam Cloud & Service Provider, enabling health checks on backup jobs helps to ensure that any corruption that could occur in transit to the offsite location is mitigated and repaired.

How Health Checks Work

Health checks always verify only the latest complete restore point in the backup chain. For forward incremental and forever forward incremental chains, this means several backup files must be opened. For this reason, the health check procedure may take a long time. 

As of Veeam Backup & Replication release version 11a, enhancements have been made to the health check process that allows the process to complete 5-10x faster than in previous releases. For this reason, it is recommended that your Veeam server be upgraded to the latest version of VBR supported by OffsiteDataSync.

When a new restore point is saved to a backup repository, CRC values for backup metadata and hash values for data blocks of VM disks are calculated and to the backup file along with the VM data. These values are then used using health check sessions to ensure restore point consistency. Depending on the backup chain type, different mechanisms are used for the health check. This article will cover only the chain types utilized by backup copy jobs, as these will be the mechanisms used for backup chains written to ODS.

The health check is performed in the following way:

1. At the beginning of a backup copy interval, Veeam performs the health check
2. If the health check does not detect data corruption, the copy job resumes its normal tasks of transmitting data to the cloud repository and performing any scheduled synthetic full merges
3. If corruption is detected, Veeam will attempt to perform several actions depending on the type of corruption:
   1. If backup metadata is corrupted in the full backup file, Veeam will attempt to re-transmit data blocks for the whole VM image from the source repository and create a new full backup
   2. If backup metadata is corrupted in the incremental backup, Veeam marks the restore point as unusable and re-transmits the data needed to synthesize a new incremental. In addition, any additional changes awaiting transmission will be added to the new incremental, so only one new incremental backup is created

Image source: Health Check for Backup Files, Veeam User Guide

How to Configure Health Checks on Backup and Backup Copy Jobs

1. Launch the Veeam Backup and Replication console
2. Select the job you wish to enable the functionality on
3. Click “Edit” from the toolbar or right-click the job and select “Edit…” from the context menu.
4. Under the “Storage” page, click the “Advanced” button and then select the “Maintenance” tab

Image source: Health Check for Backup Files, Veeam User Guide

1. Check the box “Perform backup files health check…”
2. We recommend setting health checks to run once monthly, coinciding with a Friday night or Saturday morning start time, so if needed they can run the duration of the weekend
3. Click “Ok” to close the advanced settings page
4. Click “Finish” to save the changes to the job settings

Health Check Considerations and Limitations

1. Health checks are not performed during active full backup job sessions
2. Health checks are not performed on offloaded restore points. At ODS, this means they are not performed for any files written to our archive tier
3. For per-machine chains, if a machine is added to an existing backup chain, Veeam will perform a health check for it during the next incremental session that the added machine is a part of
4. If a health check is performed on an encrypted job, Veeam will temporarily pass the encryption key to the repository the data is located on
5. Health checks will only run during the first job session of the day that it is scheduled. Therefore, if a health check is scheduled on a day that runs more than once per day, the health check will only be performed once

Additional Resources

For more information, visit the Veeam Backup & Replication User Guide.