Easy Automated Docker Volume Backups That Are Database Friendly

I recently picked up a Beelink EQR6 Mini PC to reduce some of the Docker stress on my aging Synology NAS. Since my Synology used the Btrfs filesystem, I never had to worry about file locks and corruption during a backup because that particular file system used copy-on-write (CoW). However, since I decided to use Ubuntu Server on my Mini PC and neglected choosing which filesystem I wanted to use, I ended up with ext4.

Here’s the problem though.

The ext4 filesystem does not support copy-on-write. This means that if I tried to make backups of my Docker volumes, I’d run the risk of file corruption if those files were in use at the time of backup. This is particularly more of a problem with Docker volumes that contain SQLite databases with write-ahead log (WAL) or databases in general.

There’s good news though! There are a few automated solutions for safe backups of Docker volumes that can be used with minimal effort.

If you plan to follow along, note that I’ll be referencing Docker Compose for most of the Docker work. You don’t need a Mini PC like me, but I don’t know what kind of results you’ll get on macOS or Windows. I assume the results will be the same, but you never know.

We’re going to explore two solutions, Offen and Backrest. Both do the job of backups, but the end result is slightly different.

Backups with Offen Docker Volume Backup

Offen was my first choice when it came to Docker volume backups because it was the most talked about online. With Offen, you add it to your Docker Compose and specify what volumes you want backed up. Offen will run on a schedule you define, shutdown your container, and create a TAR archive to be uploaded to a destination of your choice.

Here is an example docker-compose.yml file that I’m using for my Gitea application:

version: "3"

services:
  gitea:
    image: "gitea/gitea:latest"
    container_name: "gitea"
    restart: unless-stopped
    ports:
      - "4022:22"
      - "3080:3000"
    volumes:
      - /home/nraboy/docker/gitea:/data:rw
    environment:
      - PUID=1000
      - PGID=1000
    labels:
      - docker-volume-backup.stop-during-backup=gitea
  backup:
    image: offen/docker-volume-backup:latest
    container_name: "gitea-backup-utility"
    restart: unless-stopped
    volumes:
      - /home/nraboy/docker/gitea:/backup/gitea:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /mnt/synology/docker/backups:/archive
      - /etc/localtime:/etc/localtime:ro
    environment:
      BACKUP_STOP_DURING_BACKUP_LABEL: gitea
      BACKUP_CRON_EXPRESSION: "@daily"
      BACKUP_FILENAME: "gitea-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"
      BACKUP_PRUNING_PREFIX: "gitea-"
      BACKUP_RETENTION_DAYS: 7

There are a few things happening here.

At a basic level, you’ll notice that we have two services in this file. One service is for Gitea while the other is my backup service that makes use of Offen. Let’s dig deeper into the Offen side of things:

backup:
  image: offen/docker-volume-backup:latest
  container_name: "gitea-backup-utility"
  restart: unless-stopped
  volumes:
    - /home/nraboy/docker/gitea:/backup/gitea:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - /mnt/synology/docker/backups:/archive
    - /etc/localtime:/etc/localtime:ro
  environment:
    BACKUP_STOP_DURING_BACKUP_LABEL: gitea
    BACKUP_CRON_EXPRESSION: "@daily"
    BACKUP_FILENAME: "gitea-%Y-%m-%dT%H-%M-%S.{{ .Extension }}"
    BACKUP_PRUNING_PREFIX: "gitea-"
    BACKUP_RETENTION_DAYS: 7

Starting with the volumes, we have two mappings that affect how Offen functions:

- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/localtime:/etc/localtime:ro

The first is a requirement so Offen can stop your container and restart the container when the backup has completed. The second fixes the timezone so when you define the backup schedule, it is in the same timezone as your host computer.

Then we see the other two volume mappings:

- /home/nraboy/docker/gitea:/backup/gitea:ro
- /mnt/synology/docker/backups:/archive

The first mapping is the volume that we want to back up. In my example it is a mapping to a local directory on the host, but you can do the same with other volume types as well. The second mapping is where I want the backup to go when it is done. In my example I have a CIFS volume mounted on my host. This CIFS volume is then mapped to a path on the container. Offen does support other backup location types, but CIFS made the most sense for me because I’m still using my Synology NAS for storage.

Looking beyond the volumes, we have configuration details in the environment variable section.

At first I made the novice mistake of not including the BACKUP_STOP_DURING_BACKUP_LABEL variable. As a result, when Offen ran, it would shutdown all my containers, not just the container it was backing up. Including the BACKUP_STOP_DURING_BACKUP_LABEL variable and using the same name found in the docker-volume-backup.stop-during-backup label fixed this issue so it would only shutdown the one container.

The other variables defined the backup schedule, file name, and pruning schedule.

If you’re using the Offen strategy like I did, you’ll need to add it as a service in every stack that you deploy. The process will be the same for all of them. Define the container to shut down, define the volumes and destinations, and define the schedule.

It is important to note that Offen works because it first shuts down your container before the backup happens. If the container is shut down, the files are not locked or are in use. This results in a clean backup to whatever destination you choose with no hand-holding.

Backups with Backrest, the Restic UI with Extras

Unlike Offen which gives you an exact archive of every volume you throw at it, Backrest does incremental backups using the very popular Restic tool behind the scenes. This is good if archive size or backup speed is an issue for your volumes.

With Backrest, you don’t need to have it added to every stack that you deploy. Instead it can exist as its own container or stack. Take the following docker-compose.yml file for example:

version: "3.8"
services:
  backrest:
    image: garethgeorge/backrest:latest
    container_name: backrest
    volumes:
      - /home/nraboy/docker/backrest/data:/data
      - /home/nraboy/docker/backrest/config:/config
      - /home/nraboy/docker/backrest/cache:/cache
      - /home/nraboy/docker/backrest/tmp:/tmp
      - /home/nraboy/docker:/docker_volumes:ro # Mount local paths to backup
      - /mnt/synology/docker/restic:/repos # Mount local repos (optional for remote storage)
      - /home/nraboy/restores:/restores # Mount for restored content
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - BACKREST_DATA=/data
      - BACKREST_CONFIG=/config/config.json
      - XDG_CACHE_HOME=/cache
      - TMPDIR=/tmp
      - TZ=America/Los_Angeles
    ports:
      - "9132:9898"
    restart: unless-stopped

Let’s look at the volumes because those are the only items that aren’t default to the Backrest documentation.

- /var/run/docker.sock:/var/run/docker.sock:ro

Like with the Offen example, you do need this mount if you want Backrest to be able to restart containers. Without it the containers will not stop and you risk having corrupted files or database data.

The next four mounts are for your configuration information:

- /home/nraboy/docker/backrest/data:/data
- /home/nraboy/docker/backrest/config:/config
- /home/nraboy/docker/backrest/cache:/cache
- /home/nraboy/docker/backrest/tmp:/tmp

I mount the above directories because I have them included in my backup schedule as well. However, if you don’t ever plan to automate the backups of Backrest itself, feel free to disregard them.

These are the directories that are important:

- /home/nraboy/docker:/docker_volumes:ro # Mount local paths to backup
- /mnt/synology/docker/restic:/repos # Mount local repos (optional for remote storage)
- /home/nraboy/restores:/restores # Mount for restored content

The first mapping is where I keep all of my Docker volumes. Each volume exists as a separate directory within this path. The second mapping is a CIFS mount to my Synology NAS. Just like Offen I plan to transfer all my backups to my NAS with CIFS. You can choose a storage mechanism that works best for you. S3 with Minio is also a very popular choice.

Finally I have a restores volume mapping in case I ever need to restore a backup. Notice that my source volumes are read only. I don’t want any accidents to happen so I’m choosing to restore elsewhere and then I can just copy the files to where they belong via SSH.

If you run this Docker Compose, you’ll end up with a Backrest application. However, all the scheduling and configuration is done through the interface.

The Backrest interface is not particularly difficult to figure out. You’ll need to create a repository to store your archive and a plan to choose what to backup and on what schedule. Remember the mappings created in the Docker Compose file? This is what you’d use for each of the repository and plan configuration stage.

Here is the important part though. The containers need to be stopped if you want a backup that isn’t potentially corrupted. You can configure this in the Hooks section of each backup plan.

You’ll want two hooks to accomplish the job. The first hook will use CONDITION_SNAPSHOT_START which runs before the backup starts. This means for the script we can include docker stop <container_name> to stop the container in question. When the container is stopped the backup will start. When we include the CONDITION_SNAPSHOT_END we’re triggering a script when the backup completes. In this case we are running docker start <container_name> to start the stopped container.

When you do this for each of your containers and volumes, you’ll get a perfect incremental backup and the containers will restart when everything is complete.

Why I’m Switching to Backrest and Restic

I’ve been using both Offen and Backrest without issues on my Mini PC. I started with Offen and realized that I have a few volumes that are massive. Take Plex for example. My core Plex data is already stored on my NAS with a CIFS volume. However, Plex has a lot of configuration metadata, which in my case was 30GB. While I could opt to not back this up and regenerate it slowly in the event of a problem, I didn’t want to. Using Offen to back this up would result in having to archive 30GB for every backup.

30GB in the grand scheme of things is not a lot, but it also isn’t a little.

This brought me to Backrest and the incremental backups that Restic offers. Sure I’d have to back up the 30GB the first time, but every other time would be only the things that changed. This resulted in less space occupied on my NAS and a much quicker backup since we’re talking kilobytes instead of gigabytes every time.

Conclusion

Moving my containers away from my Synology NAS that used Btrfs hit hard when I realized that there were snapshotting issues using the ext4 filesystem. I wanted to be able to store my container configuration directly on a CIFS volume which isn’t always supported or advisable. Making use of Offen or Backrest was a huge win for my home server operation.

Both Offen and Backrest will accomplish the job because they can both stop and start your containers. Most of the difference you’ll see is in the end result. Do you want a solid archive file with your volume or do you want an incremental backup.

Nic Raboy

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in C#, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Unity. Nic writes about his development experiences related to making web and mobile development easier to understand.