Backrest on BTRFS snapshots, email diff #320

lucmos · 2024-06-04T13:53:53Z

lucmos
Jun 4, 2024

Hello,
I would like to abandon my hacky scripts to perform backups with restic 😄
After searching for a while, I think backrest would suit my usecase perfectly!

My scripts work in the following way:

#!/usr/bin/bash

btrfs subvolume snapshot -r /srv/dev-disk-by-label-data/luca/ /srv/dev-disk-by-label-data/.snapshots/srv/dev-disk-by-label-data/luca

echo -e "\n\n~~~~~\n\n"

. /root/restic_envs_passport.sh
echo -e "\n\n\n---\nBACKUP TO: ${RESTIC_REPOSITORY}\n---\n"
/root/bin/restic --verbose backup "/srv/dev-disk-by-label-data/.snapshots/srv/dev-disk-by-label-data" --tag data --exclude-file="/root/backup-exclude.txt"
echo -e "\n---\nDIFF\n---\n"
/root/bin/restic snapshots --json | python -c "import sys, json; j=json.load(sys.stdin); j = [x for x in j if 'tags' not in x or 'fireflyiii' not in x['tags'] or 'actual' not in x['tags']]; print(j[-2]['short_id'] + ' ' + j[-1]['short_id'])" | xargs /root/bin/restic diff

echo -e "\n\n~~~~~\n\n"

btrfs subvolume delete /srv/dev-disk-by-label-data/.snapshots/srv/dev-disk-by-label-data/luca

The idea is the following:

The backup scripts first creates btrfs snapshots, to freeze the state and avoid inconsistent backups (in case the state changes while the backup is in progress. A classic example is backing up a live database, that often is unreadable without this kind of state freeze)
Execute restic backup
Print the differences with an hacky one liner. This is useful because, since it's executed in a crontab, these differences are emailed directly to me
Remove the btrfs temporary snapshots

I am interested in automating all this with backrest.
Could you give some pointer on what is the best way to do it?

Any help is much appreciated! 😄

Answered by Hoempi

Nov 5, 2024

I just went ahead and tested this. Works like a charm, this is my main backup plan:

{
  "id": "main",
  "repo": "optiplex",
  "paths": [
    "/mnt/docker-data/docker-data-snapshot",
    "/opt/stacks"
  ],
  "excludes": [],
  "iexcludes": [],
  "schedule": {
    "clock": "CLOCK_LOCAL",
    "cron": "0,42 * * * *"
  },
  "retention": {
    "policyTimeBucketed": {
      "yearly": 1,
      "monthly": 3,
      "weekly": 4,
      "daily": 7,
      "hourly": 5
    }
  },
  "hooks": [
    {
      "conditions": [
        "CONDITION_SNAPSHOT_START",
        "CONDITION_SNAPSHOT_ERROR",
        "CONDITION_SNAPSHOT_SUCCESS"
      ],
      "actionHealthchecks": {
        "webhookUrl": "https://myhc.exam…

View full answer

garethgeorge · 2024-06-07T18:42:43Z

garethgeorge
Jun 7, 2024
Maintainer

Hmm, this is pretty challenging in backrest today -- backrest doesn't support a dynamic path for backup and it looks like restic won't follow symlinks.

You'll need a way to mount the btrfs snapshot in a consistent location accessible to backrest. In terms of actually running scripts before/after backup that should be doable with the CONDITION_BACKUP_START and CONDITION_BACKUP_END set on a command hook.

10 replies

Hoempi Nov 4, 2024

I actually found this as I'm plannning on doing basically the same. With one key difference: My Btrfs snapshot gets mounted on the same path everytime. So that should be a doable in general, as far as I understand.

My question, being pretty new to using Btrfs snapshots like this, is whether this could be done when running the Backrest Docker container. Probably not, as that would result in the Docker container needing a great level of access to the filesystem of the host system, correct?

So, I assume I would need to install Backrest directly on the host system to reach this.

Hoempi Nov 5, 2024

I just went ahead and tested this. Works like a charm, this is my main backup plan:

{
  "id": "main",
  "repo": "optiplex",
  "paths": [
    "/mnt/docker-data/docker-data-snapshot",
    "/opt/stacks"
  ],
  "excludes": [],
  "iexcludes": [],
  "schedule": {
    "clock": "CLOCK_LOCAL",
    "cron": "0,42 * * * *"
  },
  "retention": {
    "policyTimeBucketed": {
      "yearly": 1,
      "monthly": 3,
      "weekly": 4,
      "daily": 7,
      "hourly": 5
    }
  },
  "hooks": [
    {
      "conditions": [
        "CONDITION_SNAPSHOT_START",
        "CONDITION_SNAPSHOT_ERROR",
        "CONDITION_SNAPSHOT_SUCCESS"
      ],
      "actionHealthchecks": {
        "webhookUrl": "https://myhc.example.com/ping/76000475-this-is-a-dummy"
      }
    },
    {
      "conditions": [
        "CONDITION_SNAPSHOT_START"
      ],
      "onError": "ON_ERROR_FATAL",
      "actionCommand": {
        "command": "btrfs subvolume snapshot -r /mnt/docker-data /mnt/docker-data/docker-data-snapshot"
      }
    },
    {
      "conditions": [
        "CONDITION_SNAPSHOT_END"
      ],
      "onError": "ON_ERROR_CANCEL",
      "actionCommand": {
        "command": "btrfs subvolume delete /mnt/docker-data/docker-data-snapshot"
      }
    }
  ]
}

I also really like that the healtcheck hooks automatically choose the correct path, e.g. https://myhc.example.com/ping/76000475-this-is-a-dummy/start

Answer selected by garethgeorge

garethgeorge Nov 15, 2024
Maintainer

Really neat that this works with your btrfs setup today and very cool to see scripting used this way. If you don't mind, I might actually steal this as an example for the cookbooks section on the wiki. Is it still working well for you?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backrest on BTRFS snapshots, email diff #320

{{title}}

Replies: 1 comment 10 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Backrest on BTRFS snapshots, email diff #320

lucmos Jun 4, 2024

Replies: 1 comment · 10 replies

garethgeorge Jun 7, 2024 Maintainer

Hoempi Nov 4, 2024

Hoempi Nov 5, 2024

garethgeorge Nov 15, 2024 Maintainer

Hoempi Nov 15, 2024

robflate Dec 2, 2024

Hoempi Dec 2, 2024

lucmos
Jun 4, 2024

Replies: 1 comment 10 replies

garethgeorge
Jun 7, 2024
Maintainer

garethgeorge Nov 15, 2024
Maintainer