Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Copying snapshots between repositories #465

Open
inode64 opened this issue Sep 9, 2024 · 8 comments
Open

Support for Copying snapshots between repositories #465

inode64 opened this issue Sep 9, 2024 · 8 comments
Labels
enhancement New feature or request

Comments

@inode64
Copy link
Contributor

inode64 commented Sep 9, 2024

I would like to submit a feature request that I believe could greatly enhance the usability and reliability of your system. Specifically, I would like to request the addition of support for copying between repositories, which would enable users to create remote backups and synchronize repositories as if performing a scheduled backup.

Key Benefits:

  • Remote Backup: This feature would allow users to maintain remote backups of critical repositories, ensuring data safety and disaster recovery.
  • Scheduled Synchronization: By enabling synchronization between repositories, users could automate backup processes, making them more reliable and less prone to human error.
  • Redundancy and Security: Copying between repositories would increase redundancy, adding an extra layer of protection for important projects.

Suggested Implementation:

  • Enable copying between two different repositories, either in the same system or across different instances.
    Integrate a scheduling feature so that synchronization can occur automatically at predefined intervals, similar to a regular backup.
  • Provide options for selective copying or full repository replication, to give users control over the data being synchronized.
  • I believe this feature would be extremely valuable for teams looking to ensure the security and availability of their repositories. Thank you for considering this suggestion. Please let me know if you need any further information or clarification.
@inode64 inode64 added the enhancement New feature or request label Sep 9, 2024
@garethgeorge
Copy link
Owner

garethgeorge commented Sep 26, 2024

Hey, thanks for the FR. This is something I've seen a number of people surface as a want -- and I think it's reasonable.

So far I've been hesitant though, primarily due to this warning in the restic docs: https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories .

There are some significant footguns here. It may still be reasonable to try supporting with a warning. I think the way to go about it would be through the hook subsystem e.g. as a "CONDITION_SNAPSHOT_SUCCESS" hook that runs a restic copy command to a repo that's defined.

Edit: another piece of complexity i'm seeing is that the restic copy command uses a different set of env vars to specify options -- this one could actually be a blocker as backrest today doesn't try to interpret flags / env vars (just forwards them). Some thought is needed re: how we'd model this.

Definitely open to ideas if there's a low cost way to implement this.

@Janhouse
Copy link
Contributor

As a workaround, for now I am testing the use of repository CONDITION_CHECK_SUCCESS hook to trigger rclone sync command daily.

Using the hook ensures that while the clone is running, it won't do other actions.

Though I would prefer to have some other way to trigger this, so at the end of it I could also trigger the Discord hook and notify of the status.

Here is my config

I have custom Dockerfile that adds rclone and config for it

FROM garethgeorge/backrest:latest-alpine

RUN apk --no-cache add docker docker-compose rclone

RUN mkdir -p /root/.config/rclone/
COPY rclone.conf /root/.config/rclone/

The rclone.conf

[backup-server]
type = sftp
host = 10.10.10.10
user = backup
key_file = /root/privkey
disable_hashcheck = true

And then in docker-compose.yml I mount the private key - ./id_rsa:/root/privkey

The hook command is basically

rclone sync /backups/restic-repo/ backup-server:backups/restic-repo

@garethgeorge
Copy link
Owner

+1, I think using rclone or rsync type tools in a hook is a good way to do this, it's safe to rely on the fact that hooks block any other operation from running on the repo.

@Janhouse
Copy link
Contributor

I think the downside is that if rclone fails, you have a broken mirror. This could be handled by mirroring to a different folder and then swapping them, but that requires double disk space and existing files would always be uploaded.

I will try using the hooks and do the restic copy, but that can have some issues with file deduplication, as I read in the readme. 🫤

My big wish is still that backrest would allow selecting multiple restic repos for the same backup plan, without manually having to copy over json configs. 🙏

@garethgeorge
Copy link
Owner

The question in my mind with supporting multiple repos in the UI is what the expectations around hooks and scheduling should be.

It introduces something of a cross product e.g. plans cross repos in terms of the number of scheduled tasks. It potentially introduces some footguns as well re: how repo hooks and plan hooks might interact across those combinations that need thinking through.

I wonder if some other concept e.g. a path template or similar is the way to go such that large pieces of configuration can be kept in sync between multiple plans? I.e. flags or excludes, etc.

@mattdale77
Copy link

+1 for some way to do this.
With bare restic I have actually been running the same backup twice, once for each repository. I don't remember exactly why but it was probably to do with issues with copying snapshots.
In my case the ability to add a second repository into the plan would satisfy my requirements I think

@redoranges
Copy link

I'd like to get this working with command hooks as you suggested, however I'm stuck trying to locate the restic binary to run the command. I tried "restic" and "$BACKREST_RESTIC_COMMAND"

This is my untested first attempt:
$BACKREST_RESTIC_COMMAND copy --from-repo /config/from_repository_file --from-password-file /config/from_password_file --repository-file /config/repository_file --password-file /config/password_file

It looks like restic is maybe in a version specific directory?

@garethgeorge
Copy link
Owner

In the latest version of backrest, the binary is now consistently named in ~/.local/share/backrest/restic or will use the system version of restic if one is available and matches the expected version. Check the logs from a backup operation to see what path backrest uses as of 1.6.2 or higher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants