Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

copy snapshots to other repos after creation #279

Closed
mariomare22 opened this issue May 15, 2024 · 6 comments
Closed

copy snapshots to other repos after creation #279

mariomare22 opened this issue May 15, 2024 · 6 comments
Labels
enhancement New feature or request

Comments

@mariomare22
Copy link

mariomare22 commented May 15, 2024

Note: if you have a question or want discussion please post in the discussions area.

Is your feature request related to a problem? Please describe.
mostly overhead in backing up same plan to multiple repo

Describe the solution you'd like
Possibility to copy a plan to multiple reps to decrease the overhead of backing up multiple times instead of having to create for each repo multiple plans. could be in selecting multiple repos in the following screenshot

image

Additional context
feature provided and seen in autorestic
Link to feature described.

@mariomare22 mariomare22 added the enhancement New feature or request label May 15, 2024
@mariomare22 mariomare22 changed the title copy repo to different location copy plan to different repos May 15, 2024
@garethgeorge garethgeorge changed the title copy plan to different repos copy snapshots to other repos after creation May 19, 2024
@garethgeorge
Copy link
Owner

garethgeorge commented May 19, 2024

This is a pretty interesting idea. Dropping some ideas here re: implementation

I think this can be done as a CONDITION_SNAPSHOT_END hook that runs commands to copy snapshots to an additional configured repo in terms of the UX.

This is unfortunately complicated a bit by Backrest's data model as today it's assumed that the plan ID is singleton across all repos, but if you clone a snapshot that snapshot will be added to a different repo with the same tags and will be imported incorrectly. That's probably a bug that needs some thinking about.

Overall I'm definitely open to supporting this.

@garethgeorge
Copy link
Owner

Patched the issues related to backrest's data model in 8a43f86 , all queries now respect both the repo and plan that an operation belongs to.

@Janhouse
Copy link
Contributor

I think that having ability to somehow select multiple repos for the same plan is an important feature to have and it would be a better approach.

restic docs warn that copying snapshots has some drawbacks.

And being able to back up locally and remotely is quite important. Currently you have to duplicate the plans to achieve this.

Others suggest using rclone sync to duplicate repos, but if the main repo gets damaged, the clone will have the same issue, so at the end of the day it is better to run the backup twice, even if it is a bit slower.

@garethgeorge
Copy link
Owner

Spent a bit of time this afternoon investigating here -- the main challenge I see is that two repositories need to be specified which includes the need to be able to pass credentials. This doesn't look like it's very well supported.

e.g. the docs https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories here talk about this a bit, but it's not obvious to me how one would go about providing two different sets of AWS access keys, for example.

@Janhouse
Copy link
Contributor

But look at the notes in the restic docs, it is not a good idea to use the copy command.

In my opinion it would be better if we could reuse the same backup plan for multiple restic repos.

Currently it is tied to a plan one to one, I propose making it one (plan) to many (repos). It would run multiple times but would use less bandwidth and be faster (than restic copy), and probably be safer (than rsync/rclone).

@garethgeorge
Copy link
Owner

Since this bug a number of related issues have come in, I think the most relevant discussion is now on #508 . Going ahead and closing this to deduplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants