Skip to content

Commit

Permalink
fix: improve handling of restore operations
Browse files Browse the repository at this point in the history 
 - restore operations are split into a new flow
 - added support displaying restore operation percentage and other
   details in tree view
  • Loading branch information
@garethgeorge
garethgeorge committed Jul 3, 2024
1 parent 64aa4f2 commit 620caed
Show file tree
Hide file tree
Showing 11 changed files with 142 additions and 122 deletions.
22 changes: 17 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Backrest itself is built in Golang (matching restic's implementation) and is shi

# Installation

Backrest is packaged as a single executable. It can be run directly on Linux, macOS, and Windows. [restic](https://github.com/restic/restic) will be downloaded and installed in the data directory on first run.
Backrest is packaged as a single executable. It can be run directly on Linux, macOS, and Windows. [restic](https://github.com/restic/restic) will be downloaded and installed on first run.

Download options

Expand All @@ -62,7 +62,7 @@ Download options
Backrest is accessible from a web browser. By default it binds to `127.0.0.1:9898` and can be accessed at `http://localhost:9898`. Change the port with the `BACKREST_PORT` environment variable e.g. `BACKREST_PORT=0.0.0.0:9898 backrest` to listen on all network interfaces. On first startup backrest will prompt you to create a default username and password, this can be changed later in the settings page.

> [!Note]
> Backrest installs a specific restic version to ensure that the version of restic matches the version Backrest is tested against. This provides the best guarantees for stability. If you wish to use a different version of restic OR if you would prefer to install restic manually you may do so by setting the `BACKREST_RESTIC_COMMAND` environment variable to the path of the restic binary you wish to use.
> Backrest installs a specific restic version to ensure that the restic dependency matches backrest. This provides the best guarantees for stability. If you wish to use a different version of restic OR if you would prefer to install restic manually you may do so by setting the `BACKREST_RESTIC_COMMAND` environment variable to the path of the restic binary you wish to use.
## Running with Docker Compose

Expand Down Expand Up @@ -234,12 +234,24 @@ To run the binary on login, create a shortcut to the binary and place it in the

# Configuration

## Environment Variables
## Environment Variables (Unix)

| Variable | Description | Default |
| ------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `BACKREST_PORT` | Port to bind to | 9898 |
| `BACKREST_PORT` | Port to bind to | 127.0.0.1:9898 (or 0.0.0.0:9898 for the docker images) |
| `BACKREST_CONFIG` | Path to config file | `$HOME/.config/backrest/config.json`<br>(or, if `$XDG_CONFIG_HOME` is set, `$XDG_CONFIG_HOME/backrest/config.json`) |
| `BACKREST_DATA` | Path to the data directory | `$HOME/.local/share/backrest`<br>(or, if `$XDG_DATA_HOME` is set, `$XDG_DATA_HOME/backrest`) |
| `BACKREST_RESTIC_COMMAND` | Path to restic binary | Defaults to a Backrest managed version of restic |
| `BACKREST_RESTIC_COMMAND` | Path to restic binary | Defaults to a Backrest managed version of restic at `$XDG_DATA_HOME/backrest/restic-x.x.x` |
| `XDG_CACHE_HOME` | Path to the cache directory | |

## Environment Variables (Windows)

## Environment Variables (Linux)

| Variable | Description | Default |
| ------------------------- | --------------------------- | ------------------------------------------------------------------------------------------ |
| `BACKREST_PORT` | Port to bind to | 127.0.0.1:9898 |
| `BACKREST_CONFIG` | Path to config file | `%appdata%\backrest` |
| `BACKREST_DATA` | Path to the data directory | `%appdata%\backrest\data` |
| `BACKREST_RESTIC_COMMAND` | Path to restic binary | Defaults to a Backrest managed version of restic in `C:\Program Files\restic\restic-x.x.x` |
| `XDG_CACHE_HOME` | Path to the cache directory | |
88 changes: 44 additions & 44 deletions gen/go/v1/operations.pb.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 1 addition & 6 deletions internal/api/backresthandler.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -400,12 +400,7 @@ func (s *BackrestHandler) Restore(ctx context.Context, req *connect.Request[v1.R
}

at := time.Now()

flowID, err := tasks.FlowIDForSnapshotID(s.oplog, req.Msg.SnapshotId)
if err != nil {
return nil, fmt.Errorf("failed to get flow ID for snapshot %q: %w", req.Msg.SnapshotId, err)
}
s.orchestrator.ScheduleTask(tasks.NewOneoffRestoreTask(req.Msg.RepoId, req.Msg.PlanId, flowID, at, req.Msg.SnapshotId, req.Msg.Path, req.Msg.Target), tasks.TaskPriorityInteractive+tasks.TaskPriorityDefault)
s.orchestrator.ScheduleTask(tasks.NewOneoffRestoreTask(req.Msg.RepoId, req.Msg.PlanId, 0 /* flowID */, at, req.Msg.SnapshotId, req.Msg.Path, req.Msg.Target), tasks.TaskPriorityInteractive+tasks.TaskPriorityDefault)

return connect.NewResponse(&emptypb.Empty{}), nil
}
Expand Down
41 changes: 14 additions & 27 deletions internal/orchestrator/tasks/taskcollectgarbage.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,20 @@ import (
const (
gcStartupDelay = 60 * time.Second
gcInterval = 24 * time.Hour
// keep operations that are eligible for gc for 30 days OR up to a limit of 100 for any one plan.
// an operation is eligible for gc if:
// - it has no snapshot associated with it
// - it has a forgotten snapshot associated with it
gcHistoryAge = 30 * 24 * time.Hour
gcHistoryMaxCount = 1000
// keep stats operations for 1 year (they're small and useful for long term trends)
gcHistoryStatsAge = 365 * 24 * time.Hour
)

// gcAgeForOperation returns the age at which an operation is eligible for garbage collection.
func gcAgeForOperation(op *v1.Operation) time.Duration {
switch op.Op.(type) {
// stats, check, and prune operations are kept for a year
case *v1.Operation_OperationStats, *v1.Operation_OperationCheck, *v1.Operation_OperationPrune:
return 365 * 24 * time.Hour
// all other operations are kept for 30 days
default:
return 30 * 24 * time.Hour
}
}

type CollectGarbageTask struct {
BaseTask
firstRun bool
Expand Down Expand Up @@ -83,11 +87,8 @@ func (t *CollectGarbageTask) gcOperations(oplog *oplog.OpLog) error {
forgot, ok := snapshotForgottenForFlow[op.FlowId]
if !ok {
// no snapshot associated with this flow; check if it's old enough to be gc'd
maxAgeForType := gcHistoryAge.Milliseconds()
if _, isStats := op.Op.(*v1.Operation_OperationStats); isStats {
maxAgeForType = gcHistoryStatsAge.Milliseconds()
}
if curTime-op.UnixTimeStartMs > maxAgeForType {
maxAgeForOperation := gcAgeForOperation(op)
if curTime-op.UnixTimeStartMs > maxAgeForOperation.Milliseconds() {
forgetIDs = append(forgetIDs, op.Id)
}
} else if forgot {
Expand All @@ -107,17 +108,3 @@ func (t *CollectGarbageTask) gcOperations(oplog *oplog.OpLog) error {
zap.Any("operations_removed", len(forgetIDs)))
return nil
}

func (t *CollectGarbageTask) Cancel(withStatus v1.OperationStatus) error {
return nil
}

func (t *CollectGarbageTask) OperationId() int64 {
return 0
}

type gcOpInfo struct {
id int64 // operation ID
timestamp int64 // unix time milliseconds
isStats bool // true if this is a stats operation
}
4 changes: 2 additions & 2 deletions internal/orchestrator/tasks/taskrestore.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ func restoreHelper(ctx context.Context, st ScheduledTask, taskRunner TaskRunner,

zap.S().Infof("restore progress: %v", entry)

restoreOp.Status = entry
restoreOp.LastStatus = entry

sendWg.Add(1)
go func() {
Expand All @@ -91,7 +91,7 @@ func restoreHelper(ctx context.Context, st ScheduledTask, taskRunner TaskRunner,
if err != nil {
return fmt.Errorf("restore failed: %w", err)
}
restoreOp.Status = summary
restoreOp.LastStatus = summary

return nil
}
2 changes: 1 addition & 1 deletion proto/v1/operations.proto
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ message OperationCheck {
message OperationRestore {
string path = 1; // path in the snapshot to restore.
string target = 2; // location to restore it to.
RestoreProgressEntry status = 3; // status of the restore.
RestoreProgressEntry last_status = 3; // status of the restore.
}

// OperationStats tracks a stats operation.
Expand Down
6 changes: 3 additions & 3 deletions webui/gen/ts/v1/operations_pb.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -620,9 +620,9 @@ export class OperationRestore extends Message<OperationRestore> {
/**
* status of the restore.
*
* @generated from field: v1.RestoreProgressEntry status = 3;
* @generated from field: v1.RestoreProgressEntry last_status = 3;
*/
status?: RestoreProgressEntry;
lastStatus?: RestoreProgressEntry;

constructor(data?: PartialMessage<OperationRestore>) {
super();
Expand All @@ -634,7 +634,7 @@ export class OperationRestore extends Message<OperationRestore> {
static readonly fields: FieldList = proto3.util.newFieldList(() => [
{ no: 1, name: "path", kind: "scalar", T: 9 /* ScalarType.STRING */ },
{ no: 2, name: "target", kind: "scalar", T: 9 /* ScalarType.STRING */ },
{ no: 3, name: "status", kind: "message", T: RestoreProgressEntry },
{ no: 3, name: "last_status", kind: "message", T: RestoreProgressEntry },
]);

static fromBinary(bytes: Uint8Array, options?: Partial<BinaryReadOptions>): OperationRestore {
Expand Down
4 changes: 3 additions & 1 deletion webui/src/components/OperationRow.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ export const OperationRow = ({
} else if (operation.op.case === "operationRestore") {
const restore = operation.op.value;
const progress = Math.round((details.percentage || 0) * 10) / 10;
const st = restore.status! || {};
const st = restore.lastStatus! || {};

body = (
<>
Expand Down Expand Up @@ -288,6 +288,8 @@ export const OperationRow = ({
</Button>
</>
) : null}
<br />
Snapshot ID: {normalizeSnapshotId(operation.snapshotId!)}
<Row gutter={16}>
<Col span={12}>
<Typography.Text strong>Bytes Done/Total</Typography.Text>
Expand Down
Loading

0 comments on commit 620caed

Please sign in to comment.