Skip to content

♻️ Golang utility for syncing between s3 and local, similar to `aws s3 sync`

License

Notifications You must be signed in to change notification settings

seqsense/s3sync

Repository files navigation

s3sync

CI codecov

Golang utility for syncing between s3 and local

Usage

Use New to create a manager, and Sync function syncs between s3 and local filesystem.

import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/seqsense/s3sync"
)

func main() {
  // Creates an AWS session
  sess, _ := session.NewSession(&aws.Config{
    Region: aws.String("us-east-1"),
  })

  syncManager := s3sync.New(sess)

  // Sync from s3 to local
  syncManager.Sync("s3://yourbucket/path/to/dir", "local/path/to/dir")

  // Sync from local to s3
  syncManager.Sync("local/path/to/dir", "s3://yourbucket/path/to/dir")

  // Sync from s3 to s3
  syncManager.Sync("s3://yourbucket/path/to/dir", "s3://anotherbucket/path/to/dir")
}

Sets the custom logger

You can set your custom logger.

import "github.com/seqsense/s3sync"

...
s3sync.SetLogger(&CustomLogger{})
...

The logger needs to implement Log and Logf methods. See the godoc for details.

Sets up the parallelism

You can configure the number of parallel jobs for sync. Default is 16.

s3sync.new(sess, s3sync.WithParallel(16)) // This is the same as default.
s3sync.new(sess, s3sync.WithParallel(1)) // You can sync one by one.

License

Apache 2.0 License. See LICENSE.