Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

command/cp: add cross-region transfer support #194

Merged
merged 28 commits into from
Oct 22, 2020
Merged

Conversation

fbarotov
Copy link
Contributor

Resolves #155

command/app.go Outdated Show resolved Hide resolved
command/app.go Outdated Show resolved Hide resolved
command/cat.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
command/cp.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
Copy link
Member

@sonmezonur sonmezonur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should add multiple e2e tests

storage/s3.go Outdated Show resolved Hide resolved
command/run.go Outdated Show resolved Hide resolved
command/cp.go Outdated Show resolved Hide resolved
command/app.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
command/app.go Outdated Show resolved Hide resolved
command/cp.go Outdated Show resolved Hide resolved
command/mv.go Outdated Show resolved Hide resolved
storage/s3_test.go Show resolved Hide resolved
storage/s3_test.go Outdated Show resolved Hide resolved
storage/s3.go Show resolved Hide resolved
command/cp.go Outdated Show resolved Hide resolved
@igungor igungor changed the title command/{app, cp}, storage/{s3, s3_test}: added cross-region transfer support command/cp: add cross-region transfer support Aug 7, 2020
command/app.go Outdated Show resolved Hide resolved
command/cp.go Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
@sonmezonur
Copy link
Member

@fbarotov Hey ✋ Sorry for the late feedback. I personally found this approach better than #209 because of 2 reasons:

  1. Getting region information with API call is almost costless. It parses region information from the response after sending HEAD request (ref). I wrote simple program to benchmark GetBucketRegion call:
AWS_PROFILE=personal hyperfine --show-output --min-runs 20 './testgetregion --bucket asiaonurtest >/dev/null'
Benchmark #1: ./testgetregion --bucket asiaonurtest >/dev/null
 Time (mean ± σ):     722.0 ms ±  38.0 ms    [User: 57.9 ms, System: 23.6 ms]
 Range (min … max):   690.4 ms … 833.6 ms    20 runs

I think it is quite negligible since we make this call once and use the cached session afterward.

  1. IMHO the code will be more readable on this PR. In command/cp: added cross-region transfer support #209, each command needs to parse and handle retryable errors separately (e.g. 1, 2). While adding a new command to s5cmd, we also need to handle retryable errors in order to get region info. If we use this approach and get region info first, it will prevent nested error handling patterns.

What do you think? Dou you want to work on this PR and resolve conflicts before we conclude which implementation is better? 😸

@fbarotov
Copy link
Contributor Author

@fbarotov Hey ✋ Sorry for the late feedback. I personally found this approach better than #209 because of 2 reasons:
...

Hi @sonmezonur 🖐

Thanks a lot for the feedback.

Awesome, I also ran your program on my laptop and got the following results:

  1. when the default session and bucket have the same regions: 411.65 +- 303.64 ms
  2. when the default session and bucket have different regions: 597.10 +- 528.81 ms

indeed, as you mentioned they are negligible.

Yes, the approach in this pr is more readable in my opinion as we would be having more loose coupling between session logic and operations logic.

Let me try to resolve the conflicts :)

@fbarotov fbarotov requested a review from a team as a code owner September 30, 2020 23:29
@fbarotov fbarotov requested review from igungor and aykutfarsak and removed request for a team September 30, 2020 23:29
storage/storage.go Outdated Show resolved Hide resolved
storage/s3.go Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
command/mv.go Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
storage/s3.go Show resolved Hide resolved
storage/s3.go Outdated Show resolved Hide resolved
Copy link
Member

@igungor igungor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pursuing this.

CHANGELOG.md Outdated Show resolved Hide resolved
@igungor igungor merged commit 2dad7e1 into peak:master Oct 22, 2020
igungor added a commit that referenced this pull request Oct 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add cross region transfer support
4 participants