Skip to content
This repository has been archived by the owner on May 10, 2024. It is now read-only.

Unable to put large objects from Ireland EC2 to Ireland S3 until a day later. #1655

Closed
tbartold opened this issue Aug 6, 2013 · 4 comments · Fixed by #1731
Closed

Unable to put large objects from Ireland EC2 to Ireland S3 until a day later. #1655

tbartold opened this issue Aug 6, 2013 · 4 comments · Fixed by #1731
Assignees
@danielgtaylor
Labels
accepted needs-repro s3

Comments

@tbartold
Copy link

tbartold commented Aug 6, 2013

Steps to reproduce:

  1. create a bucket in Ireland using boto. e.g.
    python -c "import boto; conn = boto.connect_s3(); conn.create_bucket('eos-ireland-bucket-test', location=boto.s3.connection.Location.EU)"
  2. attempt to upload a file from Ireland using s3put:
    s3put -b eos-ireland-bucket-test /usr/lib/libselinux.so.1

This will fail.

Change the locations, etc. in the upload procedure, and it will succeed.

Wait to upload the file the following day, and it will succeed.
Upload a smaller file, and it will succeed.
Upload a file from us-east instead, and it will succeed.
Create the bucket in us-east instead, and it will succeed.
Also, specifying the 'host parameter' (--host s3-eu-west-1.amazonaws.com) on s3put will allow it to succeed:

Specifying the --host parameter is a successful workaround, but s3ut itself states, although it apparently works in this case with AWS S3 Ireland.
host - Hostname override, for using an endpoint other then AWS S3
@ghost ghost assigned danielgtaylor Aug 19, 2013
@danielgtaylor
Copy link
Member

This issue appears to be due to a combination of two issues:

  1. DNS propagation time
  2. The fact that s3put uses DNS for bucket location routing (the default behavior in boto)

As you mentioned, manually specifying a --host will mitigate the issue because it will bypass this DNS lookup and use the given host directly.

When using boto itself, you can actually specify a different calling format for S3 connections via the calling_format parameter. The default value is SubdomainCallingFormat but it can be set to OrdinaryCallingFormat, which uses s3.amazonaws.com/<bucket name> instead of the subdomain virtual host format. I'm investigating whether this solves the problem and whether it should be exposed via an argument to s3put.

@danielgtaylor
Copy link
Member

It appears that calling_format doesn't make a difference here. I'm investigating other options.

@danielgtaylor
Copy link
Member

This also appears to be related to #621

danielgtaylor added a commit to danielgtaylor/boto that referenced this issue Sep 17, 2013
@danielgtaylor
Attempt to auto-detect bucket region information and if no manual --h…
75d5c7b 
…ost or --region has been passed, set the region automatically so things just work. This fixes boto#1655.
@danielgtaylor danielgtaylor mentioned this issue Sep 17, 2013
Merged
danielgtaylor added a commit that referenced this issue Sep 17, 2013
@danielgtaylor
Merge pull request #1731 from danielgtaylor/s3put-fix
d9c28f6 
Fix s3put to auto-detect bucket region and act accordingly. Fixes #1731, #1655, and can act as an example around the issues in #621.
@dperetti
Copy link

dperetti commented Nov 3, 2014

This is still happening with both 2.34.
I'm having issues right now with every bucket newly created in Ireland. Works with a bucket created last week... and works with buckets created in default US region.
The broken pipe occurs in boto/s3/key.py > python ssl.py.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@danielgtaylor danielgtaylor

Labels
accepted needs-repro s3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix s3put to auto-detect bucket region and act accordingly danielgtaylor/boto
3 participants
@danielgtaylor @tbartold @dperetti