Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Requester Pays S3 Buckets #430

Closed
senorkrabs opened this issue Oct 19, 2020 · 4 comments
Closed

Support Requester Pays S3 Buckets #430

senorkrabs opened this issue Oct 19, 2020 · 4 comments
Assignees
@igorborgest
Labels
feature minor release Will be addressed in the next minor release ready to release
Milestone
2.4.0

Comments

@senorkrabs
Copy link

Is your idea related to a problem? Please describe.
It appears that the s3 methods in Data Wrangler don't support retrieving data in a bucket with Requester Pays enabled.

I verified the below code works non a bucket with Requester Pays disabled, but produces an AccessDenied error when it is enabled on the same bucket.

df = wr.s3.read_parquet('{}/{}'.format(path, "markets/")
ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Describe the solution you'd like
Add an argument to s3 methods to indicate/acknowledge requester pays in the S3 API request.

P.S. Don't attach files. Please, prefer add code snippets directly in the message body.
@senorkrabs senorkrabs added the enhancement New feature or request label Oct 19, 2020
@igorborgest igorborgest added feature help wanted Extra attention is needed and removed enhancement New feature or request labels Jan 27, 2021
@igorborgest igorborgest self-assigned this Jan 30, 2021
@igorborgest igorborgest added WIP Work in progress and removed help wanted Extra attention is needed labels Jan 30, 2021
@igorborgest igorborgest added this to the 2.4.0 milestone Jan 30, 2021
igorborgest added a commit that referenced this issue Jan 30, 2021
@igorborgest
Add support for requester pays on s3. #430
54a266c
@igorborgest igorborgest added minor release Will be addressed in the next minor release ready to release and removed WIP Work in progress labels Jan 30, 2021
@igorborgest
Copy link
Contributor

Hi @senorkrabs, thanks for reaching out.

I've just added support for requester pays on most functions/features we have (commit above). You just need to pass s3_additional_kwargs={"RequestPayer": "requester"} as argument.

e.g.

wr.s3.to_parquet(
    df=df,
    path=f"s3://{BUCKET}/test/",
    mode="overwrite",
    dataset=True,
    s3_additional_kwargs={"RequestPayer": "requester"}
)

df = wr.s3.read_parquet(
    path=f"s3://{BUCKET}/test/",
    dataset=True,
    s3_additional_kwargs={"RequestPayer": "requester"}
)

P.S. If you face some specific function still w/o resquester pays support, please open a new issue here o GitHub and we will address that.

P.P.S. It will be release on version 2.4.0 next week.

@igorborgest
Copy link
Contributor

Released on version 2.4.0.

@senorkrabs
Copy link
Author

Nice! Thanks for adding this! I recommend this package to everyone!

@vgkowski vgkowski mentioned this issue May 9, 2022
Closed
2 tasks
@tykishore-snowflake
Copy link

tykishore-snowflake commented Oct 12, 2023 •
edited
Loading

I am trying to integrate Sage Maker Data wrangler flow with S3 Bucket (Access point) and getting an error. I dont see an option to pass requestor pay parameter in datawrangler flow. Is it possible? Please note that Sage Maker is in a different account (Consumer) than the S3 Bucket ( Producer Account).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@igorborgest igorborgest

Labels
feature minor release Will be addressed in the next minor release ready to release
Projects
None yet
Milestone
2.4.0
Development

No branches or pull requests
3 participants
@igorborgest @senorkrabs @tykishore-snowflake