Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPE in requesterPays check if credentials don't match provided user project #849

Closed
lbergelson opened this issue Mar 8, 2022 · 0 comments · Fixed by #850
Closed

NPE in requesterPays check if credentials don't match provided user project #849

lbergelson opened this issue Mar 8, 2022 · 0 comments · Fixed by #850
Assignees
@cojenco
Labels
api: storage Issues related to the googleapis/java-storage-nio API.

Comments

@lbergelson
Copy link
Contributor

The last release introduced a new NullPointerException when trying to use an invalid service account to access a requester pays bucket. The null pointer hides the actual cause of the error which makes it hard to debug. (Although the cause is not particularly clear from the original error message anyway)

Expect to see something like this:

Error getting access token for service account: 400 Bad Request
POST https://oauth2.googleapis.com/token
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}, iss: 806222273987-uilktks3j6i7962rp0v7nusveer58497@developer.gserviceaccount.com
com.google.cloud.storage.StorageException: Error getting access token for service account: 400 Bad Request
....

But instead you get


java.lang.NullPointerException
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.requesterPays(CloudStorageFileSystemProvider.java:970)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.<init>(CloudStorageFileSystem.java:209)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:196)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:164)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:154)
	at org.broadinstitute.hellbender.utils.gcs.BucketUtils.getPathOnGcs(BucketUtils.java:410)
	at org.broadinstitute.hellbender.utils.io.IOUtils.getPath(IOUtils.java:773)
	at org.broadinstitute.hellbender.utils.gcs.BucketUtilsUnitTest.throwNPE(BucketUtilsUnitTest.java:421)

Environment details

  1. Cloud NIO
  2. OS type and version: all
  3. Java version: all
  4. version(s): 0.123.23

Steps to reproduce

  1. Attempt to check requester pays status of a bucket under these conditions:
    • correctly formatted but revoked service account credentials
    • autoDetectRequesterPays = true
  2. NPE

Code example

    @Test
    public void throwNPE() throws IOException {
            CloudStorageFileSystemProvider.setDefaultCloudStorageConfiguration(
                    CloudStorageConfiguration.builder()
                    .autoDetectRequesterPays(true)
                    .userProject("ANY_PROJECT_NAME")
                    .build());
            CloudStorageFileSystemProvider.setStorageOptions(
                    StorageOptions.newBuilder()
                            .setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream(INVALID_KEY)))
                            .build()
            );
            final Path path = IOUtils.getPath(FILE_IN_REQUESTER_PAYS_BUCKET);
    }

Stack trace

java.lang.NullPointerException
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.requesterPays(CloudStorageFileSystemProvider.java:970)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.<init>(CloudStorageFileSystem.java:209)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:196)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:164)
	at com.google.cloud.storage.contrib.nio.CloudStorageFileSystem.forBucket(CloudStorageFileSystem.java:154)
	at org.broadinstitute.hellbender.utils.gcs.BucketUtils.getPathOnGcs(BucketUtils.java:410)
	at org.broadinstitute.hellbender.utils.io.IOUtils.getPath(IOUtils.java:773)
	at org.broadinstitute.hellbender.utils.gcs.BucketUtilsUnitTest.throwNPE(BucketUtilsUnitTest.java:421)
	...
@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/java-storage-nio API. label Mar 8, 2022
lbergelson added a commit to lbergelson/java-storage-nio that referenced this issue Mar 8, 2022
@lbergelson
fix: prevent NPE when checking requester pays status
de10ff9 
Fixes a NullPointerException introduced in googleapis#841 which occured when trying
to check the the requesterPays status of a file while authenticated with an
invalid service account.

Refs: googleapis#849, googleapis#841
@lbergelson lbergelson mentioned this issue Mar 8, 2022
Merged
4 tasks
@cojenco cojenco self-assigned this Mar 9, 2022
cojenco pushed a commit that referenced this issue Mar 9, 2022
@lbergelson
fix: prevent NPE when checking requester pays status (#850)
ce50209 
Fixes a NullPointerException introduced in #841 which occured when trying
to check the the requesterPays status of a file while authenticated with an
invalid service account.

Refs: #849, #841
lbergelson added a commit to broadinstitute/gatk that referenced this issue Mar 18, 2022
@lbergelson
Fix requester pays access harder. #7716
bd176ad 
* The previous attempt to fix requester pays didn't fix it in many cases.
This incorporates a newer version of the NIO library with several patches to fix
edge cases we were hitting.
  * googleapis/java-storage-nio#849
  * googleapis/java-storage-nio#856
  * googleapis/java-storage-nio#857
* upgrade com.google.cloud:google-cloud-nio:0.123.23 ->0.123.25
* fixes #7716
lbergelson added a commit to broadinstitute/gatk that referenced this issue Mar 18, 2022
@lbergelson
Fix requester pays access harder. #7716
e7c0add 
* The previous attempt to fix requester pays didn't fix it in many cases.
This incorporates a newer version of the NIO library with several patches to fix
edge cases we were hitting.
  * googleapis/java-storage-nio#849
  * googleapis/java-storage-nio#856
  * googleapis/java-storage-nio#857
* upgrade com.google.cloud:google-cloud-nio:0.123.23 ->0.123.25
* fixes #7716
@lbergelson lbergelson mentioned this issue Mar 18, 2022
Merged
droazen added a commit to broadinstitute/gatk that referenced this issue Apr 7, 2022
@lbergelson @droazen
Fix Google Cloud requester pays access harder. #7716 (#7730)
9d4fe23 
* The previous attempt to fix requester pays didn't fix it in many cases.
This incorporates a newer version of the NIO library with several patches to fix
edge cases we were hitting.
  * googleapis/java-storage-nio#849
  * googleapis/java-storage-nio#856
  * googleapis/java-storage-nio#857
* upgrade com.google.cloud:google-cloud-nio:0.123.23 ->0.123.25

fixes #7716

Co-authored-by: David Roazen <droazen@broadinstitute.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cojenco cojenco

Labels
api: storage Issues related to the googleapis/java-storage-nio API.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: prevent NPE when checking requester pays status lbergelson/java-storage-nio
2 participants
@lbergelson @cojenco