2018-03-19 (GCS 1.6.4, BQ 0.10.5)
Changelog
Cloud Storage connector:
-
Fixed an issue where JSON auth files containing user auth (e.g.
application_default_credentials.json
) does not work withgoogle.cloud.auth.service.account.json.keyfile
. -
Honor
GOOGLE_APPLICATION_DEFAULT_CREDENTIALS
environment variable. For Google Application Default Credentials (but not other defaults). -
Make
fs.gs.project.id
optional. It is still required for listing buckets, creating buckets, and entire BigQuery connector. -
Disable GCS Metadata Cache by default (e.g. set default value of
fs.gs.metadata.cache.enable
property tofalse
). -
Support GCS Requester Pays feature that could be configured with new properties:
fs.gs.requester.pays.mode (default=DISABLED) fs.gs.requester.pays.project.id (no default value) fs.gs.requester.pays.buckets (no default value)
-
Add support for specifying marker files pattern that should be copied last during folder rename operation. Pattern is configured with property:
fs.gs.marker.file.pattern (no default value)
BigQuery connector:
- POM updates for GCS connector 1.6.4.
- Remove Avro and Gson classes from Hadoop 2 shaded jar because they are already included in the Hadoop 2 distribution.