Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements AWS SigV4 for the HTTP output plugin. #4459

Merged
merged 4 commits into from
Apr 5, 2024
Merged

Implements AWS SigV4 for the HTTP output plugin. #4459

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f717320
Implements AWS SigV4 for the HTTP output plugin. Resolves #4444.
dlvenable Apr 1, 2024
4375321
Make the AWS dependencies optional requirements and only require them…
dlvenable Apr 3, 2024
e4c2044
Remove the Host header from HTTP output requests.
dlvenable Apr 3, 2024
c26d8e7
Addressed PR comments - update the gem installation to allow for more…
dlvenable Apr 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make the AWS dependencies optional requirements and only require them… 
… as needed for SigV4 authentication.

Signed-off-by: David Venable <dlv@amazon.com>
  • Loading branch information
@dlvenable
dlvenable committed Apr 3, 2024
commit 43753219e215a9e8da53d2fd57000d687dfbc7fe
6 changes: 3 additions & 3 deletions fluentd.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ Gem::Specification.new do |gem|
gem.add_runtime_dependency("tzinfo-data", ["~> 1.0"])
gem.add_runtime_dependency("strptime", [">= 0.2.4", "< 1.0.0"])
gem.add_runtime_dependency("webrick", ["~> 1.4"])
gem.add_runtime_dependency("aws-sigv4", ["~> 1.8"])
gem.add_runtime_dependency("aws-sdk-sts", ["~> 1.11"])
gem.add_runtime_dependency("rexml", ["~> 3.2"])

# gems that aren't default gems as of Ruby 3.4
gem.add_runtime_dependency("base64", ["~> 0.2"])
Expand Down Expand Up @@ -59,4 +56,7 @@ Gem::Specification.new do |gem|
gem.add_development_dependency("oj", [">= 2.14", "< 4"])
gem.add_development_dependency("async", "~> 1.23")
gem.add_development_dependency("async-http", ">= 0.50.0")
gem.add_development_dependency("aws-sigv4", ["~> 1.8"])
gem.add_development_dependency("aws-sdk-core", ["~> 3.191"])
gem.add_development_dependency("rexml", ["~> 3.2"])
end
34 changes: 22 additions & 12 deletions lib/fluent/plugin/out_http.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@
require 'fluent/tls'
require 'fluent/plugin/output'
require 'fluent/plugin_helper/socket'
require 'aws-sigv4'
require 'aws-sdk-core'

# patch Net::HTTP to support extra_chain_cert which was added in Ruby feature #9758.
# see: https://github.com/ruby/ruby/commit/31af0dafba6d3769d2a39617c0dddedb97883712
Expand Down Expand Up @@ -131,6 +129,12 @@ def configure(conf)
end

if @auth and @auth.method == :aws_sigv4
begin
require 'aws-sigv4'
require 'aws-sdk-core'
rescue LoadError
raise Fluent::ConfigError, "The aws-sigv4 and aws-sdk-core gems are required for aws_sigv4 auth. Run: 'gem install aws-sigv4 -v 1.8.0' and 'gem install aws-sdk-core -v 3.191'."
ashie marked this conversation as resolved.
Show resolved Hide resolved
end
cosmo0920 marked this conversation as resolved.
Show resolved Hide resolved

raise Fluent::ConfigError, "aws_service is required for aws_sigv4 auth" unless @auth.aws_service != nil
raise Fluent::ConfigError, "aws_region is required for aws_sigv4 auth" unless @auth.aws_region != nil
Expand Down Expand Up @@ -262,16 +266,7 @@ def set_headers(req, uri, chunk)
req['Host'] = uri.host
ashie marked this conversation as resolved.
Show resolved Hide resolved
end

def create_request(chunk, uri)
req = case @http_method
when :post
Net::HTTP::Post.new(uri.request_uri)
when :put
Net::HTTP::Put.new(uri.request_uri)
end
set_headers(req, uri, chunk)
req.body = @json_array ? "[#{chunk.read.chop}]" : chunk.read

def set_auth(req, uri)
if @auth
if @auth.method == :basic
req.basic_auth(@auth.username, @auth.password)
Expand All @@ -291,9 +286,24 @@ def create_request(chunk, uri)
req.add_field('authorization', signature.headers['authorization'])
end
end
end

def create_request(chunk, uri)
req = case @http_method
when :post
Net::HTTP::Post.new(uri.request_uri)
when :put
Net::HTTP::Put.new(uri.request_uri)
end
set_headers(req, uri, chunk)
req.body = @json_array ? "[#{chunk.read.chop}]" : chunk.read

# At least one authentication method requires the body and other headers, so the order of this call matters
set_auth(req, uri)
req
end


def send_request(uri, req)
res = if @proxy_uri
Net::HTTP.start(uri.host, uri.port, @proxy_uri.host, @proxy_uri.port, @proxy_uri.user, @proxy_uri.password, @http_opt) { |http|
Expand Down