Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace whitelist and dummy terminology #3065

Merged
merged 4 commits into from
Jul 7, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions example/copy_roundrobin.conf
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
<source>
@type dummy
@type sample
@label @test
tag test.copy
auto_increment_key id
</source>

<source>
@type dummy
@type sample
@label @test
tag test.rr
auto_increment_key id
Expand Down Expand Up @@ -36,4 +36,4 @@
output_type ltsv
</store>
</match>
</label>
</label>
2 changes: 1 addition & 1 deletion example/counter.conf
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
</system>

<source>
@type dummy
@type sample
tag "test.data"
auto_increment_key number
</source>
Expand Down
4 changes: 2 additions & 2 deletions example/filter_stdout.conf
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<source>
@type dummy
tag dummy
@type sample
tag sample
</source>

<filter **>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
<source>
@type dummy
tag dummy
@type sample
tag sample
rate 100
dummy {"message":"yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay"}
sample {"message":"yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay"}
</source>

<match dummy>
<match sample>
@type null
never_flush true
<buffer>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
<source>
@type dummy
@type sample
@label @main
tag "test.data"
size 2
rate 10
dummy {"message":"yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay"}
sample {"message":"yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaay"}
auto_increment_key number
</source>

<label @main>
<match test.data>
@type buffered_stdout
@type stdout
<buffer>
@type file
path "#{Dir.pwd}/compressed_buffers"
Expand Down
10 changes: 5 additions & 5 deletions example/logevents.conf
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
<source>
@type dummy
@label @dummylog
@type sample
@label @samplelog
tag "data"
dummy {"message":"yay"}
sample {"message":"yay"}
</source>
<label @dummylog>
<label @samplelog>
<match **>
@type stdout
</match>
Expand All @@ -22,4 +22,4 @@
# hostname_key "host"
# </inject>
# </match>
</label>
</label>
2 changes: 1 addition & 1 deletion example/multi_filters.conf
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# This example is to measure optimized filter pipeline performance.

<source>
@type dummy
@type sample
tag test
size 1000
</source>
Expand Down
4 changes: 2 additions & 2 deletions example/out_exec_filter.conf
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
<source>
@type dummy
@type sample
@label @exec
tag exec_input
rate 10
auto_increment_key num
dummy {"data":"mydata"}
sample {"data":"mydata"}
</source>

<label @exec>
Expand Down
2 changes: 1 addition & 1 deletion example/out_forward.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<source>
@type dummy
@type sample
tag test
</source>

Expand Down
2 changes: 1 addition & 1 deletion example/out_forward_buf_file.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<source>
@type dummy
@type sample
tag test
</source>

Expand Down
10 changes: 5 additions & 5 deletions example/out_forward_client.conf
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
<source>
@type dummy
@type sample
tag test
</source>
<source>
@type dummy
@type sample
tag test2
</source>
<source>
@type dummy
@type sample
tag test3
</source>
<source>
@type dummy
@type sample
tag test4
</source>
<source>
@type dummy
@type sample
tag test5
</source>

Expand Down
2 changes: 1 addition & 1 deletion example/out_forward_heartbeat_none.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<source>
@type dummy
@type sample
tag test
</source>

Expand Down
2 changes: 1 addition & 1 deletion example/out_forward_sd.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<source>
@type dummy
@type sample
tag test
</source>

Expand Down
4 changes: 2 additions & 2 deletions example/out_forward_shared_key.conf
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
<source>
@type dummy
@type sample
tag test
</source>
<source>
@type dummy
@type sample
tag test2
</source>

Expand Down
2 changes: 1 addition & 1 deletion example/out_forward_tls.conf
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<source>
@type dummy
@type sample
tag test
</source>

Expand Down
6 changes: 3 additions & 3 deletions example/out_forward_users.conf
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<source>
@type dummy
@type sample
tag test
</source>
<source>
@type dummy
@type sample
tag test2
</source>
<source>
@type dummy
@type sample
tag test3
</source>

Expand Down
8 changes: 4 additions & 4 deletions example/out_null.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@
# bundle exec bin/fluentd -c example/out_buffered_null.conf
# (+ --emit-error-log-interval 10)
<source>
@type dummy
tag dummy
@type sample
tag sample
rate 500000000
dummy [
sample [
{"message": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
{"message": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"},
{"message": "ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"}
]
</source>

<match dummy.**>
<match sample.**>
@type null
<buffer>
flush_interval 60s
Expand Down
2 changes: 1 addition & 1 deletion example/secondary_file.conf
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
</system>

<source>
@type dummy
@type sample
tag test
</source>

Expand Down
125 changes: 2 additions & 123 deletions lib/fluent/plugin/in_dummy.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,126 +14,5 @@
# limitations under the License.
#

require 'json'

require 'fluent/plugin/input'
require 'fluent/config/error'

module Fluent::Plugin
class DummyInput < Input
Fluent::Plugin.register_input('dummy', self)

helpers :thread, :storage

BIN_NUM = 10
DEFAULT_STORAGE_TYPE = 'local'

desc "The value is the tag assigned to the generated events."
config_param :tag, :string
desc "The number of events in event stream of each emits."
config_param :size, :integer, default: 1
desc "It configures how many events to generate per second."
config_param :rate, :integer, default: 1
desc "If specified, each generated event has an auto-incremented key field."
config_param :auto_increment_key, :string, default: nil
desc "The boolean to suspend-and-resume incremental value after restart"
config_param :suspend, :bool, default: false,deprecated: 'This parameters is ignored'
desc "The dummy data to be generated. An array of JSON hashes or a single JSON hash."
config_param :dummy, default: [{"message"=>"dummy"}] do |val|
begin
parsed = JSON.parse(val)
rescue JSON::ParserError => ex
# Fluent::ConfigParseError, "got incomplete JSON" will be raised
# at literal_parser.rb with --use-v1-config, but I had to
# take care at here for the case of --use-v0-config.
raise Fluent::ConfigError, "#{ex.class}: #{ex.message}"
end
dummy = parsed.is_a?(Array) ? parsed : [parsed]
dummy.each_with_index do |e, i|
raise Fluent::ConfigError, "#{i}th element of dummy, #{e}, is not a hash" unless e.is_a?(Hash)
end
dummy
end

def initialize
super
@storage = nil
end

def configure(conf)
super
@dummy_index = 0
config = conf.elements.select{|e| e.name == 'storage' }.first
@storage = storage_create(usage: 'suspend', conf: config, default_type: DEFAULT_STORAGE_TYPE)
end

def multi_workers_ready?
true
end

def start
super

@storage.put(:increment_value, 0) unless @storage.get(:increment_value)
@storage.put(:dummy_index, 0) unless @storage.get(:dummy_index)

if @auto_increment_key && !@storage.get(:auto_increment_value)
@storage.put(:auto_increment_value, -1)
end

thread_create(:dummy_input, &method(:run))
end

def run
batch_num = (@rate / BIN_NUM).to_i
residual_num = (@rate % BIN_NUM)
while thread_current_running?
current_time = Time.now.to_i
BIN_NUM.times do
break unless (thread_current_running? && Time.now.to_i <= current_time)
wait(0.1) { emit(batch_num) }
end
emit(residual_num) if thread_current_running?
# wait for next second
while thread_current_running? && Time.now.to_i <= current_time
sleep 0.01
end
end
end

def emit(num)
begin
if @size > 1
num.times do
router.emit_array(@tag, Array.new(@size) { [Fluent::EventTime.now, generate] })
end
else
num.times { router.emit(@tag, Fluent::EventTime.now, generate) }
end
rescue => _
# ignore all errors not to stop emits by emit errors
end
end

def generate
d = @dummy[@dummy_index]
unless d
@dummy_index = 0
d = @dummy[@dummy_index]
end
@dummy_index += 1
if @auto_increment_key
d = d.dup
d[@auto_increment_key] = @storage.update(:auto_increment_value){|v| v + 1 }
end
d
end

def wait(time)
start_time = Time.now
yield
sleep_time = time - (Time.now - start_time)
sleep sleep_time if sleep_time > 0
end
end
end
# Remove this file in fluentd v2
require_relative 'in_sample'
4 changes: 2 additions & 2 deletions lib/fluent/plugin/in_http.rb
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ class HttpInput < Input
desc 'Add REMOTE_ADDR header to the record.'
config_param :add_remote_addr, :bool, default: false
config_param :blocking_timeout, :time, default: 0.5
desc 'Set a white list of domains that can do CORS (Cross-Origin Resource Sharing)'
desc 'Set a allow list of domains that can do CORS (Cross-Origin Resource Sharing)'
config_param :cors_allow_origins, :array, default: nil
desc 'Respond with empty gif image of 1x1 pixel.'
config_param :respond_with_empty_img, :bool, default: false
Expand Down Expand Up @@ -490,7 +490,7 @@ def on_message_complete
# CORS check
# ==========
# For every incoming request, we check if we have some CORS
# restrictions and white listed origins through @cors_allow_origins.
# restrictions and allow listed origins through @cors_allow_origins.
unless @cors_allow_origins.nil?
unless @cors_allow_origins.include?('*') or include_cors_allow_origin
send_response_and_close(RES_403_STATUS, {'Connection' => 'close'}, "")
Expand Down
Loading