Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility layers for v0.12 plugins #912

Merged
merged 22 commits into from
May 17, 2016
Merged

Compatibility layers for v0.12 plugins #912

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0c50578
fix test with added (missing) error definition
tagomoris Apr 27, 2016
51a1a24
Add Base#inspect to show this object in simple/short way
tagomoris Apr 21, 2016
e5769f7
add fluent/compat/filter and make Fluent::Filter with it
tagomoris Apr 21, 2016
a1d6754
add fluent/compat/input and make Fluent::Input with it
tagomoris Apr 21, 2016
bf3dbcc
add comments about time handling
tagomoris Apr 25, 2016
d5e90de
fix to extend chunk to add method for event iteration
tagomoris Apr 25, 2016
9714328
remove fileutils
tagomoris Apr 27, 2016
d096726
add optimization
tagomoris Apr 27, 2016
8c07b0c
take care about NaN & Inf
tagomoris Apr 27, 2016
0688ba0
add buffer path pattern only for suffix
tagomoris Apr 27, 2016
a5a6b98
initialize minimal internal things as early as possible for tests
tagomoris Apr 27, 2016
5e93092
Add compatibility layers for output plugin
tagomoris Apr 27, 2016
33264de
fix tests for core modules/classes with newer API
tagomoris Apr 27, 2016
a52e4d1
fix to use test driver for buffered output
tagomoris Apr 27, 2016
b70b508
re-implement symlink_path feature on v0.14 buffer APIs
tagomoris Apr 27, 2016
fc2c76d
add break condition to wait checks to run
tagomoris Apr 28, 2016
903f4c2
update comments for correctness
tagomoris May 9, 2016
403cf5f
add deprecation warning for "type" in secondary section
tagomoris May 9, 2016
e0afba5
fix to use more explicit name
tagomoris May 9, 2016
dc9fe04
fix tests for exponential_backoff
tagomoris May 9, 2016
0785ec5
for readability
tagomoris May 9, 2016
a424505
remove/merge dup code
tagomoris May 9, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
re-implement symlink_path feature on v0.14 buffer APIs
  • Loading branch information
@tagomoris
tagomoris committed May 13, 2016
commit b70b5082175af2183035fb6787747afe08816d85
23 changes: 22 additions & 1 deletion lib/fluent/plugin/out_file.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,21 @@ class FileOutput < TimeSlicedOutput
desc "Create symlink to temporary buffered file when buffer_type is file."
config_param :symlink_path, :string, default: nil

module SymlinkBufferMixin
def symlink_path=(path)
@_symlink_path = path
end

def generate_chunk(metadata)
chunk = super
latest_chunk = metadata_list.sort_by(&:timekey).last
if chunk.metadata == latest_chunk
FileUtils.ln_sf(chunk.path, @_symlink_path)
end
chunk
end
end

def initialize
require 'zlib'
require 'time'
Expand Down Expand Up @@ -87,7 +102,13 @@ def configure(conf)
@formatter = Plugin.new_formatter(@format)
@formatter.configure(conf)

@buffer.symlink_path = @symlink_path if @symlink_path
if @symlink_path && @buffer.respond_to?(:path)
(class << @buffer; self; end).module_eval do
prepend SymlinkBufferMixin
Copy link
Member

@repeatedly repeatedly Apr 28, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prepend doesn't work on Ruby 1.9 so this code should not be backported to v0.12, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't add label v0.12 on this pull-request... and this hack is required because Buffer plugin of v0.14 API doesn't have symbolic link feature.
There's no reason to backport this change to v0.12.

Copy link
Member

@repeatedly repeatedly Apr 28, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is this plugin is also used in v0.12.
So if someone writes the patch to v0.12 / v0.14 out_file, v0.14 changes may conflict.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conflicts may occur if someone want to change L90 (in previous version), but any other changes will not conflict this change.

end
@buffer.symlink_path = @symlink_path
end

@dir_perm = system_config.dir_permission || DIR_PERMISSION
@file_perm = system_config.file_permission || FILE_PERMISSION
end
Expand Down
11 changes: 8 additions & 3 deletions test/plugin/test_out_file.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,17 +306,22 @@ def test_write_with_symlink
begin
d.instance.start
10.times { sleep 0.05 }

time = Time.parse("2011-01-02 13:14:15 UTC").to_i
es = Fluent::OneEventStream.new(time, {"a"=>1})
d.instance.emit('tag', es, Fluent::NullOutputChain.instance)
d.instance.emit_events('tag', es)

assert File.exist?(symlink_path)
assert File.symlink?(symlink_path)

d.instance.enqueue_buffer
es = Fluent::OneEventStream.new(event_time("2011-01-03 14:15:16 UTC"), {"a"=>2})
d.instance.emit_events('tag', es)

assert !File.exist?(symlink_path)
assert File.exist?(symlink_path)
assert File.symlink?(symlink_path)

meta = d.instance.metadata('tag', event_time("2011-01-03 14:15:16 UTC"), {})
assert_equal d.instance.buffer.instance_eval{ @stage[meta].path }, File.readlink(symlink_path)
ensure
d.instance.shutdown
FileUtils.rm_rf(symlink_path)
Expand Down