buffer_strategy = "disk" stops working after a while #16314
Assignees
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Recently I turned on
buffer_strategy = "disk"for an application writing line protocol into telegraf'sinputs.influxdb_v2_listenerand telegraf writing intooutputs.influxdb_v2.Background
At the time, I was using 1.32 and the setup was:
This was all working fine with
memory. After changing todiskthings worked for awhile, but then I started getting client errors of this form:There are no corresponding telegraf logs. The timeout is curious as it is all just under flush_interval which is set to
flush_interval = "10s".... (but see below)Once I set it back to
memory, the client errors went away.As an additional data point, the client was seeing some intermittent network issues which might cause disconnects or slow downs. My goal was to deploy telegraf as a proxy between my clients and InfluxDB with
buffer_strategy = "disk"as a safeguard so as not to lose points in the event of network errors and the like.Reproducer
In an effort to create a simple(r) reproducer, I created a python script to emulate a 'slow server':
Eg:
I then setup telegraf to listen on port 8086 and write out to the local server and use
curlin a loop to write out to the server. Specifically:In one terminal, start the server without any added delays (ie, it isn't 'slow' yet):
$ python3 ./slow-server.py -d 0Create a telegraf configuration to proxy and use disk buffer
create the disk buffer directory:
mkdir /tmp/tgstart telegraf with
telegraf --config ./telegraf.configdownload a line protocol generator:
go install github.com/influxdata/influxdb-comparisons/cmd/bulk_data_gen@latestthen send line protocol into telegraf in a loop. After a short while, telegraf starts outputting
curl: (52) Empty reply from serverinstead ofHTTP/1.1 204 No Content. Eg:With this setup, I can see files getting populated in
/tmp/tg. The clientcurlstarts out outputtingHTTP/1.1 204 No Contentbut after about a minute, I start seeingcurl: (52) Empty reply from server. I believe this is successfully demonstrating the problem. Interestingly, the python server is showing telegraf writing throughout (it sends 1000 points at a time (due tometric_batch_size = 1000); see the terminal output where the server is running).NOTE 1: once telegraf starts sending back empty responses, it does not recover even if I pause the client (ie, whenever the client resumes, telegraf will only send back empty responses). This is true whether telegraf is restarted or not. Only if I empty the
buffer_directorywill telegraf start sending back 204s (but it eventually only sends empty responses again)NOTE 2: in the above I slept 10 seconds after the curl. If I use other values (eg, 11 seconds, 20 seconds, 30 seconds all after starting with an empty
buffer_directory), I see the same thing. With a 10 second sleep, it took ~1 minute before seeing the empty responses. With 20 second sleep it took ~2 minutes before seeing empty responses. Since I'm sending 9999 at a time with curl (and telegraf is breaking that up into 1000 chunks), the number of metrics might be a factor (eg, ~60,000 metrics before it starts to show the problem).NOTE 3: with the above, I haven't introduced 'slowness' with either
-for-d, but I see similar issues when doing so (but might seedid not complete within its flush intervalfrom telegraf depending on how slow I make the server). I provided the full script in case it is useful for other debugging and developing a fix for this issue.The text was updated successfully, but these errors were encountered: