[Packetbeat] Restrict max buffer size before send to logstash #516
Closed
Description
When running Packetbeat into MySQL server, the server was memory growth (10.834GB)
when logstash is down and Packetbeat supports buffering the events for waiting to send
the memory is over and kills mysql process.
The data collected by topbeat of the packetbeat process
Time | proc.name | proc.mem.size |
---|---|---|
11-12-2015 10:19:04 | packetbeat | 8.011GB |
11-12-2015 10:19:14 | packetbeat | 8.041GB |
11-12-2015 10:19:24 | packetbeat | 8.041GB |
11-12-2015 10:19:34 | packetbeat | 8.041GB |
11-12-2015 10:19:44 | packetbeat | 8.041GB |
11-12-2015 10:19:54 | packetbeat | 8.159GB |
11-12-2015 10:20:04 | packetbeat | 8.159GB |
11-12-2015 10:20:14 | packetbeat | 8.159GB |
11-12-2015 10:20:24 | packetbeat | 8.159GB |
11-12-2015 10:20:34 | packetbeat | 8.216GB |
11-12-2015 10:20:44 | packetbeat | 8.216GB |
11-12-2015 10:20:54 | packetbeat | 8.216GB |
11-12-2015 10:21:04 | packetbeat | 8.218GB |
11-12-2015 10:21:14 | packetbeat | 8.229GB |
11-12-2015 10:21:24 | packetbeat | 8.33GB |
11-12-2015 10:21:34 | packetbeat | 8.33GB |
11-12-2015 10:21:44 | packetbeat | 8.33GB |
11-12-2015 10:21:54 | packetbeat | 8.33GB |
11-12-2015 10:22:04 | packetbeat | 8.332GB |
11-12-2015 10:22:14 | packetbeat | 8.451GB |
11-12-2015 10:22:24 | packetbeat | 8.451GB |
11-12-2015 10:22:34 | packetbeat | 8.451GB |
11-12-2015 10:22:44 | packetbeat | 8.451GB |
11-12-2015 10:22:54 | packetbeat | 8.755GB |
11-12-2015 10:23:04 | packetbeat | 8.755GB |
11-12-2015 10:23:14 | packetbeat | 8.755GB |
11-12-2015 10:23:24 | packetbeat | 8.755GB |
11-12-2015 10:23:34 | packetbeat | 8.755GB |
11-12-2015 10:23:44 | packetbeat | 9.016GB |
11-12-2015 10:23:54 | packetbeat | 9.016GB |
11-12-2015 10:24:04 | packetbeat | 9.016GB |
11-12-2015 10:24:14 | packetbeat | 9.016GB |
11-12-2015 10:24:24 | packetbeat | 9.016GB |
11-12-2015 10:24:34 | packetbeat | 9.266GB |
11-12-2015 10:24:44 | packetbeat | 9.266GB |
11-12-2015 10:24:54 | packetbeat | 9.266GB |
11-12-2015 10:25:04 | packetbeat | 9.266GB |
11-12-2015 10:25:14 | packetbeat | 9.266GB |
11-12-2015 10:25:24 | packetbeat | 9.266GB |
11-12-2015 10:25:34 | packetbeat | 9.372GB |
11-12-2015 10:25:44 | packetbeat | 9.381GB |
11-12-2015 10:25:54 | packetbeat | 9.381GB |
11-12-2015 10:26:04 | packetbeat | 9.381GB |
11-12-2015 10:26:14 | packetbeat | 9.381GB |
11-12-2015 10:26:24 | packetbeat | 9.381GB |
11-12-2015 10:26:34 | packetbeat | 9.381GB |
11-12-2015 10:26:44 | packetbeat | 9.587GB |
11-12-2015 10:26:54 | packetbeat | 9.587GB |
11-12-2015 10:27:04 | packetbeat | 9.588GB |
11-12-2015 10:27:14 | packetbeat | 9.588GB |
11-12-2015 10:27:24 | packetbeat | 9.588GB |
11-12-2015 10:27:34 | packetbeat | 9.588GB |
11-12-2015 10:27:44 | packetbeat | 9.993GB |
11-12-2015 10:27:54 | packetbeat | 9.993GB |
11-12-2015 10:28:04 | packetbeat | 9.993GB |
11-12-2015 10:28:14 | packetbeat | 9.993GB |
11-12-2015 10:28:24 | packetbeat | 9.993GB |
11-12-2015 10:28:34 | packetbeat | 9.993GB |
11-12-2015 10:28:44 | packetbeat | 9.993GB |
11-12-2015 10:28:56 | packetbeat | 10.313GB |
11-12-2015 10:29:04 | packetbeat | 10.34GB |
11-12-2015 10:29:15 | packetbeat | 10.34GB |
11-12-2015 10:29:25 | packetbeat | 10.34GB |
11-12-2015 10:29:34 | packetbeat | 10.34GB |
11-12-2015 10:29:44 | packetbeat | 10.34GB |
11-12-2015 10:29:54 | packetbeat | 10.34GB |
11-12-2015 10:30:04 | packetbeat | 10.34GB |
11-12-2015 10:30:14 | packetbeat | 10.34GB |
11-12-2015 10:30:24 | packetbeat | 10.379GB |
11-12-2015 10:30:34 | packetbeat | 10.786GB |
11-12-2015 10:30:44 | packetbeat | 10.834GB |
11-12-2015 10:30:54 | packetbeat | 10.834GB |
11-12-2015 10:31:04 | packetbeat | 105.926MB |
This point MySQL was restarted after memory is over
Using:
Centos 6.6 2.6.32-504.8.1.el6.x86_64
packetbeat version 1.0.0 (amd64)
logstash 1.5.6
Topology:
Packetbeat -> Logstash -> Elasticsearch
Configuration
interfaces:
device: any
protocols:
dns:
include_authorities: true
include_additionals: true
mysql:
ports: [3306]
logstash:
hosts: ["localhost:5044"]
I think the Packetbeat it shoulds a setting to prevent max memory consumption for buffering