Smart and lightweight Log Parser and Log Shipper written in Node. It can ship logs to Elasticsearch and thus also to Logsene. See Documentation.
This is the repository for logagent v2.x, currently in beta.
- See logagent 2.x features and status
- Please use logagent-js 1.x for production. Source code for logagent-js 1.x
- new config file format (YAML)
- support for multiple pattern definition files (including hot reload)
- log routing to multiple elasticsearch indices by event source name
- support for plugins as 3rd party npm modules, e.g.:
- logagent-tcp-input - accepting logs via tcp port from remote machines
- logagent-nodejs-monitor - to monitor logagent process itself and CPU, memory, disk IO usage of the the server machine
- naming of command line arguments run logagent --help to see all options
Official Node.js downloads and instructions. E.g. for Debian/Ubuntu:
curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm i -g @sematext/logagent
Install service for logagent using systemd, upstart, lauchd
sudo logagent-setup LOGSENE_TOKEN
Edit the config file in /etc/sematext/logagent.conf - by default all logs from /var/log/**/*.log are shipped to Logsene.
sudo service logagent restart # upstart
This project contains a library and patterns for log parsing and cli tools and installers to use logagent-js as log shipper with the following features:
- log format detection and intelligent pattern matching
- pattern library included
- easy to extend with custom patterns and JS transform functions
- recognition of Date and Number fields
- replace sensitive data with SHA-1 hash codes
- GeoIP lookup with automatic GeoIP db updates (maxmind geopip-lite files)
-
log format converter (e.g. text to JSON, line delimited JSON or YAML)
-
log shipper for Logsene
- including cli, launchd (Mac OS X), upstart and systemd (Linux) service installer
- disk buffer for failed inserts during network outage
- Standard input (stdin) that can read the output stream from any Linux cli tool
- patterns are applied to each incoming text line; includes support for multi-line patters, e.g. for Java Stack Traces and JSON input.
- Syslog Server (UDP) listener - logagent-js can also act as a syslog server and receive Syslog messages via UDP. The parser is applied to the message field.
- Heroku Log Drain makes it easy to ship Heroku logs to Elasticsearch or Logsene
- Cloud Foundry Log Drain
- logagent-js applies patterns defined in patterns.yml to all logs and creates structured logs from plain-text log lines
- GeoIP lookups for IP address fields, including automatic download and update of the GeoIP lite database from Maxmind
Logagent doesn't lose data. It stores parsed logs to a disk buffer if the network connection to the Elasticsearch API fails. Logagent retries shipping logs later, when the network or Elasticsearch is available again.
- bulk inserts to Logsene / Elasticsearch API
- JSON, line delimited JSON and YML to standard output
- Deployable as a system service: systemd, upstart (Linux), or launchd (Mac OS X)
- Docker Container to receive logs via syslog
- Deployment to Heroku as Heroku Log drain
- Deployment to Cloud Foundry as Cloud Foundry Log drain (thus usable with Pivotal, Bluemix, etc.)
- Node.js module to integrate parsers into Node.js programs
- logagent-js is a part of SPM for Docker to parse Container Logs
The documentation is available here.
- Sematext Agent for Docker - collects metrics, events and logs from Docker API and CoreOS. Logagent-js is a component of sematext-agent-docker. More Information: Innovative Docker Log Management
- Logsene-CLI - Enables searching Logsene log entries from the command-line.
- SPM Agent for Node.js - collects performance metrics for Node and io.js applications
- Custom Metrics - Custom Metrics for SPM
- Winston-Logsene - Logging for Node.js - Winston transport layer for Logsene
- Twitter: @sematext
- Blog: sematext.com/blog
- Homepage: sematext.com