Add Splunk logging driver #16207 #16488

outcoldman · 2015-09-22T17:02:49Z

Allow to send Splunk logs using Http Event Collector

Signed-off-by: Denis Gladkikh denis@gladkikh.email

Fixes #16207

glennblock · 2015-09-22T17:04:18Z

This PR is for #16207

tiborvass · 2015-09-22T23:40:06Z

Updated OP's description to include the issue it fixes (so that it automatically closes it if this PR is merged)

albers · 2015-09-23T08:52:10Z

contrib/completion/bash/docker

@@ -270,6 +270,7 @@ __docker_log_drivers() {
 		json-file
 		none
 		syslog
+		splunk


Please insert this in alphabetical order. This also applies to the other additions.

albers · 2015-09-23T09:00:57Z

Thanks very much for also implementing bash completion. That's really cool stuff!

outcoldman · 2015-09-23T17:19:32Z

@albers thank you for review! Just updated PR

albers · 2015-09-23T20:26:03Z

Bash completion LGTM. Thanks for updating.

tiborvass · 2015-09-23T20:49:04Z

Didn't read the code, but I'm +1 for a splunk driver since we already have many similar logging drivers.
I do have to state again the fact that it's unfortunate we have to compile all of these drivers in, and would prefer to have these as run-time Go plugins whenever they are ready. I understand having out-of-process logging plugins can be a significant performance hit.

albers · 2015-09-24T07:53:50Z

ping @sdurrheimer zsh completion: new log-driver

LK4D4 · 2015-09-25T18:06:48Z

@outcoldman Is there any easy way to test driver?

outcoldman · 2015-09-25T18:16:48Z

@LK4D4 it depends on the definition of "easy". I did all my tests manually. To test it we need to

Download Splunk
Extract it somewhere, in case if it will be inside docker container - we will need to have access to any mapped volume with supported file system.
The test scripts can preconfigure HTTP Event Collector and start Splunk.
After we can just send some logs to Splunk and verify that these events were indexed.

Do you think that we can perform first 2 steps? Do you have an example?

LK4D4 · 2015-09-25T18:18:41Z

@outcoldman I mean test manually, not unit-tests. Yes, I thought maybe you have docker image with splunk.

outcoldman · 2015-09-25T18:30:54Z

Ah, I see, so you are asking for instructions how to test it manually. This make sense.

You can use my own Docker Image with Splunk (https://hub.docker.com/r/outcoldman/splunk/) - this is not officially supported by Splunk image, but will be good enough for testing. Or you can download Splunk directly in Docker Dev image in mapped volume, let's say at /opt/splunk, you can download it from http://www.splunk.com/en_us/download.html
You need to expose port 8000 for Web Interface and 8088 for HTTP Event Collector.
After you will start Splunk open 8000 to get access to the Web Interface, you will need to configure HTTP Event Collector, see http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/UsetheHTTPEventCollector (use port 8088 for HTTP Event Collector, it is a default port, you can use HTTP or HTTPS, if you want to use HTTPS - you will need to configure custom certificates or use ca certificate for Docker daemon to configure Splunk driver or just ignore verification).
I used these parameters for HTTP (let me know if you also want to test HTTPS) to start nginx container (replace path to your Splunk instance and token given to you by Splunk on previous step)

docker run --log-driver=splunk --log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 --log-opt splunk-url=http://splunk:8088  -d -p 80:80 nginx

After to test the driver I have send few curl requests to nginx container using

curl http://localhost:80/?justatest=hello

In Splunk Web Interface in search string just run "*" and you should see your events.

outcoldman · 2015-09-25T18:32:35Z

daemon/logger/splunk/splunk.go

@@ -0,0 +1,256 @@
+// +build linux


@LK4D4 one question about this line. I saw it in other logging drivers. What is the purpose of this line? As we don't use anything special - our driver should work everywhere where you can run the daemon.
Is it some kind of transition to Windows? Our driver should work on Windows out of box. Should I remove it?

Yup, you should remove it then.
Actually let's ask @jhowardmsft too :)

Yeah, if it works on Windows and you've verified it does, then remove the line. But you'll need a line added to daemon\logdrivers_windows.go too to add splunk.

Supporting Windows is not critical right now as the WIndows support for Docker is till being developed. Once it settles, we'll revisit. We certainly would like long term for it to work on Windows.

Ok, so per @glennblock we do not support Windows right now. I mean it should work, but we have not tested it.
Should I keep this line for now?
Btw, do I need to have special file "splunk_unsupported.go"?

@jhowardmsft the driver technically should work, but the test effort for us to verify on Windows is pretty significant today, based on our recent eval. Is this something that you guys could help us with? We'd be happy to get you a Splunk instance you could test pushing to.

@LK4D4 the Splunk instance can be running on any platform. The driver is just making simple HTTP requests, so I don't see why it would not work on Windows.

@glennblock - Would be happy to verify, just not immediately, busy getting the last few pieces in for technical preview 4 of Windows Server 2016. Drop me a line.

Will do, thanks @jhowardmsft!

Let's remove tag now. It'll work on windows as well.

LK4D4 · 2015-09-25T18:34:54Z

@outcoldman thanks, I'll try. Moving to code-review in the meantime.

LK4D4 · 2015-09-29T22:09:01Z

btw need rebase, sorry :)

outcoldman · 2015-09-30T02:13:15Z

@LK4D4 np, rebased!

halr9000 · 2015-09-30T15:15:43Z

docs/reference/api/docker_remote_api_v1.21.md

@@ -304,7 +304,7 @@ Json Parameters:
        systems, such as SELinux.
    -   **LogConfig** - Log configuration for the container, specified as a JSON object in the form
          `{ "Type": "<driver_name>", "Config": {"key1": "val1"}}`.
-          Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `none`.
+          Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `splunk`, `none`.


This is getting long. Maybe it should be sorted alpha, with "none" remaining at the end?

hm, yes, we can do that; possible none as the first. I'm fine with leaving that for another PR though.

@outcoldman since this won't make it into docker 1.9, these changes should be moved to docker_remote_api_v1.22.md now

@thaJeztah good point, will move it to 1.22

phemmer · 2015-09-30T17:16:59Z

I would recommend accepting both username & password, as well as http event collector tokens.
As it is, I am unable to test this PR because I am using splunk cloud, which does not yet support the http event collector.
Also, anyone using a self-hosted splunk which is older than 6.3 won't be able to use them either.

glennblock · 2015-09-30T19:35:45Z

Hi @phemmer

Thanks for the input. Event Collector is available in Splunk Cloud, but it needs to be enabled via a support request. There's no UI yet for self-service. But it is definitely available. Please contact Splunk Cloud support and they will get you up and running.

As to the username / password. Going forward we will encourage people not to use the REST API to send events as it requires a username and password that can potentially leak access to Splunk. Also EC has been designed in a more scalable / available fashion than the receiver endpoints. Event Collector tokens are not attached to user credentials, they are specifically designed just for logging events. It is a special endpoint that only allows sending data with a valid token. This makes it more secure from a Splunk perspective as using the token will not allow accessing any part of Splunk, you can't even authenticate to the management port (8089) with the token.

Does that make sense?

You are correct that it is only in Splunk 6.3, but you can stand up dedicated Event Collector instances within a 6.2 environment and they will work fine. EC can run anywhere, an indexer, a forwarder, etc.

glennblock · 2015-10-23T17:32:50Z

Sweet!
On Fri, Oct 23, 2015 at 10:30 AM Sebastiaan van Stijn <
notifications@github.com> wrote:

@moxiegirl https://github.com/moxiegirl gave me a "LGTM" in person [image:
👍] waiting for Janky to finish

—
Reply to this email directly or view it on GitHub
#16488 (comment).

moxiegirl · 2015-10-23T18:16:18Z

I'd recommend clarifying that one paragraph. Otherwise, LGTM

thaJeztah · 2015-10-23T20:21:29Z

Oh, boy.. looks like this needs a rebase :'( sorry for that @outcoldman

edit: sorry for the wrong ping @mountkin, autocomplete hit me :(

Allow to send Splunk logs using Http Event Collector Signed-off-by: Denis Gladkikh <denis@gladkikh.email>

outcoldman · 2015-10-23T20:38:40Z

@thaJeztah np, rebased!

thaJeztah · 2015-10-24T01:22:02Z

Al green! merging \o/

Add Splunk logging driver #16207

LK4D4 · 2015-10-24T03:48:54Z

@outcoldman @glennblock congrats!

outcoldman · 2015-10-24T03:56:57Z

Thank you all for helping! And my personal huge thank to the @LK4D4.
Docker folks, you are great!

thaJeztah · 2015-10-24T04:01:27Z

Thanks for contributing @outcoldman open source FTW!

glennblock · 2015-10-24T16:11:29Z

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!

—
Reply to this email directly or view it on GitHub
#16488 (comment).

glennblock · 2015-10-24T16:12:34Z

Thank you all! We are seeing a lot of interest in Docker in our Splunk
customer base. This is an awesome day!
On Sat, Oct 24, 2015 at 9:11 AM Glenn Block glenn.block@gmail.com wrote:

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!

—
Reply to this email directly or view it on GitHub
#16488 (comment).

pr0xr · 2015-10-24T16:19:24Z

Keewl! =)

moxiegirl · 2015-12-10T22:42:23Z

@thaJeztah Looks like this was missed and not cherry picked in. Any idea when 1.9.3 is going out?

LK4D4 · 2015-12-10T23:01:14Z

@moxiegirl I dunno why you think it should be in a milestone. It's a new feature == shouldn't be in the minor release.

thaJeztah · 2015-12-11T00:46:44Z

There was some confusion about the timing, but this PR was merged after the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th).

This driver will be part of the 1.10 release. For people wanting to give it a spin before that, it is available in the "Experimental" or "Master" builds (not intended for production)

glennblock · 2015-12-11T01:32:41Z

Our understanding was it was part of 1.10. Thanks for confirming.

On Thu, Dec 10, 2015 at 4:48 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

There was some confusion about the timing, but this PR was merged after
the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th
https://github.com/docker/docker/releases/tag/v1.9.0-rc1).

This driver will be part of the 1.10 release. For people wanting to give
it a spin before that, it is available in the "Experimental" or "Master"
builds (not intended for production)

—
Reply to this email directly or view it on GitHub
#16488 (comment).

juniorjbn · 2016-08-17T18:46:55Z

Maybe the splunk driver could receive more than one host in "--log-opt splunk-url" for the cases splunk running in cluster ?

glennblock · 2016-08-17T18:50:31Z

Thanks for the suggestion. With HEC the recommendation is always to have a load balancer in front of the HEC pool. Based on that I don't think we need clustering support. Also that will complicate the implementation.

GordonTheTurtle added the status/0-triage label Sep 22, 2015

albers reviewed Sep 23, 2015
View reviewed changes

bfirsh added status/1-design-review and removed status/0-triage labels Sep 23, 2015

GordonTheTurtle added the dco/no Automatically set by a bot when one of the commits lacks proper signature label Sep 24, 2015

outcoldman force-pushed the splunk_log_driver branch from 0c6955f to 5c8a50b Compare September 24, 2015 13:56

GordonTheTurtle removed the dco/no Automatically set by a bot when one of the commits lacks proper signature label Sep 24, 2015

outcoldman reviewed Sep 25, 2015
View reviewed changes

LK4D4 added status/2-code-review and removed status/1-design-review labels Sep 25, 2015

outcoldman force-pushed the splunk_log_driver branch from b6f7df6 to 9f9c075 Compare September 30, 2015 02:12

halr9000 reviewed Sep 30, 2015
View reviewed changes

Add Splunk logging driver moby#16207

1f1dbf3

Allow to send Splunk logs using Http Event Collector Signed-off-by: Denis Gladkikh <denis@gladkikh.email>

outcoldman force-pushed the splunk_log_driver branch from ca2d315 to 1f1dbf3 Compare October 23, 2015 20:38

thaJeztah added a commit that referenced this pull request Oct 24, 2015

Merge pull request #16488 from splunk/splunk_log_driver

8d4888d

Add Splunk logging driver #16207

thaJeztah merged commit 8d4888d into moby:master Oct 24, 2015

glennblock mentioned this pull request Oct 25, 2015

Splunk driver #16207

Closed

moxiegirl added this to the 1.9.2 milestone Dec 10, 2015

thaJeztah modified the milestones: 1.10, 1.9.2 Dec 11, 2015

bfirsh mentioned this pull request Dec 11, 2015

Logging driver plugins #18604

Closed

thaJeztah added the impact/changelog label Jan 15, 2016

djenriquez mentioned this pull request Feb 5, 2016

Support for Splunk logging driver aws/amazon-ecs-agent#304

Closed

outcoldman deleted the splunk_log_driver branch August 25, 2016 18:30

thaJeztah added the area/logging label Nov 22, 2019

Add Splunk logging driver #16207 #16488

Add Splunk logging driver #16207 #16488

Conversation

outcoldman commented Sep 22, 2015

glennblock commented Sep 22, 2015

tiborvass commented Sep 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

albers commented Sep 23, 2015

outcoldman commented Sep 23, 2015

albers commented Sep 23, 2015

tiborvass commented Sep 23, 2015

albers commented Sep 24, 2015

LK4D4 commented Sep 25, 2015

outcoldman commented Sep 25, 2015

LK4D4 commented Sep 25, 2015

outcoldman commented Sep 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LK4D4 commented Sep 25, 2015

LK4D4 commented Sep 29, 2015

outcoldman commented Sep 30, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phemmer commented Sep 30, 2015

glennblock commented Sep 30, 2015

glennblock commented Oct 23, 2015

moxiegirl commented Oct 23, 2015

thaJeztah commented Oct 23, 2015

outcoldman commented Oct 23, 2015

thaJeztah commented Oct 24, 2015

LK4D4 commented Oct 24, 2015

outcoldman commented Oct 24, 2015

thaJeztah commented Oct 24, 2015

glennblock commented Oct 24, 2015

glennblock commented Oct 24, 2015

pr0xr commented Oct 24, 2015

moxiegirl commented Dec 10, 2015

LK4D4 commented Dec 10, 2015

thaJeztah commented Dec 11, 2015

glennblock commented Dec 11, 2015

juniorjbn commented Aug 17, 2016

glennblock commented Aug 17, 2016