Description
I'm using docker-in-docker inside of kubernetes in jenkins. My containers appear to start up fine, but the host/port never becomes accessible and so testcontainers cannot connect to them.
Everything works fine when I run the tests locally using docker desktop, it's only on when running in jenkins using docker-in-docker that they fail.
The container ip address is 172.17.0.2
but it's trying to connect to 172.17.0.1
which is the gateway. Is that normal?
I've tested this on both testcontainers 1.16.0 and 1.16.1 and they both fail. This could be related to #4594, except my problem happens on 1.16.0 as well. This also might be related to #4591 although I am not using TESTCONTAINERS_HOST_OVERRIDE
Here is my test case where I start up an elasticsearch container, log the ipAddress and hostAddress, and then run docker inspect
and log the results:
class ElasticsearchContainerTest {
companion object {
private val logger = KontextLogger.logger {}
private const val ELASTICSEARCH_VERSION = "6.7.0"
private val container = ElasticsearchContainer(
DockerImageName.parse("{REDACTED CONTAINER MIRROR URL}/elasticsearch")
.withTag(ELASTICSEARCH_VERSION)
.asCompatibleSubstituteFor("docker.elastic.co/elasticsearch/elasticsearch")
).withLogConsumer(Slf4jLogConsumer(KontextLogger.logger {}))
@BeforeAll
@JvmStatic
fun beforeAll() {
container.start()
}
@AfterAll
@JvmStatic
fun afterAll() {
logger.error("Container IP address: ${container.containerIpAddress}")
logger.error("Container url: ${container.httpHostAddress}")
val dockerInspect = container.dockerClient.inspectContainerCmd(container.containerId).exec()
val mapper = ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
val jsonOutput = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dockerInspect)
logger.error("docker inspect elasticsearch: $jsonOutput")
container.stop()
}
}
@Test
fun `check that container is running`() {
Assert.assertTrue("Container is not running", container.isRunning)
}
}
The test fails due to Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200
:
:integrationTest > 0 tests completed> :integrationTest > Executing test com.example.test.Elasticsea
ElasticsearchContainerTest > initializationError FAILED
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
at com.example.test.ElasticsearchContainerTest$Companion.beforeAll(ElasticsearchContainerTest.kt:28)
at com.example.test.ElasticsearchContainerTest.beforeAll(ElasticsearchContainerTest.kt)
Caused by:
org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
... 3 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 4 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:264)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
... 6 more
Here is the output of docker inspect
:
{
"Args" : [ "eswrapper" ],
"Config" : {
"healthcheck" : null,
"AttachStderr" : false,
"AttachStdin" : false,
"AttachStdout" : false,
"Cmd" : [ "eswrapper" ],
"Domainname" : "",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Hostname" : "8d8c72fac5a9",
"Image" : "{REDACTED CONTAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
},
"MacAddress" : null,
"NetworkDisabled" : null,
"OnBuild" : null,
"OpenStdin" : false,
"PortSpecs" : null,
"StdinOnce" : false,
"Tty" : false,
"User" : "",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Healthcheck" : null
},
"Created" : "2021-10-20T20:26:52.934006535Z",
"Driver" : "overlay2",
"ExecDriver" : null,
"HostConfig" : {
"Binds" : [ ],
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"MemorySwappiness" : null,
"NanoCpus" : 0,
"CapAdd" : null,
"CapDrop" : null,
"ContainerIDFile" : "",
"CpuPeriod" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpuShares" : 0,
"CpuQuota" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"DiskQuota" : null,
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"LxcConf" : null,
"Memory" : 0,
"MemorySwap" : 0,
"MemoryReservation" : 0,
"KernelMemory" : 0,
"NetworkMode" : "default",
"OomKillDisable" : false,
"Init" : null,
"AutoRemove" : false,
"OomScoreAdj" : 0,
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"RestartPolicy" : {
"MaximumRetryCount" : 0,
"Name" : ""
},
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"VolumesFrom" : [ ],
"Mounts" : null,
"PidMode" : "",
"Isolation" : null,
"SecurityOpt" : null,
"StorageOpt" : null,
"CgroupParent" : "",
"VolumeDriver" : "",
"ShmSize" : 67108864,
"PidsLimit" : null,
"Runtime" : "runc",
"Tmpfs" : null,
"UTSMode" : "",
"UsernsMode" : "",
"Sysctls" : null,
"ConsoleSize" : [ 0, 0 ]
},
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log",
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"SizeRootFs" : null,
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"MountLabel" : "",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"PortMapping" : null,
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02"
}
}
},
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"ProcessLabel" : "",
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"State" : {
"oomkilled" : false,
"pidLong" : 873,
"exitCodeLong" : 0,
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z",
"Health" : null
},
"Volumes" : null,
"VolumesRW" : null,
"Node" : null,
"Mounts" : [ ],
"GraphDriver" : {
"Name" : "overlay2",
"Data" : {
"RootDir" : null,
"DeviceId" : null,
"DeviceName" : null,
"DeviceSize" : null,
"dir" : null
}
},
"Platform" : "linux",
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"Args" : [ "eswrapper" ],
"ProcessLabel" : "",
"Mounts" : [ ],
"Platform" : "linux",
"Config" : {
"Hostname" : "8d8c72fac5a9",
"Domainname" : "",
"User" : "",
"AttachStdin" : false,
"AttachStdout" : false,
"AttachStderr" : false,
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Tty" : false,
"OpenStdin" : false,
"StdinOnce" : false,
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"Cmd" : [ "eswrapper" ],
"Image" : "{REDACTED CONAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"OnBuild" : null,
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
}
},
"Driver" : "overlay2",
"AppArmorProfile" : "",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"GraphDriver" : {
"Data" : {
"LowerDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5-init/diff:/home/rootless/.local/share/docker/overlay2/1e71332ad2a429a4d06d157b6c4e2c4401da2b44b640d412557feb3c39452010/diff:/home/rootless/.local/share/docker/overlay2/1302968f5de6c81580eadb218555c0bc49bd3af7c766e77cbdb0ea9f02371d2a/diff:/home/rootless/.local/share/docker/overlay2/3b2b6c22a55172df81ac6af6f439aa3ec1cfe0576269d9fab8c25772e49a65ca/diff:/home/rootless/.local/share/docker/overlay2/42a8e273145d92d47a2c6827fa4d276800e98189da117ca69b25808ce1013edc/diff:/home/rootless/.local/share/docker/overlay2/be84a76283d29eac19bb5e262a4b26578218877d2302cea775fc57c29e195fb0/diff:/home/rootless/.local/share/docker/overlay2/05a94b779ac49607562a956de5a274e5cd000747ef966e612f156ba3f5018cb3/diff:/home/rootless/.local/share/docker/overlay2/57967e2c3b2027a4b67e5d97544d36b11573228ae195fb1cb0799b0ca954c669/diff",
"MergedDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/merged",
"UpperDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/diff",
"WorkDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/work"
},
"Name" : "overlay2"
},
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"Created" : "2021-10-20T20:26:52.934006535Z",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02",
"DriverOpts" : null
}
}
},
"MountLabel" : "",
"State" : {
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z"
},
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"HostConfig" : {
"Binds" : [ ],
"ContainerIDFile" : "",
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"NetworkMode" : "default",
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"RestartPolicy" : {
"Name" : "",
"MaximumRetryCount" : 0
},
"AutoRemove" : false,
"VolumeDriver" : "",
"VolumesFrom" : [ ],
"CapAdd" : null,
"CapDrop" : null,
"CgroupnsMode" : "host",
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"OomScoreAdj" : 0,
"PidMode" : "",
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"SecurityOpt" : null,
"UTSMode" : "",
"UsernsMode" : "",
"ShmSize" : 67108864,
"Runtime" : "runc",
"ConsoleSize" : [ 0, 0 ],
"Isolation" : "",
"CpuShares" : 0,
"Memory" : 0,
"NanoCpus" : 0,
"CgroupParent" : "",
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"CpuPeriod" : 0,
"CpuQuota" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"KernelMemory" : 0,
"KernelMemoryTCP" : 0,
"MemoryReservation" : 0,
"MemorySwap" : 0,
"MemorySwappiness" : null,
"OomKillDisable" : false,
"PidsLimit" : null,
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"MaskedPaths" : [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ],
"ReadonlyPaths" : [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ]
},
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log"
}
Some additional log lines which might help:
INFO org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /focused_roentgen: Waiting for 120 seconds for URL: http://172.17.0.1:49154/ (where port 49154 maps to container port 9200)
Log output from the failed container:
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] using [1] data paths, mounts [[/ (overlay)]], net usable_space [79.4gb], net total_space [242.2gb], types [overlay]
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] heap size [990.7mb], compressed ordinary object pointers [true]
[INFO ][o.e.n.Node ] [dP6fGhj] node name derived from node ID [dP6fGhj8TES6gRpq72sI0Q]; set [node.name] to override
[INFO ][o.e.n.Node ] [dP6fGhj] version[6.7.0], pid[1], build[default/docker/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/5.4.0-1028-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[INFO ][o.e.n.Node ] [dP6fGhj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2290863965764670913, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [aggs-matrix-stats]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [analysis-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-geoip]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-user-agent]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-expression]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-mustache]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-painless]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [mapper-extras]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [parent-join]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [percolator]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [rank-eval]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [reindex]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [repository-url]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [transport-netty4]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [tribe]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ccr]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-core]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-deprecation]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-graph]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ilm]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-logstash]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ml]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-monitoring]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-rollup]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-security]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-sql]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-upgrade]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-watcher]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] no plugins loaded
[INFO ][o.e.x.s.a.s.FileRolesStore] [dP6fGhj] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dP6fGhj] [controller/88] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV
[INFO ][o.e.d.DiscoveryModule ] [dP6fGhj] using discovery type [single-node] and host providers [settings]
[INFO ][o.e.n.Node ] [dP6fGhj] initialized
[INFO ][o.e.n.Node ] [dP6fGhj] starting ...
[INFO ][o.e.t.TransportService ] [dP6fGhj] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[INFO ][o.e.h.n.Netty4HttpServerTransport] [dP6fGhj] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[INFO ][o.e.n.Node ] [dP6fGhj] started
[2021-10-20T20:28:53.705Z] [2021-10-20T20:27:00,528][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dP6fGhj] Failed to clear cache for realms [[]]
[INFO ][o.e.g.GatewayService ] [dP6fGhj] recovered [0] indices into cluster_state
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watches] for index patterns [.watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[INFO ][o.e.l.LicenseService ] [dP6fGhj] license [2dcb2824-6c74-4c3a-b8ac-d06233221f44] mode [basic] - valid
When I run a similar test for postgres, it passes but that's only because the postgres testcontainer uses a log watching wait strategy and doesn't try to connect to the exposed port. If I try to actually connect to the postgres container it fails as well.
My kubernetes podspec is like this:
podSpec:
spec:
containers:
- name: main
image: <redacted, but my image is based on debian>
command:
- cat
tty: true
volumeMounts:
- mountPath: /var/run
name: docker-socket
env:
- name: TESTCONTAINERS_RYUK_DISABLED
value: true
- name: dind-daemon
image: docker:20.10.9-dind-rootless
securityContext:
privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- name: docker-socket
mountPath: /run/user/1000
volumes:
- name: docker-socket
emptyDir: {}
Activity
jameslamine commentedon Oct 21, 2021
Setting
TESTCONTAINERS_HOST_OVERRIDE
tolocalhost
fixed the issue. So it's not automatically detecting that correctly. The fix worked with testcontainers1.16.1
. I didn't test it with1.16.0
.kiview commentedon Oct 21, 2021
Thanks for reporting @jameslamine.
Indeed, there seem to be a couple of issues around this and they might very well be connected. We have to yet investigate the root cause of this.
Thanks a lot for sharing the workaround as well, much appreciated.
kiview commentedon Oct 21, 2021
Can you please set your Testcontainers logs to DEBUG level (see here https://www.testcontainers.org/supported_docker_environment/logging_config/) and provide all the logs of the test run?
If you would be able to create some kind of local reproducer, this would also be helpful, but I get that this might be challenging or nearly impossible.
bsideup commentedon Oct 21, 2021
this is normal, as we are not connecting to container's IP, but Docker Gateway on which all ports are exposed
I think these issues are unrelated. What happening is:
localhost
won't work, and the gateway IP should be used instead172.17.0.1
but your pod cannot see it, as you have multiple layers of containerizationSetting
TESTCONTAINERS_HOST_OVERRIDE
tolocalhost
works because containers in k8s pods share the same network namespace. Unfortunately, there isn't much we can do in this scenario, as the setup is custom. But that's exactly why we introducedTESTCONTAINERS_HOST_OVERRIDE
- to be able to override the detection in the most advanced environments.I hope this explanation helps understanding what's going on better 👍
jameslamine commentedon Oct 22, 2021
Thanks, that's helpful.
I am running with the testcontainers logs set to debug. Unfortunately the jenkins output is a bit of a mess and it would be a lot of work for me to clean it up and redact any info from it which I should't post publicly.
I can tell you that testcontainers detects the docker socket and detects that it's running in a container and detects that the daemon is docker-in-docker rootless. You're right that it's detecting the docker host as
172.17.0.1
. If there's anything specific you're looking for in the logs let me know and I can check.Would it be possible to add an extra step to the environment detection where it starts up a docker container and tries to connect to it at both "localhost" and the gateway IP? That would be one option for automatically detecting this sort of environment. Perhaps you could use
alpine
for this since you're already pulling it, or make a custom tiny container to replacealpine
. I don't know how much this would slow down setup.Another option would be to try to detect that you're running inside of kubernetes and used that to help decide what to do.
jameslamine commentedon Oct 22, 2021
I can confirm that this shows up in the logs:
kiview commentedon Oct 22, 2021
Hey @jameslamine, we are currently focused on investigating the other issues you have linked (which indeed seem to be a different issue) and then we can think about how to improve this use case.
Is the
TESTCONTAINERS_HOST_OVERRIDE
approach a productively usable workaround for you?jameslamine commentedon Oct 23, 2021
Yup,
TESTCONTAINERS_HOST_OVERRIDE
is working great. Thanks for your help with this.6 remaining items