Description
Jetty version(s)
Observed in 12.0.6-12.0.10 with ee10.
First observed in March 2024 when we deployed a release of our server that upgraded from 11.0.20 to 12.0.6+ee10. It had never been observed prior to that (at least, I can't find it in our retained error logs from the last year).
Jetty Environment
ee10
Java version/vendor (use: java -version)
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Corretto-21.0.3.9.1 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.3.9.1 (build 21.0.3+9-LTS, mixed mode, sharing)
OS type/version
Amazon Linux 2
Description
We periodically but rarely (a few times a month) see EofException caused by ClosedChannelException. We more routinely see plain IOException when a client disconnects without reading the response, and we ignore that error. We've considered ignoring this error too, but our understanding is that it should only happen if Jetty closed its side of the connection and then subsequently attempted to write, so it seems like this could be some kind of connection state management issue in Jetty. The relevant filters/servlets are purely synchronous: they do not use startAsync, and they do not use HttpServletRequest/HttpServletResponse from other threads. Recent stack trace from 12.0.10+ee10 (as logged by Logback, which writes the explicit null for null messages).
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:117)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:368)
at org.eclipse.jetty.server.internal.HttpConnection$SendCallback.process(HttpConnection.java:842)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:250)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:231)
at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1429)
at org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1263)
at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:751)
at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:288)
at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:205)
at org.eclipse.jetty.ee10.servlet.HttpOutput.close(HttpOutput.java:562)
redacted
at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:97)
at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
at org.eclipse.jetty.server.Server.handle(Server.java:179)
at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:635)
at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException: null
at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:222)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:568)
at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:660)
at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:111)
... 52 common frames omitted
How to reproduce?
I don't know. I've tried various things against a local Jetty instance (early client close, delayed client close, slow client write, slow server write, different buffered amounts to trigger an error from .write vs .close, etc.), but the only relevant error I can cause is EofException with IOException: Broken pipe as expected.