Skip to content
This repository has been archived by the owner on Jun 20, 2023. It is now read-only.

idle connection reaper #149

Merged
merged 4 commits into from
Sep 30, 2014

Conversation

matthewbogner
Copy link
Contributor

This adds a feature to close idle connections in the connection pool, only if configured. I'd like to add tests for this, but I must admit that even without making any changes, I can't get the tests to pass. If you have any tips on environment settings to get the tests to pass (mvn clean verify), then I'd love to add some tests as well.

The Apache HttpClient requires a separate thread to close idle connections periodically, which is a bit of a bummer. This pull request uses a similar design as the NodeChecker to periodically close any idle connections lying around in the connection pool.

My intent is to make this an opt-in behavior (i.e. no change to existing configurations), and to work for both the HTTP and Droid clients. If I could run the tests, I'd feel more confident that I succeeded.

Without this ability, it is possible for connections lying around in the connection pool to have actually been terminated by the server but the client app won't know this until it attempts to make it's first query, which will result in a SocketTimeoutException.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.71%) when pulling 4713a40 on matthewbogner:idle-connection-reaping into 68975bc on searchbox-io:master.

@kramer
Copy link
Member

kramer commented Sep 25, 2014

Hi Matthew,

Thanks a lot for the contribution 🍻 me or some other committer will have a look at the changes as soon as we have the time. In the meantime what is the problem you are having with tests? In the past some of the tests were not properly isolated so running them on parallel created problems but I have not run into this problem recently.

@matthewbogner
Copy link
Contributor Author

Hey @kramer , here is a gist of the test failures I am seeing locally. I did a netstat and ps at the beginning of the gist to show that I have no elasticsearch nodes running locally when I run "mvn clean verify"

https://gist.github.com/matthewbogner/fb11db12dad3c6a5b509

These are the failing tests:

Failed tests: 
  JestClientFactoryIntegrationTest.testDiscovery:34->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 All 4 nodes should be discovered and be in the client's server list expected:<4> but was:<1>

Tests in error: 
  CloseIndexIntegrationTest.testClose:33 » Socket Invalid argument
  FlushIntegrationTest.testFlushSpecificIndices:51 » Socket Invalid argument
  GetMappingIntegrationTest.testWithSingleIndex:61 » Socket Invalid argument
  GetMappingIntegrationTest.testWithMultipleIndices:81 » Socket Invalid argument
  StatsIntegrationTest.testDefaultStats:32 » NullPointer
  TypeExistIntegrationTest.indexTypeExists:34 » Socket Invalid argument

@kramer
Copy link
Member

kramer commented Sep 26, 2014

Could you try with -Djava.net.preferIPv4Stack=true flag? (also do you have multiple network interfaces by any chance?)

@matthewbogner
Copy link
Contributor Author

I tried with the additional JVM flag and got the same results. I do have several network interfaces - one of which is a VPN connection. I tried both with my VPN on and also with it off and get the same failing results.

I'll try and take some time today and figure out in more detail what exactly is going on.

@kramer
Copy link
Member

kramer commented Sep 26, 2014

Yup VPN caused the same problem for me in the past but things were back to normal once I turned VPN connection off. Actual reason probably have something to with routing/name resolution handling with multiple NICs in Java but I was too lazy to do any further research.

@matthewbogner
Copy link
Contributor Author

Turns out that the test suite doesn't like to be run with java 1.7.0_51 on OSX Mavericks. I updated to 1.7.0_67 and had better luck with "mvn clean verify".

However...

I authored some tests which run just fine locally, but Travis is either having issues or disagrees with the quality of my tests. In both builds, the logs reached a 4mb limit and the build got terminated. 🎰

The quantity of "Connection Refused" exceptions coming from the NodeChecker is kind of staggering while running the tests.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.23%) when pulling f0b47ed on matthewbogner:idle-connection-reaping into 68975bc on searchbox-io:master.


@Override
protected void runOneIteration() throws Exception {
logger.info("closing idle connections...");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will probably get too chatty for regular usage, won't it? I suggest changing the level to debug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed (was following lead of NodeChecker).

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.23%) when pulling f587e90 on matthewbogner:idle-connection-reaping into 68975bc on searchbox-io:master.

kramer pushed a commit that referenced this pull request Sep 30, 2014
New feature: Idle connection reaper - opt-in ability to clean idle connections in the pool periodically.
@kramer kramer merged commit 2f428ff into searchbox-io:master Sep 30, 2014
@kramer
Copy link
Member

kramer commented Sep 30, 2014

Thanks again for all your efforts @matthewbogner ! 🍻

@kramer kramer added this to the v0.1.3 milestone Sep 30, 2014
@matthewbogner
Copy link
Contributor Author

Awesome, thank you. Are you guys planning another maven central build/release anytime soon?

@kramer
Copy link
Member

kramer commented Sep 30, 2014

Yup - as tracked on Changelog.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants