Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Connection Pool Validation #6119

Merged
merged 17 commits into from
Jan 29, 2024
Merged

Conversation

tnleeuw
Copy link
Contributor

@tnleeuw tnleeuw commented Jan 16, 2024

Add some configuration properties for connection pool max-idle connections, and connection validation.

@tnleeuw tnleeuw requested a review from jkosternl January 16, 2024 15:03
@jkosternl
Copy link
Contributor

I am not sure if adding a maxIdle connections is the solution. The other existing variable maxIdleTime should already be sufficient for a good working mechanism. Because if e.g. 5 connections are idle for 2 seconds, I don't want them to be disconnected and removed directly. The timer should start counting and discard them after e.g. 30 or 60 seconds. But since we are seeing a few issues reported on this topic, I wonder if the maxIdleTime is working überhaupt.

@tnleeuw
Copy link
Contributor Author

tnleeuw commented Jan 16, 2024

I am not sure if adding a maxIdle connections is the solution. The other existing variable maxIdleTime should already be sufficient for a good working mechanism. Because if e.g. 5 connections are idle for 2 seconds, I don't want them to be disconnected and removed directly. The timer should start counting and discard them after e.g. 30 or 60 seconds. But since we are seeing a few issues reported on this topic, I wonder if the maxIdleTime is working überhaupt.

maxIdleTime is not available in all pool implementations. It seems to be only available when using BTM's own pool. For non-XA connections we always use DBCP2's pool, and for Narayana we also always use DBCP2's pool implementation (for either XA or non-XA connections).

So it seemed to make sense to me, to add a setting for maxIdle as well. But perhaps we can tweak the default.

BTW I made this PR when on a call with Jeroen & Ali to discuss an issue they are having with database connections in one of the systems. I'm not sure if this PR will even help them.

When you are back, we might decide you finish the changes since you are already working on similar sets of changes.

@jkosternl
Copy link
Contributor

When you are back, we might decide you finish the changes since you are already working on similar sets of changes.

Yes, I wasn't aware of the fact that only BTM understands that maxIdleTime. I think adding a maxIdle make sense. I would not put the setting at 1 because then it directly closes every open idle connection, which basically disables the idea of a connection pool.

If maxPoolSize = 20, and minPoolSize = 0, I'd consider a maxIdle e.g. at 10. Then we assure pooling is effectively used and also enough connections are really closed when not in use.

@tnleeuw
Copy link
Contributor Author

tnleeuw commented Jan 22, 2024

When you are back, we might decide you finish the changes since you are already working on similar sets of changes.

Yes, I wasn't aware of the fact that only BTM understands that maxIdleTime. I think adding a maxIdle make sense. I would not put the setting at 1 because then it directly closes every open idle connection, which basically disables the idea of a connection pool.

If maxPoolSize = 20, and minPoolSize = 0, I'd consider a maxIdle e.g. at 10. Then we assure pooling is effectively used and also enough connections are really closed when not in use.

Some more settings should be added I think, for validation of connections while idle and also, importantly, for the time in between idle connection checking. Otherwise there's no regular validation of idle connections and if my understanding is correct, also not a cleanup of idle connections at regular intervals.

Those settings are only for DBCP2, not for BTM, but still needed in the BTM configuration properties for non-XA connections pools.

@jkosternl
Copy link
Contributor

jkosternl commented Jan 22, 2024

Some more settings should be added I think, for validation of connections while idle and also, importantly, for the time in between idle connection checking. Otherwise there's no regular validation of idle connections and if my understanding is correct, also not a cleanup of idle connections at regular intervals.

We're already setting pooledConnectionFactory.setConnectionCheckInterval(connectionCheckInterval); at NarayanaConnectionFactoryFactory.java file. That starts a connection thread which regularly checks connections already.

@tnleeuw tnleeuw linked an issue Jan 23, 2024 that may be closed by this pull request
@tnleeuw tnleeuw self-assigned this Jan 26, 2024
@tnleeuw tnleeuw added the 7.9 label Jan 26, 2024
@tnleeuw tnleeuw marked this pull request as ready for review January 26, 2024 08:18
@tnleeuw tnleeuw requested a review from nielsm5 January 26, 2024 08:21
Copy link
Member

@nielsm5 nielsm5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ik denk dat je nog wel JJ's wijzigingen moet mergen?

Comment on lines 72 to 74
if (dataSource instanceof XADataSource) {
return createXAPool((XADataSource) dataSource, dataSourceName);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deze moet ook in de maxPoolSize if check?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ik hernoem de method naar createXADataSource() om verwarring te voorkomen. Bij BTM wordt al altijd een pool gecreerd voor XA sessions, ongeacht maxPoolSize. Die zet daarin ook een aantal andere properties, die niet direct gerelateerd zijn aan pooling, dus ik durf daar niet teveel aan te veranderen.

Met een andere naam van de method is het hopelijk minder verwarrend.

return createXAPool((XADataSource) dataSource, dataSourceName);
}

log.info("DataSource [{}] is not XA enabled, unable to register with an Transaction Manager", dataSourceName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deze log klopt niet helemaal meer 😃

@tnleeuw
Copy link
Contributor Author

tnleeuw commented Jan 26, 2024

Ik denk dat je nog wel JJ's wijzigingen moet mergen?

Klopt maar dan moeten die eerst naar 7.9 gebackport en gemerged worden.

tnleeuw and others added 2 commits January 26, 2024 14:17
…actory.java

Co-authored-by: Niels Meijer <nielsmeijer@hotmail.com>
@tnleeuw tnleeuw requested a review from nielsm5 January 26, 2024 14:34
@nielsm5
Copy link
Member

nielsm5 commented Jan 28, 2024

Nu helaas conflicten 😞

@tnleeuw
Copy link
Contributor Author

tnleeuw commented Jan 29, 2024

Nu helaas conflicten 😞

Dat wisten we, dat dat ging gebeuren.

Ik heb gisteren niet meer gekeken naar de mail of je nog dingen had gemerged, sorry.

# Conflicts:
#	core/src/main/java/nl/nn/adapterframework/jta/btm/BtmDataSourceFactory.java
#	core/src/main/java/nl/nn/adapterframework/jta/narayana/NarayanaDataSourceFactory.java
@tnleeuw tnleeuw requested a review from nielsm5 January 29, 2024 10:08

@Override
public String toString() {
return "ManagedDataSource [], DBCP2 Pool Info: " + super.getPool();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waarom de []?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ik had er iets over de datasource tussen will zetten en toen bleek dat toch niet nodig.

@tnleeuw tnleeuw changed the title Add Connection Pool Validation (#6117) Add Connection Pool Validation Jan 29, 2024
…erfere with application servers which do their own pooling and transaction management.
@nielsm5 nielsm5 merged commit 8a26d34 into 7.9-release Jan 29, 2024
10 of 11 checks passed
@nielsm5 nielsm5 deleted the issue/7.9/6117_JdbcConnPoolSettings branch January 29, 2024 16:09
tnleeuw added a commit that referenced this pull request Feb 1, 2024
nielsm5 pushed a commit that referenced this pull request Feb 5, 2024
* Add Connection Pool Validation (#6119)
(Cherrypick from 8a26d34)
* PoolingJndiDataSourceFactory shouldn't check for DataSource to be XA or not XA (#6187)
* Try to determine if a datasource already does pooling (#6199)

Only works for non-XA datasources currently, for XA datasources we need to still consider what is best to do as many XA drivers already implement pooling even if it's not configured.
tnleeuw added a commit that referenced this pull request Feb 5, 2024
* Add Connection Pool Validation (#6119)
(Cherrypick from 8a26d34)
* PoolingJndiDataSourceFactory shouldn't check for DataSource to be XA or not XA (#6187)
* Try to determine if a datasource already does pooling (#6199)

Only works for non-XA datasources currently, for XA datasources we need to still consider what is best to do as many XA drivers already implement pooling even if it's not configured.

(cherry picked from commit 281bf1e)
nielsm5 pushed a commit that referenced this pull request Feb 5, 2024
* Add Connection Pool Validation (#6119)
(Cherrypick from 8a26d34)
* PoolingJndiDataSourceFactory shouldn't check for DataSource to be XA or not XA (#6187)
* Try to determine if a datasource already does pooling (#6199)

Only works for non-XA datasources currently, for XA datasources we need to still consider what is best to do as many XA drivers already implement pooling even if it's not configured.

(cherry picked from commit 281bf1e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Db connections are left open in pool
3 participants