Replies: 2 comments 4 replies
-
|
In refreshAfterWrite, after the duration has passed the next request to that entry triggers an asynchronous reload. When that occurs is non-deterministic, and often it is combined with expiration so inactive entries fade away. Since this is not a scheduled periodic reload, I don't think the cache should try to perform some type of jitter because it is all reactionary rather than a deterministic action. If you want to introduce jitter then consider adding an artificial delay to the reload future. This could be done by having the @Override public CompletableFuture<? extends V> asyncReload(K key, V oldValue, Executor executor) {
return super.asyncReload(key, oldValue, CompletableFuture.delayedExecutor(
ThreadLocalRandom.current().nextLong(0, 2_000), TimeUnit.MILLISECONDS, executor));
}You might also find batching the reloads to be assist in performance. See coalescing-bulkloader-reactor for an example using a reactive streams library, which makes this fairly trivial. This can buffer over a time/space interval to make more efficient requests, with a max parallelism to throttle the number of queries being made at once. does any of that help? |
Beta Was this translation helpful? Give feedback.
-
|
Excellent! Will try one or both of these. Thank you so much for answering so clearly and quickly. 👍 |
Beta Was this translation helpful? Give feedback.
-
|
I think these are fair workarounds, but from what I understand, when working with networked caches, jitter is considered best practice. Of course, Caffeine is in-memory, and tbh I can't really find any reliable-looking articles about best practices for those. We run ~80 tasks requesting the same resource, so I'll start with per-task jitter (e.g. constant for a single running application), and can report back if we see any issues with that. |
Beta Was this translation helpful? Give feedback.
-
|
Jitter for networked caches typically discuss expiration to avoid stampedes when a popular item expires. Since those caches cannot load on a miss, the clients produce query storms that may exhaust the resource. Thus, some type of probabilistic early expiration allows one client to reload early to refresh the entry. This is due to a shared expiration time. That’s basically what refreshAfterWrite does. If the entry is accessed before its hard expiration limit within the reload period, then a proactive load is performed to obtain a fresh value. For a given key only one reload will be performed by the cache while still valid. Since accessing the entry is user initiated, when and if is probabilistic. The expiration TTL typically differs per machine, e.g. 5 minutes since it was loaded, but when it was loaded is varied. The refresh is an explicit feature since the cache is given a means to fetch the value. A jittered refresh then doesn’t make much sense. When the entry is within the refresh period and when it’s accessed is not uniform. The cache will only perform a single load for a key, whereas a remote cache may have the same client machine try to fetch per caller. It cannot know if the client memoizes to avoid stampedes, making it very likely the client was unaware or has limited tooling. Jittered expiration is of course possible, e.g. if remotely defined by the system of record. That can be done using an Expiry. That makes sense if there is an absolute timestamp, e.g. a networked cache for an external service that requires a fresh query. Google Maps for instance allows reverse geocoding to be cached for 30 days, so one might layer to avoid api calls. For local refresh, batching makes more sense than jitter. You can jitter the request too by delaying the future, but that likely doesn’t actually help you much because refresh solves the same problem as expiration jitter for local caches. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick reply! That makes sense to me given your assumptions, but let me know if this would be an accurate way to look at our use case:
(again, I don't think we need any changes to Caffeine to add jitter here, I was just curious if you'd agree jitter would help spread out load on the DB) |
Beta Was this translation helpful? Give feedback.
-
|
Oh, I see. Thanks for the clarification. There are so many different scenarios to consider for how a feature is used and I forgot about that one in my reply. Having them implicitly synchronized due to a deployment and a high read rate would certainly cause bursts. Your idea of jitter makes sense, and I think doing that in |
Beta Was this translation helpful? Give feedback.
-
We have a problem where a lot of our cache entries all get refreshed at the same time, causing problems for our service and downstream service. An idea is to be able to configure a jitter time interval, for example (0ms to 2000ms) so that the refreshes get spread out in time. Wdyt?
From the Java docs:
Beta Was this translation helpful? Give feedback.
All reactions