Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HazelcastClientNotActiveException on exporting record #226

Open
harmjanblok opened this issue Nov 17, 2022 · 3 comments
Open

HazelcastClientNotActiveException on exporting record #226

harmjanblok opened this issue Nov 17, 2022 · 3 comments
Labels
bug Something isn't working

Comments

@harmjanblok
Copy link

We've deployed hazelcast inside kubernetes using a headless service for hazelcast. When hazelcast restarts, the exporter should reconnect as the IP for hazelcast changed (headless service).

2022-11-17 10:46:43.724 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-2] WARN
      io.camunda.zeebe.broker.exporter.hazelcast - Error on exporting record with key -1
com.hazelcast.client.HazelcastClientNotActiveException: Client is not active.
        at com.hazelcast.internal.serialization.impl.bufferpool.BufferPoolThreadLocal.get(BufferPoolThreadLocal.java:107) ~[?:?]
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:222) ~[?:?]
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:214) ~[?:?]
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:199) ~[?:?]
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:154) ~[?:?]
        at com.hazelcast.client.impl.spi.ClientProxy.toData(ClientProxy.java:76) ~[?:?]
        at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.add(ClientRingbufferProxy.java:130) ~[?:?]
        at io.zeebe.hazelcast.exporter.HazelcastExporter.export(HazelcastExporter.java:149) ~[?:?]
        at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$export$3(ExporterContainer.java:156) ~[zeebe-broker-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:34) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.export(ExporterContainer.java:155) ~[zeebe-broker-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:142) ~[zeebe-broker-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:510) ~[zeebe-broker-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:74) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:42) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:125) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) ~[zeebe-util-8.0.6.jar:8.0.6]
        at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) ~[zeebe-util-8.0.6.jar:8.0.6]

A catch for HazelcastClientNotActiveException has been added in #123, maybe a similar catch should be added here?
@harmjanblok
Copy link
Author

Some more background information, we configured hazelcast to use DNS lookup instead of the (default) Kubernetes API mode to discover members.

This error only seem to appear with a single-member cluster (cluster.memberCount of 1 using the hazelcast helm chart).

@saig0 saig0 added the bug Something isn't working label Dec 9, 2022
@saig0
Copy link
Collaborator

saig0 commented Dec 9, 2022

@harmjanblok thank you for reporting. 👍

The issue is that the Hazelcast exporter doesn't reconnect to the new Hazelcast node, correct?

I'm not sure how to reproduce this behavior in a test. Could you test that catching the exception would fix the issue?

@alfisti166
Copy link

I have the same issue. Camunda version 8.3.3, hazelcast 5.3.6, hazelcast exporter 1.4.0
ZEEBE_HAZELCAST_REMOTE_ADDRESS set to hazelcast cluster.
If I run hazelcast and then camunda (both helm projects), everything is fine. But after uninstalling hazelcast and installing again, following error happens (and nothing is exported to hazelcast):

      io.camunda.zeebe.broker.exporter.hazelcast - Error on exporting record with key -1
com.hazelcast.client.HazelcastClientNotActiveException: Client is not active.
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.serializerFor(AbstractSerializationService.java:558) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:230) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:217) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:202) ~[?:?]
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:157) ~[?:?]
	at com.hazelcast.client.impl.spi.ClientProxy.toData(ClientProxy.java:76) ~[?:?]
	at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.add(ClientRingbufferProxy.java:130) ~[?:?]
	at io.zeebe.hazelcast.exporter.HazelcastExporter.export(HazelcastExporter.java:149) ~[?:?]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$export$5(ExporterContainer.java:181) ~[zeebe-broker-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:34) ~[zeebe-util-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.export(ExporterContainer.java:180) ~[zeebe-broker-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.exportRecord(ExporterContainer.java:167) ~[zeebe-broker-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:528) ~[zeebe-broker-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:96) [zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:47) [zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:120) [zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.3.3.jar:8.3.3]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.3.3.jar:8.3.3]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@saig0 @harmjanblok @alfisti166