[🐛 BUG]: grpc client is closed to early after graceful shutdown #1986
Description
No duplicates 🥲.
- I have searched for a similar issue in our bug tracker and didn't find any solutions.
What happened?
RR closes to early grpc connection so some running process are not able to complete correctly.
Version (rr --version)
2024.1.5
How to reproduce the issue?
(with temporal)
- start activity which takes some time (for instance - makes a HTTP request)
- send sigterm signal before end of the activity
In such scenario activity finishes but is not able to send result to temporal with grpc client.
Relevant log output
2024-08-12T06:59:36+02:00 {"level":"info","ts":1723438776025103636,"logger":"temporal","msg":"Task processing failed with error","Namespace":"orders-service","TaskQueue":"default","WorkerID":"default:159998ee-1ff5-4ab9-bfc6-ad74e2924210","WorkerType":"ActivityWorker","Error":"grpc: the client connection is closing"}
2024-08-12T06:59:36+02:00 {"message":"API Request finished"}
2024-08-12T06:59:35+02:00 {"level":"info","ts":1723438775062632589,"logger":"server","msg":"destroy signal received","timeout":60}
2024-08-12T06:59:35+02:00 {"level":"info","ts":1723438775062567608,"logger":"server","msg":"destroy signal received","timeout":60}
2024-08-12T06:59:35+02:00 {"level":"info","ts":1723438775062591477,"logger":"server","msg":"destroy signal received","timeout":30}
2024-08-12T06:59:35+02:00 {"level":"info","ts":1723438775062558877,"logger":"temporal","msg":"Stopped Worker","Namespace":"orders-service","TaskQueue":"default","WorkerID":"default:159998ee-1ff5-4ab9-bfc6-ad74e2924210"}
2024-08-12T06:59:35+02:00 stop signal received, grace timeout is: 30 seconds
2024-08-12T06:59:34+02:00 {"message":"API Request started"}Metadata
Assignees
Type
Projects
Status
✅ Done