Skip to content

[BUG] Spring Cloud Stream Service Bus delivers a message sometimes twice #43428

Open
@mlohmeyer

Description

Describe the bug
Last Month we had 6000 cases (out of 5 Mio), where a Message gets delivered twice with Spring Cloud Stream Service Bus.

  • The message stays for the Timeout in the Queue (2 Min, thats the minor problem)
  • Then we process the message and complete it
  • Then the message gets delivered again. (after another 2,8 mins)
  • Then Spring Cloud Stream Azure is doing a renewMessageLock which fails, becuase the message is already completed
    But thats an internal exception, thus we process the message again. That the major problem.
    At the End complete fails, because the message is already processed.
  • the error happens at arbitrary time. Not only on high Load on service Bus (Premium) and Container App.

Exception or Stack Trace
com.azure.messaging.servicebus.ServiceBusException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue. For more information please see ...

To Reproduce
unclear

Expected behavior
A Message get delivered only one time.

Screenshots

Image

** Logs **

Logs from Service Bus Side (From Azure Support Ticket)
Message sent to queue | 0:12:01:01.529093
First delivery (msg lock) | 0:12:01:01.529272
Abandoned | 0:12:03:01.520244
Second delivery | 0:12:03:01.530253
Completed Successfully | 0:12:03:02.480113

what we can see in the application Insights Logs (100% of all logs are saved in Insight at that time)
first processing    on 2024-11-28T12:03:01.527Z (UTC)
complete         on 2024-11-28T12:03:02.366Z (UTC)
second processing on 2024-11-28T12:05:48.082Z (UTC) why?

**Setup **

  • OS: Linux Docker Image
  • Library/Libraries:
    spring-messaging-azure-4.15.0,
    spring-messaging-azure-servicebus-4.15.0
    spring-cloud-azure-stream-binder-servicebus-4.15.0
    spring-cloud-stream-4.0.3
    spring-boot-3.1.5
  • Java version: 17
  • App Server/Environment: Azure Container Apps
  • Frameworks: Spring Boot

Information Checklist

  • [ x ] Bug Description Added
  • [ x ] Repro Steps Added
  • [ x ] Setup information Added

Metadata

Assignees

Labels

azure-springAll azure-spring related issuescustomer-reportedIssues that are reported by GitHub users external to the Azure organization.questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

  • Status

    In Progress

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions