Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kombu 5.4.1 not working with Celery[SQS] 5.4.0 #2113

Closed
Hawkline451 opened this issue Sep 12, 2024 · 15 comments · Fixed by #2119
Closed

Kombu 5.4.1 not working with Celery[SQS] 5.4.0 #2113

Hawkline451 opened this issue Sep 12, 2024 · 15 comments · Fixed by #2119

Comments

@Hawkline451
Copy link

Celery worker is running but is not picking any messages from the SQS queue.

Kombu 5.4.0 works fine.

@Nusnus
Copy link
Member

Nusnus commented Sep 13, 2024

The changes regarding SQS in v5.4.1 were:

@Hawkline451 can you please share more info?
Is there any error? Does it sometimes work sometimes doesn’t?

If I had to guess, I’d say it’s the SSL changes (validate_cert changed from default false to default true), but the original PR says

... I'm running python 3.12.4 and tested with the latest Celery+Kombu from github.

So I am not completely sure.
@ivanprjcts @poundifdef can you please give it a look as well in case I am missing something?

Thank you!

@imran8294
Copy link

Hi @Nusnus
Kombu 5.4.1 is not compatible with Celery 5.4.0. As a result, the messages are not being consumed. Although the Celery workers are initialized, they are unable to retrieve messages from the queue.

With kombu <= 5.4.0 its working as expected.

@Nusnus
Copy link
Member

Nusnus commented Sep 13, 2024

Hi @Nusnus
Kombu 5.4.1 is not compatible with Celery 5.4.0. As a result, the messages are not being consumed. Although the Celery workers are initialized, they are unable to retrieve messages from the queue.

With kombu <= 5.4.0 its working as expected.

@thedrow @auvipy

Fyi.

As this is an SQS only issue I prefer not to yank the last release but revert the PR that caused it (once found) and release a new patch.

@auvipy
Copy link
Member

auvipy commented Sep 13, 2024

Yeah that should work

@Hawkline451
Copy link
Author

@Hawkline451 can you please share more info? Is there any error? Does it sometimes work sometimes doesn’t?

If I had to guess, I’d say it’s the SSL changes (validate_cert changed from default false to default true), but the original PR says

... I'm running python 3.12.4 and tested with the latest Celery+Kombu from github.

There is no errors or warnings, we test this running the worker in an EC2 server in debug.

Aditional info

boto3==1.35.5
botocore==1.35.5
celery==5.4.0
Django==4.2.13

Python 3.11

@ivanprjcts
Copy link
Contributor

Hi @Nusnus !
I checked the issue using a local sample celery application using SQS, and it seems to be related to this commit.

I fixed it after reverting -> https://github.com/ivanprjcts/kombu/commits/bugfix/revert-cert-validation/

You can try it using:
pip install git+https://github.com/ivanprjcts/kombu.git@fe5fb64627f17031799bc98be7cab14155eb0682#egg=kombu

@Nusnus
Copy link
Member

Nusnus commented Sep 14, 2024

Hi @Nusnus !
I checked the issue using a local sample celery application using SQS, and it seems to be related to this commit.

I fixed it after reverting -> https://github.com/ivanprjcts/kombu/commits/bugfix/revert-cert-validation/

You can try it using:
pip install git+https://github.com/ivanprjcts/kombu.git@fe5fb64627f17031799bc98be7cab14155eb0682#egg=kombu

Thank you!

@poundifdef FYI - if you have a potential fix please submit a PR. If not, we will have to revert the change unfortunately 🙏

@poundifdef
Copy link
Contributor

poundifdef commented Sep 16, 2024

Let me take a look! Also, if you need to revert in the meantime, please go ahead and don't let me block this. I can always make a separate PR with a more complete update.

@Nusnus
Copy link
Member

Nusnus commented Sep 16, 2024

@poundifdef

Let me take a look! Also, if you need to revert in the meantime, please go ahead and don't let me block this. I can always make a separate PR with a more complete update.

Cool, thanks!

@Nusnus
Copy link
Member

Nusnus commented Sep 16, 2024

@Hawkline451
Change reverted in main.
New patch version will be released soon.

@Hawkline451
Copy link
Author

@Hawkline451 Change reverted in main. New patch version will be released soon.

Thanks

@pranavdxl
Copy link

Any ETA for release?

@Nusnus
Copy link
Member

Nusnus commented Sep 19, 2024

Any ETA for release?

Today 🙂

@mhdzumair
Copy link

+1

@Nusnus Nusnus self-assigned this Sep 19, 2024
@Nusnus Nusnus linked a pull request Sep 19, 2024 that will close this issue
@Nusnus
Copy link
Member

Nusnus commented Sep 19, 2024

v5.4.2 Released.
https://pypi.org/project/kombu/5.4.2/

@Hawkline451 @imran8294 @ivanprjcts @poundifdef @pranavdxl @mhdzumair

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants