Oh, I forgot to mention that the database was in the status "green" when the issues happened. I tried both the "index" query and the "exact" query, and the same thing happened: nothing could be found.

And it's a single server instance: no clusters, no shardings.

How to troubleshoot and debug such things? We might sometimes forcibly shut down the qdrant server with SIGINT or SIGQUIT signals. Will that be a potential cause here?

Though this shouldn't be a problem, please use graceful shutdowns when possible.

For each new qdrant version, we always destroy the old collections' vectors and insert all the vectors all over again (just to make sure it's clean).

Is that just to ensure everything is there? Because normally, that should not be necessary.

Oh, I forgot to mention that the database was in the status "green" when the issues happened. I tried both the "index" query and the "exact" query, and the same thing happened: nothing could be found.

Did you check to explicitly retrieve the points you thought were missing?

You can use GET /collections/my_collection/points/123.

I assume you have been searching by exact vector to check whether a point still exists. With regular search it may be possible for it not being found. Though, when using exact search you should always see it.

@timvisee Thanks for your suggestion! We'll try that GET request to check the points the next ime. The crime scene is already gone since it is a production server and we had to reinsert all the vectors to bring it back ASAP.

Thanks!

I don't want to state the obvious, but, we definitely don't expect this with restarts or version upgrades.

We ran into the same problem with qdrant 1.11.5 yesterday again. Deleting all the data and re-importing the data makes the issue go away. The symptom is that sometimes some data cannot be found in vector search even when exact flag is set. We'll keep an eye on this.

The data set is static and never changes. That's why we've been scratching our head when the vector search cannot find something that is definitely there. Very frustrating...

We also have a static test suite that may occasionally fail intermittently. The tests also never change.

And I saw there's a newer version, 1.12.1. Maybe we should upgrade to the latest version?

Just upgraded qdrant to the latest 1.12.1 and also upgraded rustc/cargo to the latest 1.82.0 version and will keep watching.

Hi @agentzh! We did find a problem with payload indices and point deletions. It seems this was introduced in 1.11.4 indeed.

Out of curiosity, are you using payload indices during deletion by any chance? If so I'd assume this to be related.

@timvisee yes! we've been using a lot of payload indexes on string and number typed fields. Glad you pin the bug down! May I ask which version has this issue fixed? Thanks a lot!

And yes, we use indexes during selective vector/payload deletion since we re-import existing data and any new data in different batches every few days.

Thanks for confirming.

We fixed it in #5346. It is not released yet, but we'll do so shortly. We now have a test in place that ensures the same problem doesn't happen in the future.

Sorry for the inconvenience.

We've just released Qdrant 1.12.2 which does fix this problem: https://github.com/qdrant/qdrant/releases/tag/v1.12.2

Please feel free to give it a try and let us know if issues persist.