Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

disable mlock #1885

Merged
merged 1 commit into from
May 12, 2023
Merged

disable mlock #1885

merged 1 commit into from
May 12, 2023

Conversation

generall
Copy link
Member

Resolves:

mnt-qdrant_bench-1  | [2023-05-12T10:40:42.502Z ERROR segment::vector_storage::mmap_vectors] Failed to lock deleted flags for quantized mmap segment in memory: Cannot allocate memory (os error 12)

in case of using inside docker with memory limits

@generall generall requested a review from timvisee May 12, 2023 12:47
timvisee
timvisee approved these changes May 12, 2023
View reviewed changes
Copy link
Member

@timvisee timvisee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

If we remove the mlock functions all together we can possibly simplify the mmap interface a bit. But this is good enough for now!

timvisee

This comment was marked as duplicate.

Sign in to view

@generall generall merged commit ce2763f into dev May 12, 2023
@timvisee
Copy link
Member

To give some more insight in this: the amount of lockable memory is restricted by the lock resource usage limit.

There are global ulimit settings for all containers. Providing or not providing --memory= has no effect on this. I quickly checked ulimit on my machine. The (default) low limit for lockable memory in containers explain the locking issue within Docker.

Host machine:

$ ulimit -l
4096216

Docker container:

$ docker run --rm -it --memory=1G debian
root@a53653df85ce:/# ulimit -l
8192

@timvisee timvisee mentioned this pull request May 16, 2023
Merged
8 tasks
generall added a commit that referenced this pull request May 17, 2023
@generall
disable mlock (#1885)
bbae883
@agourlay agourlay deleted the disable-mmap-mlock branch July 12, 2023 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timvisee timvisee timvisee approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@generall @timvisee