Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [null & default] Group by field does not support null #36264

Open
1 task done
binbinlv opened this issue Sep 13, 2024 · 8 comments
Open
1 task done

[Bug]: [null & default] Group by field does not support null #36264

binbinlv opened this issue Sep 13, 2024 · 8 comments
Assignees
@smellthemoon
Labels
kind/bug Issues or changes related a bug kind/enhancement Issues or changes related to enhancement triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.3

Comments

@binbinlv
Copy link
Contributor

binbinlv commented Sep 13, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20240913-375cb44b
- Deployment mode(standalone or cluster):both
- MQ type(rocksmq, pulsar or kafka):    all
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc78
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Group by field does not support null

RPC error: [search], <MilvusException: (code=1100, message=groupBy field(nullableFid) not support nullable == true: invalid parameter)>, <Time:{'RPC start': '2024-09-13 19:23:44.507598', 'RPC error': '2024-09-13 19:23:44.557512'}>
Traceback (most recent call last):
  File "./default.py", line 50, in <module>
    res1 = collection.search(vectors[:nq], "float_vector", default_search_params, limit, group_by_field="nullableFid", output_fields=["nullableFid", "int32"])
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 801, in search
    resp = conn.search(
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/decorators.py", line 141, in handler
    raise e from e
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/decorators.py", line 137, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/decorators.py", line 176, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/decorators.py", line 116, in handler
    raise e from e
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 800, in search
    return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 741, in _execute_search
    raise e from e
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 734, in _execute_search
    check_status(response.status)
  File "/Users/binbin/milvus_latest/lib/python3.8/site-packages/pymilvus/client/utils.py", line 63, in check_status
    raise MilvusException(status.code, status.reason, status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=groupBy field(nullableFid) not support nullable == true: invalid parameter)>

Expected Behavior

Group by field supports null

Steps To Reproduce

from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
import json
import random

connections.connect()

dim = 128
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
double_field = FieldSchema(name="nullableFid", dtype=DataType.VARCHAR, nullable=True, max_length=100, is_partition_key=True)
int32_field = FieldSchema(name="int32", dtype=DataType.INT64, default_value=3)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim, nullable=False)
schema = CollectionSchema(fields=[int64_field, double_field, int32_field,float_vector])
utility.drop_collection("test")
collection = Collection("test", schema=schema)
res = collection.schema
print(res)
varchar_scalar_index = "TRIE"
scalar_index_params = {"index_type": varchar_scalar_index, "params": {}}

collection.create_index("nullableFid", scalar_index_params, index_name="index_name_0")
#index = {"index_type": "DISKANN", "metric_type": "L2", "params": {}}

index = "HNSW"
params = {'ef': 64}
default_index = {"index_type": index, "params": params, "metric_type": "COSINE"}

nb = 5000
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
data = [[i for i in range(nb)], [None for _ in range(nb)],[], vectors]
#  equals to data1 = [[1,2], [None,None],[None,None], vectors]
data1 = [[1,2], [],[], vectors]

collection.insert(data=data)
#collection.upsert(data=data1)
collection.create_index("float_vector", index, index_name="index_name_1")
collection.load()
collection.flush()
res = collection.num_entities
print(res)
default_search_params = {"metric_type": "", "params": {}}
limit = 1000
nq = 2
import time
start = time.time()
res1 = collection.search(vectors[:nq], "float_vector", default_search_params, limit, group_by_field="nullableFid", output_fields=["nullableFid", "int32"])
end = time.time() - start
print(res1)
print("search successfully in %f s" % end)

Milvus Log

https://grafana-4am.zilliz.cc/explore?orgId=1&panes=%7B%22Woc%22:%7B%22datasource%22:%22vhI6Vw67k%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22devops%5C%22,namespace%3D%5C%22chaos-testing%5C%22,pod%3D~%5C%22default-null-test-bgkie.%2A%5C%22%7D%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1

Anything else?

No response
@binbinlv binbinlv added kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Sep 13, 2024
@binbinlv binbinlv added this to the 2.5.0 milestone Sep 13, 2024
@binbinlv binbinlv added the kind/enhancement Issues or changes related to enhancement label Sep 13, 2024
@smellthemoon
Copy link
Contributor

by design.

@xiaofan-luan
Copy link
Collaborator

maybe we can improve this?
null doesn't below to any group

@smellthemoon
Copy link
Contributor

maybe we can improve this?
null doesn't below to any group

yes, null will support it later. may be in 2.5.x?

@stale Stale
Copy link

stale bot commented Oct 29, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Oct 29, 2024
@binbinlv
Copy link
Contributor Author

binbinlv commented Nov 7, 2024

keep this as defer issue.

@stale stale bot closed this as completed Nov 16, 2024
@binbinlv
Copy link
Contributor Author

binbinlv commented Nov 16, 2024 •
edited
Loading

keep this open to track as defer issue.

@binbinlv binbinlv reopened this Nov 16, 2024
@stale stale bot removed the stale indicates no udpates for 30 days label Nov 16, 2024
@stale Stale
Copy link

stale bot commented Dec 20, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Dec 20, 2024
@yanliang567 yanliang567 modified the milestones: 2.5.0, 2.5.1 Dec 24, 2024
@stale stale bot removed the stale indicates no udpates for 30 days label Dec 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.5.1, 2.5.2 Dec 30, 2024
@yanliang567 yanliang567 modified the milestones: 2.5.2, 2.5.3 Jan 6, 2025
@yanliang567
Copy link
Contributor

keep it active

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smellthemoon smellthemoon

Labels
kind/bug Issues or changes related a bug kind/enhancement Issues or changes related to enhancement triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.3
Development

No branches or pull requests
4 participants
@smellthemoon @yanliang567 @xiaofan-luan @binbinlv