feat: support json index #36750

sunby · 2024-10-10T10:46:53Z

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later.

basic usage:

collection.create_index("json_field", {"index_type": "INVERTED",
    "params": {"json_cast_type": DataType.STRING, "json_path":
'json_field["a"]["b"]'}})

There are some limits to use this index:

If a record does not have the json path you specify, it will be ignored and there will not be an error.
If a value of the json path fails to be cast to the type you specify, it will be ignored and there will not be an error.
A specific json path can have only one json index.
If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version.

mergify · 2024-10-10T10:48:06Z

@sunby Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

mergify · 2024-10-10T11:11:52Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-10T11:17:16Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-10-12T04:00:08Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

internal/core/src/segcore/SegmentInterface.h

internal/datacoord/index_service.go

internal/querynodev2/segments/segment_loader.go

mergify · 2024-10-31T09:41:37Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-31T10:36:01Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2024-10-31T10:54:12Z

Codecov Report

Attention: Patch coverage is 69.34673% with 61 lines in your changes missing coverage. Please review.

Project coverage is 82.92%. Comparing base (2a02bbe) to head (42a790c).
Report is 7 commits behind head on master.

❗ Current head 42a790c differs from pull request most recent head d0a0b18

Please upload reports for the commit d0a0b18 to get more accurate results.

Files with missing lines	Patch %	Lines
internal/datacoord/index_service.go	67.88%	26 Missing and 9 partials ⚠️
internal/datacoord/index_meta.go	60.00%	8 Missing and 4 partials ⚠️
internal/util/indexparamcheck/inverted_checker.go	30.00%	6 Missing and 1 partial ⚠️
internal/querynodev2/segments/segment.go	81.81%	3 Missing and 1 partial ⚠️
internal/querynodev2/segments/segment_l0.go	0.00%	2 Missing ⚠️
internal/querynodev2/segments/segment_loader.go	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #36750      +/-   ##
==========================================
+ Coverage   81.11%   82.92%   +1.80%     
==========================================
  Files        1395     1099     -296     
  Lines      197434   170918   -26516     
==========================================
- Hits       160154   141728   -18426     
+ Misses      31641    23548    -8093     
- Partials     5639     5642       +3

Components	Coverage Δ
Client	`79.53% <ø> (ø)`
Core	`∅ <ø> (∅)`
Go	`83.06% <70.77%> (+0.01%)`	⬆️

Files with missing lines	Coverage Δ
internal/querycoordv2/checkers/index_checker.go	`79.09% <100.00%> (ø)`
internal/querycoordv2/meta/segment_dist_manager.go	`89.84% <ø> (ø)`
internal/querynodev2/services.go	`87.42% <100.00%> (+0.65%)`	⬆️
pkg/common/common.go	`77.18% <ø> (ø)`
internal/querynodev2/segments/segment_loader.go	`73.00% <95.83%> (+0.13%)`	⬆️
internal/querynodev2/segments/segment_l0.go	`70.40% <0.00%> (-1.47%)`	⬇️
internal/querynodev2/segments/segment.go	`65.24% <81.81%> (+0.08%)`	⬆️
internal/util/indexparamcheck/inverted_checker.go	`65.00% <30.00%> (-35.00%)`	⬇️
internal/datacoord/index_meta.go	`94.06% <60.00%> (-1.26%)`	⬇️
internal/datacoord/index_service.go	`89.69% <67.88%> (-3.60%)`	⬇️

... and 318 files with indirect coverage changes

mergify · 2024-10-31T10:59:20Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaofan-luan · 2024-11-01T00:24:03Z

@sunby

Are you done with this feature so I can start to review it?

mergify · 2024-11-04T07:07:56Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T07:07:56Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T07:14:05Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T09:00:37Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T09:01:18Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:01:01Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:34:47Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-01-02T06:56:19Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-02T06:59:39Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-02T07:14:37Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-01-02T09:04:49Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

sunby · 2025-01-03T02:36:21Z

rerun go-sdk

mergify · 2025-01-03T03:21:35Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-06T11:13:36Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-06T12:18:21Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sunby · 2025-01-09T02:26:22Z

/run-cpu-e2e

mergify · 2025-01-09T03:32:17Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sunby · 2025-01-09T06:07:23Z

/run-cpu-e2e

mergify · 2025-01-09T07:24:33Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-01-10T05:02:12Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-10T06:05:50Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later. basic usage: ``` collection.create_index("json_field", {"index_type": "INVERTED", "params": {"json_cast_type": DataType.STRING, "json_path": 'json_field["a"]["b"]'}}) ``` There are some limits to use this index: 1. If a record does not have the json path you specify, it will be ignored and there will not be an error. 2. If a value of the json path fails to be cast to the type you specify, it will be ignored and there will not be an error. 3. A specific json path can have only one json index. 4. If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version. Signed-off-by: sunby <sunbingyi1992@gmail.com>

Signed-off-by: sunby <sunbingyi1992@gmail.com>

mergify · 2025-01-13T03:25:15Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T03:41:28Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: sunby <sunbingyi1992@gmail.com>

mergify · 2025-01-13T09:17:23Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T09:30:32Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sre-ci-robot added area/internal-api size/XL Denotes a PR that changes 500-999 lines. labels Oct 10, 2024

sre-ci-robot requested review from aoiasd and bigsheeper October 10, 2024 10:47

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Oct 10, 2024

sunby force-pushed the add_json_index branch from 0f2e61d to e2a08b7 Compare October 10, 2024 10:47

mergify bot added the do-not-merge/missing-related-issue label Oct 10, 2024

sunby force-pushed the add_json_index branch 2 times, most recently from 7ececff to 68a0644 Compare October 10, 2024 10:52

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Oct 12, 2024

tedxu reviewed Oct 14, 2024

View reviewed changes

sunby force-pushed the add_json_index branch from ff47ada to 7eac44a Compare October 31, 2024 09:07

sunby force-pushed the add_json_index branch from bf1de33 to f87196f Compare November 5, 2024 08:16

sre-ci-robot added the area/compilation label Nov 5, 2024

sunby force-pushed the add_json_index branch from c4d0ae4 to be52ebb Compare January 2, 2025 08:11

sunby force-pushed the add_json_index branch from b60ba93 to 5e44e8b Compare January 6, 2025 11:12

sunby force-pushed the add_json_index branch from 5e44e8b to d8aa397 Compare January 10, 2025 04:10

sunby added 5 commits January 13, 2025 10:28

fix format

50f344d

Signed-off-by: sunby <sunbingyi1992@gmail.com>

rename

db571e5

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix format

d493255

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix

42a790c

Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby force-pushed the add_json_index branch from d8aa397 to 42a790c Compare January 13, 2025 02:29

fix ut

d0a0b18

Signed-off-by: sunby <sunbingyi1992@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support json index #36750

feat: support json index #36750

sunby commented Oct 10, 2024 •

edited

Loading

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 12, 2024

mergify bot commented Oct 31, 2024

mergify bot commented Oct 31, 2024

codecov bot commented Oct 31, 2024 •

edited

Loading

mergify bot commented Oct 31, 2024

xiaofan-luan commented Nov 1, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

sunby commented Jan 3, 2025

mergify bot commented Jan 3, 2025

mergify bot commented Jan 6, 2025

mergify bot commented Jan 6, 2025

sunby commented Jan 9, 2025

mergify bot commented Jan 9, 2025

sunby commented Jan 9, 2025

mergify bot commented Jan 9, 2025

mergify bot commented Jan 10, 2025

mergify bot commented Jan 10, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

feat: support json index #36750

Are you sure you want to change the base?

feat: support json index #36750

Conversation

sunby commented Oct 10, 2024 • edited Loading

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 12, 2024

mergify bot commented Oct 31, 2024

mergify bot commented Oct 31, 2024

codecov bot commented Oct 31, 2024 • edited Loading

Codecov Report

mergify bot commented Oct 31, 2024

xiaofan-luan commented Nov 1, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 2, 2025

sunby commented Jan 3, 2025

mergify bot commented Jan 3, 2025

mergify bot commented Jan 6, 2025

mergify bot commented Jan 6, 2025

sunby commented Jan 9, 2025

mergify bot commented Jan 9, 2025

sunby commented Jan 9, 2025

mergify bot commented Jan 9, 2025

mergify bot commented Jan 10, 2025

mergify bot commented Jan 10, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

sunby commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 31, 2024 •

edited

Loading