Generally, update is implemented through delete+add, and delete is just marking deletion. When there are a large number of updates, a large amount of invalid data will be stored in the system, which brings two problems: 1. Waste of memory resources; 2. There are also a large number of invalid indexes in the inverted index, and the retrieval performance decreases with the number of updates. In order to solve these two problems, the inplace-update+compaction scheme was designed and implemented.

1. For fixed-length data, directly overwrite the old value with the new value;
2. For variable length data, mark the old value as invalid, add new value, and do compaction regularly to clean up invalid old value

Table includes fixed-length fields, such as Int, Long, Double, etc., and variable-length fields such as string.

• When a fixed-length field is updated, the new value is directly copied to the field storage address, overwriting the old value.
• If the string field is updated, there are two situations
  1. The length of the new value is less than or equal to the old value, which means that the space of the old value can store the new value, then the new value is directly copied to the address of the old value, and the length of the field s is updated;
  2. If the length of the new value is greater than the old value, copy the new value to the end of the string field memory block (str_mem), and then update the address offset and length of the string field

Because the dimension will not change when the vector is updated, it directly overwrites the old value when updating vector. However, after the vector is updated, the index needs to be rebuilt to be valid in the search results, so the updated vector id is pushed to a queue, and the real-time indexing thread pops the updated vector id from the queue, and then calculates the index and adds it to the inverted index.

When updating an inverted index, first determine whether the new_bucket to which the new index belongs is the same as the old_bucket of the old index. If it is, the old index is directly overwritten with the new index; if not, the new index is added to the end of new_bucket. Update the bucket number and position offset of the index, and mark the old index in old_bucket as deleted(vid|0x80000000)

Compaction is executed in the bucket, the specific steps are as follows

1. Create a new memory space
2. Traverse all the vids in the bucket to determine whether they are marked for deletion
3. If the vid is not deleted, copy its index to the new memory space and update its position offset

Two CPU machines: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz 32 core, memory: 64G

• machine1：master, router
• machine2：ps

The version implemented in this article will be called the compact version, which will be compared and tested with the previous stable version v1.0.2. Each version performs 3 tests: 1. add 300,000 documents; 2. update all documents for 10 times, equivalent to 3 million insert operations; 3. update all documents for 30 times, equivalent to 9 million insert operations.

{
    "name":"space_re",
    "partition_num": 1,
    "replica_num": 1,
    "engine": {
        "name": "gamma",
        "index_size": 10000,
        "max_size": 20000000,
        "nprobe": 10,
    "metric_type": "InnerProduct",
    "ncentroids": 256,
    "nsubvector": 4
    },
    "properties": {
        "article_type": {
            "type": "string",
            "index": false
        },
        "article_vector": {
            "type": "vector",
            "dimension": 128
        }
    }
}

The performance test tool uses ab, and the tested parameters are as follows. In param.json is a batch query of 5 128-dimensional vectors ab -n 100000 -c 80 -p param.json -T application/json http://ip:port/db_re/space_re/_msearch

As can be seen from Figure 5 and Figure 6, the compact version has a stable QPS at 1700 and tp99 at 141ms when the data number is 300,000, 3 million and 9 million. With the increase of the data number, the v1.0.2 version QPS dropped significantly, and latency rose sharply. In summary, the compact version implemented in this article prevents retrieval performance from degrading as the update data increases.