Fix #1505, #998, #1225 - results after remove are correct #1528
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes #1505, #998, and #1225 by making changes to
t_gstate::get_pkeyed_table
, which is used to "seed" new views with the state of the underlying table. However, if the table had removed rows and a string primary key, it would enter a block of code that rebuilt the index column and did so with an offset of 0 if an empty string ("") was in the primary keys, and 1 otherwise. This caused the internalpsp_pkey
index column to be off-by-one, which does not appear as a problem visually but caused issues when filtering, as Perspective was now returning the wrong rows entirely in the filter.By throughly testing the logic and removing the offending code block, the issue is fixed and filters after removes and updates should now return the correct rows. This behavior also only materializes when a new view is created after the remove/update, as
get_pkeyed_table
is only used for new views. Thus, we've added a battery of tests asserting the equality of behavior when a new view is created and when we reuse existing views. This provides confidence that removing the block of code is, in fact, the correct solution and guarantees correct behavior.Additionally, I've fixed #1225 by making
table.size()
return the size of the mapping of the gnode state instead of the gnode state master table itself.A moment to opine; all
t_data_tables
are allocated as a contiguous chunk of memory, and when rows are removed, the physical row is not removed from the master table, as it would cause a re-allocation. Instead, the row is cleared (its values removed) and its primary key removed fromm_mapping
, which is a mapping of primary key scalars to row indices on the master table. The row on the master table is then kept for later updates, which will seek to reuse existing rows that have been removed instead of appending new rows and thus causing more allocations.This means that
t_gstate.m_table.size()
is always the totality of all rows appended throughout its lifetime, and will always be>=
to the size ofm_mapping
. In contrast,m_mapping
is a hashmap and primary keys can be evicted at will. Thus,m_mapping.size()
is alwaysappended rows - removed rows
, and is the correctsize()
of the table to be provided to the external user-facing API. I've held off on making this change for some time in order to better understand the remove mechanics, and I think this batch of fixes is a good chance to correct that behavior.Finally I had to make some ESLint changes after rebasing to master, and this PR includes those changes as well to Javascript test files (commenting out some unused imports).