Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorder semi and anti joins. #11815

Merged
Merged
Changes from 1 commit
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
18b7e51
I think we can reorder semi joins
Tmonster Jan 25, 2024
77d503e
I have access to the left and right sets and the join type. Now I sho…
Tmonster Jan 25, 2024
34ad120
can now estimate a semi join to be 20% of the left table
Tmonster Jan 25, 2024
d8f8f9f
have most everything working. some CEs not quite. Q05 tpch needs to b…
Tmonster Jan 26, 2024
8e6b730
Merge remote-tracking branch 'upstream/main' into reorder-semi-and-an…
Tmonster Jan 26, 2024
4ef7415
some debugging statements
Tmonster Jan 29, 2024
d0e4892
make format-fix
Tmonster Jan 29, 2024
9eecc85
fix code where numerator relations were not properly being merged. Al…
Tmonster Jan 29, 2024
2732612
add cross product join type
Tmonster Jan 29, 2024
bd1d164
fix join is reorderabe function. generate enums
Tmonster Jan 29, 2024
8be872d
pausing point
Tmonster Jan 29, 2024
b9885f6
fix last join issue and better min/max stats help on distinct count
Tmonster Jan 30, 2024
e7eaf10
remove min max stats changesg
Tmonster Jan 30, 2024
2183142
add back in removed test
Tmonster Jan 30, 2024
7f49a57
pausing to work on adding benchmarks for ingestion
Tmonster Jan 31, 2024
206109e
trying to fix this column lifetime analyzer bug
Tmonster Jan 31, 2024
830cdec
you can also change the bindings filters return. but that's not a goo…
Tmonster Feb 1, 2024
dfab1b6
remove changing what bindings filters return
Tmonster Feb 1, 2024
72cd1be
check tests one more time. still need to figure out logical as of joi…
Tmonster Feb 1, 2024
990b296
Merge branch 'reorder-semi-and-anti-joins' of github.com:Tmonster/duc…
Tmonster Feb 2, 2024
333f252
make format-fix
Tmonster Feb 2, 2024
7d7690c
might be a better solution, lets see if debug passes
Tmonster Feb 12, 2024
4414834
Merge remote-tracking branch 'upstream/main' into reorder_semi_anti_j…
Tmonster Feb 13, 2024
b22940f
remove debugging statements
Tmonster Feb 13, 2024
8f8edbd
fix logic for cross product relations
Tmonster Feb 13, 2024
d651dcf
skip test that I will remove eventually
Tmonster Feb 13, 2024
1fdb318
these ideas dont work tbh
Tmonster Feb 13, 2024
8fc0048
so close, but cant figure it out yet
Tmonster Feb 13, 2024
9827732
still need to track down this bug
Tmonster Feb 13, 2024
3ee5e65
follow this path. Somewhere in the test visit_operator_expressions_in…
Tmonster Feb 14, 2024
49548c1
I think we can reorder semi joins
Tmonster Jan 25, 2024
4ad73e1
I have access to the left and right sets and the join type. Now I sho…
Tmonster Jan 25, 2024
f665ea7
can now estimate a semi join to be 20% of the left table
Tmonster Jan 25, 2024
6c39fb1
have most everything working. some CEs not quite. Q05 tpch needs to b…
Tmonster Jan 26, 2024
c45bc52
some debugging statements
Tmonster Jan 29, 2024
c9bbc47
make format-fix
Tmonster Jan 29, 2024
7f1ab5e
fix code where numerator relations were not properly being merged. Al…
Tmonster Jan 29, 2024
57a21f2
add cross product join type
Tmonster Jan 29, 2024
30983ec
fix join is reorderabe function. generate enums
Tmonster Jan 29, 2024
818bceb
pausing point
Tmonster Jan 29, 2024
8b250a7
fix last join issue and better min/max stats help on distinct count
Tmonster Jan 30, 2024
168e1d0
remove min max stats changesg
Tmonster Jan 30, 2024
bf24aed
add back in removed test
Tmonster Jan 30, 2024
7802160
pausing to work on adding benchmarks for ingestion
Tmonster Jan 31, 2024
be6f854
trying to fix this column lifetime analyzer bug
Tmonster Jan 31, 2024
54d13eb
you can also change the bindings filters return. but that's not a goo…
Tmonster Feb 1, 2024
23a1854
remove changing what bindings filters return
Tmonster Feb 1, 2024
d57c740
check tests one more time. still need to figure out logical as of joi…
Tmonster Feb 1, 2024
3108292
make format-fix
Tmonster Feb 2, 2024
5afbd5b
might be a better solution, lets see if debug passes
Tmonster Feb 12, 2024
75f6d9d
remove debugging statements
Tmonster Feb 13, 2024
1d0b442
fix logic for cross product relations
Tmonster Feb 13, 2024
8012a31
skip test that I will remove eventually
Tmonster Feb 13, 2024
c8d88e0
these ideas dont work tbh
Tmonster Feb 13, 2024
21817fe
so close, but cant figure it out yet
Tmonster Feb 13, 2024
fbd8162
still need to track down this bug
Tmonster Feb 13, 2024
996fd64
follow this path. Somewhere in the test visit_operator_expressions_in…
Tmonster Feb 14, 2024
14dcd16
skip failing test
Tmonster Mar 28, 2024
d9441ad
Merge branch 'feature' into reorder_semi_anti_joins_easier_fix
Tmonster Apr 4, 2024
922fb49
no longer fails that test
Tmonster Apr 4, 2024
ee258b3
disable other test, see what else fails
Tmonster Apr 4, 2024
9910429
make format-fix
Tmonster Apr 4, 2024
a08db68
disable other failing test, see what else fails
Tmonster Apr 4, 2024
46d0114
basically everything works now. Just need to fix the cardinality esti…
Tmonster Apr 4, 2024
a26d06b
fix join order optimizer seg fault failure
Tmonster Apr 5, 2024
a3adc84
remove/clean up tests
Tmonster Apr 5, 2024
5076c09
clean up some unused code
Tmonster Apr 5, 2024
9ab4ca4
Merge branch 'reorder_semi_anti_joins_easier_fix' of github.com:Tmons…
Tmonster Apr 5, 2024
5a00256
remove randomly failing test
Tmonster Apr 5, 2024
faa0755
more code clean up
Tmonster Apr 5, 2024
71da791
make format-fix
Tmonster Apr 5, 2024
e4352af
comment out ensuring statististcs because arrow has not statistics
Tmonster Apr 5, 2024
1cb53ad
make format-fix (again)
Tmonster Apr 5, 2024
4d7b82e
rewrite mark join needs to be in more places
Tmonster Apr 8, 2024
e4d78f4
Merge remote-tracking branch 'upstream/feature' into reorder_semi_ant…
Tmonster Apr 8, 2024
9e538dd
distinct count should be max of 1 and the reported distinct count
Tmonster Apr 8, 2024
d9df4df
removed unused variable
Tmonster Apr 8, 2024
e01ab55
clean up test fileg
Tmonster Apr 8, 2024
790bd63
remove hashtag
Tmonster Apr 8, 2024
b6bb52a
clang tidy fixes #1
Tmonster Apr 9, 2024
c5a2a42
make format-fixes 2
Tmonster Apr 9, 2024
c9f56d1
pr comments 1
Tmonster Apr 9, 2024
7dc4bcb
add more comments for readability
Tmonster Apr 10, 2024
b998e86
it points to unused memory is because we remove entries from the subg…
Tmonster Apr 10, 2024
004f308
refactoring is a lot easier to think about if it is just a spanning tree
Tmonster Apr 12, 2024
6c8644f
lots of refactor done, need to understand where query graph edges com…
Tmonster Apr 22, 2024
d53a5f7
Merge branch 'feature' into reorder_semi_anti_joins_easier_fix_refactor
Tmonster Apr 22, 2024
3093b9a
got it all to compile and work finally
Tmonster Apr 22, 2024
3be1487
fixes with join order optimizer
Tmonster Apr 23, 2024
5bac735
why is q09 regressing?
Tmonster Apr 23, 2024
50ed8cb
refactor should be done now
Tmonster Apr 23, 2024
f30f26c
more small fixes
Tmonster Apr 23, 2024
28b9e76
remove break point
Tmonster Apr 23, 2024
02fbf90
revert script changes, actually remove unneeded break point
Tmonster Apr 23, 2024
01dc878
remove Print tdom info and other unused code
Tmonster Apr 23, 2024
a347cf7
Merge branch 'feature' into reorder_semi_anti_joins_easier_fix_refactor
Tmonster Apr 24, 2024
7553dd4
fix regression and refactor function name
Tmonster Apr 25, 2024
601ea05
remove break_here
Tmonster Apr 25, 2024
75bcac6
fixed q95 again.
Tmonster Apr 25, 2024
0d3c357
Merge remote-tracking branch 'upstream/feature' into reorder_semi_ant…
Tmonster Apr 30, 2024
6bc5e0e
remove unused filter strength variable. add constant semi_anti filter…
Tmonster May 1, 2024
eab427f
Merge branch 'feature' into reorder_semi_anti_joins_easier_fix_refactor
Tmonster May 3, 2024
a8ad4d0
will this pass CI?
Tmonster May 3, 2024
13e58e9
make format-fix
Tmonster May 3, 2024
fe6fdef
remove unused variable
Tmonster May 3, 2024
3fe233f
Merge remote-tracking branch 'upstream/feature' into reorder_semi_ant…
Tmonster May 3, 2024
db16bac
make format-fix
Tmonster May 3, 2024
4c5ef7d
Merge branch 'feature' into reorder_semi_anti_joins_easier_fix_refactor
Tmonster May 15, 2024
b3d7955
add explicit constructor
Tmonster May 15, 2024
9a02adb
make format-fix
Tmonster May 15, 2024
82043f3
change default semi anti selectivity to 5
Tmonster May 17, 2024
a7b60eb
add test to make sure push down is happening
Tmonster May 17, 2024
72ec97a
make format-fix
Tmonster May 21, 2024
3f039bc
CI should run again please
Tmonster May 30, 2024
f3ffde1
add patch for substrait for semi and anti joins
Tmonster Jun 4, 2024
fe56a6b
Merge remote-tracking branch 'upstream/feature' into reorder_semi_ant…
Tmonster Jun 4, 2024
8fa3935
actually apply patches
Tmonster Jun 4, 2024
336dafa
fix patch fileg
Tmonster Jun 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove changing what bindings filters return
  • Loading branch information
Tmonster committed Mar 28, 2024
commit 23a1854f26fa28a2c2cd9c0ba30cd42f659487f8
16 changes: 1 addition & 15 deletions src/planner/operator/logical_filter.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,21 +16,7 @@ void LogicalFilter::ResolveTypes() {
}

vector<ColumnBinding> LogicalFilter::GetColumnBindings() {
idx_t mark_index;
bool is_mark_join = false;
if (children[0]->type == LogicalOperatorType::LOGICAL_COMPARISON_JOIN) {
auto &join = children[0]->Cast<LogicalComparisonJoin>();
if (join.join_type == JoinType::MARK) {
is_mark_join = true;
mark_index = join.mark_index;
}
}
auto bindings = children[0]->GetColumnBindings();
if (is_mark_join) {
bindings.erase(std::remove_if(bindings.begin(), bindings.end(),
[&mark_index](ColumnBinding x) { return x.table_index == mark_index; }));
}
return MapBindings(bindings, projection_map);
return MapBindings(children[0]->GetColumnBindings(), projection_map);
}

// Split the predicates separated by AND statements
Expand Down