opt: increase cost for table descriptor fetch during virtual scan #56345
Conversation
This commit bumps the cost of each virtual scan to 25*randIOCostFactor from its previous value of 10*randIOCostFactor. This new value threads the needle so that a lookup join will still be chosen if the predicate is very selective, but the plan for the PGJDBC query identified in cockroachdb#55140 no longer includes lookup joins. Fixes cockroachdb#55140 Release note (performance improvement): Adjusted the cost model in the optimizer so that the optimizer is less likely to plan a lookup join into a virtual table. Performing a lookup join into a virtual table is expensive, so this change will generally result in better performance for queries involving joins with virtual tables.
|
This change is |
|
I ran the sample workload in #55140 and confirmed that the new plan is much faster (I think I was confused before since I didn't realize how much caching affects performance for this query). On my single-node cluster on my laptop, the old plan executes in 3.565s and the new plan executes in 275ms. |
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @mgartner, @RaduBerinde, @rafiss, and @rytaft)
pkg/sql/opt/exec/execbuilder/testdata/virtual, line 106 at r1 (raw file):
EXPLAIN SELECT * FROM pg_constraint INNER LOOKUP JOIN pg_index on true # Test that a gnarly ORM query from ActiveRecord uses lookup joins for speed.
Looks like this was specifically testing that we use lookup join
|
pkg/sql/opt/exec/execbuilder/testdata/virtual, line 106 at r1 (raw file): Previously, RaduBerinde wrote…
We are still doing two lookup joins... Looks like the relevant issue is #48178. My sense from the issue description is that we definitely want to use at least some lookup joins, but it's not clear to me that it has to be this exact plan (and the existing plan does still have some hash/merge joins). I tried running this query on my laptop to see if there was a performance difference, but I couldn't see any difference. I may not have the right set of tables/settings to see any difference, though. @rafiss do you know how I might be able to test this? |
There was a problem hiding this comment.
Reviewable status:
pkg/sql/opt/exec/execbuilder/testdata/virtual, line 106 at r1 (raw file):
Previously, rytaft (Rebecca Taft) wrote…
We are still doing two lookup joins...
Looks like the relevant issue is #48178. My sense from the issue description is that we definitely want to use at least some lookup joins, but it's not clear to me that it has to be this exact plan (and the existing plan does still have some hash/merge joins).
I tried running this query on my laptop to see if there was a performance difference, but I couldn't see any difference. I may not have the right set of tables/settings to see any difference, though. @rafiss do you know how I might be able to test this?
A good way to test this is to try running it on a cluster that has ~100 tables with multiple foreign keys per table.
There was a problem hiding this comment.
Reviewed 6 of 6 files at r1.
Reviewable status:
|
pkg/sql/opt/exec/execbuilder/testdata/virtual, line 106 at r1 (raw file): Previously, rafiss (Rafi Shamim) wrote…
I did this experiment and found that the times are nearly identical (~120 ms in each case). |
|
TFTR! bors r+ |
|
This PR was included in a batch that was canceled, it will be automatically retried |
|
Build succeeded: |
This commit bumps the cost of each virtual scan to
25*randIOCostFactorfrom its previous value of
10*randIOCostFactor. This new value threadsthe needle so that a lookup join will still be chosen if the predicate
is very selective, but the plan for the PGJDBC query identified in #55140
no longer includes lookup joins.
Fixes #55140
Release note (performance improvement): Adjusted the cost model in
the optimizer so that the optimizer is less likely to plan a lookup
join into a virtual table. Performing a lookup join into a virtual
table is expensive, so this change will generally result in better
performance for queries involving joins with virtual tables.