Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-24.3: opt: add new optimizer session settings #140255

Merged

Conversation

mgartner
Copy link
Collaborator

@mgartner mgartner commented Jan 31, 2025

Backport:

Please see individual PRs for details.

/cc @cockroachdb/release

Release justification: Optimizer changes gated behind session settings.

This is a mechanical change that will set up the ability to use
CostFlags in a future commit.

Release note: None
@mgartner mgartner requested a review from a team January 31, 2025 19:52
@mgartner mgartner requested review from a team as code owners January 31, 2025 19:52
@mgartner mgartner requested review from rytaft and removed request for a team January 31, 2025 19:52
Copy link

blathers-crl bot commented Jan 31, 2025

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Backports should only be created for serious
    issues
    or test-only changes.
  • Backports should not break backwards-compatibility.
  • Backports should change as little code as possible.
  • Backports should not change on-disk formats or node communication protocols.
  • Backports should not add new functionality (except as defined
    here).
  • Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
  • All backports must be reviewed by the owning areas TL. For more information as to how that review should be conducted, please consult the backport
    policy
    .
If your backport adds new functionality, please ensure that the following additional criteria are satisfied:
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
  • Your backport must be accompanied by a post to the appropriate Slack
    channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

@blathers-crl blathers-crl bot added the backport Label PR's that are backports to older release branches label Jan 31, 2025
Copy link

blathers-crl bot commented Jan 31, 2025

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@mgartner mgartner force-pushed the backport24.3-137984-139985-140065 branch from 4bf23a1 to 6033c3d Compare January 31, 2025 20:27
Copy link
Collaborator

@rytaft rytaft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 53 of 53 files at r1, 35 of 35 files at r2, 15 of 15 files at r3, 20 of 20 files at r4, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @mgartner)

@mgartner
Copy link
Collaborator Author

blathers backport 24.3.4-rc

rytaft and others added 2 commits January 31, 2025 16:35
Added a hint to avoid full scans (see release note below for details).
To support this change, added a field to memo.Cost with a new type
memo.CostFlags, which contains a number of boolean flags and supports
"multi-dimensional costing". This allows the optimizer to compare plans
based on the flags set in addition to the single-dimensional float64
cost. For example, plans with the new FullScanPenalty cost flag enabled
will always be more expensive than plans without any cost flags, even
if the base float64 cost is lower.

The new CostFlags type also includes a flag for HugeCostPenalty, which
must be set for plans with "hugeCost". This ensures that existing
hints that use hugeCost still work if some other cost flags are set,
since HugeCostPenalty takes precedence over other cost flags.

This new CostFlags field is needed to support hints that do not cause an
error if the optimizer cannot find a plan complying with the hint. This
is needed because the previous approach of simply using "hugeCost" to
avoid certain plans meant that if such plans were unavoidable, we could
not effectively compare plans with cost greater than hugeCost due to
loss of floating point precision.

Informs cockroachdb#79683

Release note (sql change): Added support for a new index hint,
AVOID_FULL_SCAN, which will prevent the optimizer from planning a
full scan for the specified table if any other plan is possible. The
hint can be used in the same way as other existing index hints. For
example, SELECT * FROM table_name@{AVOID_FULL_SCAN};. This hint is
similar to NO_FULL_SCAN, but will not error if a full scan cannot be
avoided. Note that normally a full scan of a partial index would not
be considered a "full scan" for the purposes of the NO_FULL_SCAN and
AVOID_FULL_SCAN hints, but if the user has explicitly forced the
partial index via FORCE_INDEX=index_name, we do consider it a full
scan.
Release note (sql change): The `optimizer_prefer_bounded_cardinality`
session setting has been added which instructs the optimizer to prefer
query plans where every expression has a guaranteed upper-bound on the
number of rows it will process. This may help the optimizer produce
better query plans in some cases. This setting is disabled by default.
@mgartner mgartner force-pushed the backport24.3-137984-139985-140065 branch from 6033c3d to 9e1288a Compare January 31, 2025 22:19
Informs cockroachdb#64570
Informs cockroachdb#130201

Release note (sql change): The `optimizer_min_row_count` session setting
has been added which sets a lower bound on row count estimates for
relational expressions during query planning. A value of zero, which is
the default, indicates no lower bound. Note that if this is set to a
value greater than zero, a row count of zero can still be estimated for
expressions with a cardinality of zero, e.g., for a contradictory
filter. Setting this to a value higher than 0, such as 1, may yield
better query plans in some cases, such as when statistics are frequently
stale and inaccurate.
@mgartner mgartner force-pushed the backport24.3-137984-139985-140065 branch from 9e1288a to 279d22f Compare January 31, 2025 23:19
@mgartner mgartner merged commit 462d36a into cockroachdb:release-24.3 Jan 31, 2025
20 of 21 checks passed
@mgartner mgartner deleted the backport24.3-137984-139985-140065 branch January 31, 2025 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Label PR's that are backports to older release branches
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants