lightning: evaluate all generated columns even if they are virtual (#1407)

[ti-chi-bot]

This is an automated cherry-pick of #1407

What problem does this PR solve?

Fix #1404

What is changed and how it works?

Expression indices rely on a virtual generated column, which is not calculated by Lightning (unless depended by a stored generated column). This caused the virtual generated column to contain invalid value and thus invalidate the index.

This PR just calculates all generated columns no matter they are virtual or stored.

Check List

Tests

• Integration test

Code changes

Side effects

• Possible performance regression
  • Tables with only virtual generated columns will experience longer encoding time.

Related changes

• Need to cherry-pick to the release branch
  • release-4.0, 5.0, 5.1 (everywhere that creating an index on virtual generated column is allowed)

Release note

• Expression index and index depending on virtual generated columns are now valid. Previously these indices are broken when importing through Lightning local or importer backend.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• glorv
• kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[kennytm]

/run-integration-test

[kennytm]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: c4329dc

[ti-chi-bot]

@ti-chi-bot: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[kennytm]

/build