Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Umbrella][KPIP-2] Support FlinkSQL Engine #1322

Closed
45 tasks done
yanghua opened this issue Nov 2, 2021 · 15 comments
Closed
45 tasks done

[Umbrella][KPIP-2] Support FlinkSQL Engine #1322

yanghua opened this issue Nov 2, 2021 · 15 comments
Assignees
@yanghua @SteNicholas @link3280 @pan3793
Labels
kind:feature Feature request kind:umbrella This a umbrella ticket module:flink priority:major release highlight
Milestone
v1.5.0

Comments

@yanghua
Copy link
Contributor

yanghua commented Nov 2, 2021 •
edited by pan3793
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the proposal

Task list

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@yaooqinn yaooqinn added this to the 1.5.0 milestone Nov 2, 2021
@yaooqinn
Copy link
Member

yaooqinn commented Nov 2, 2021

I can’t wait to see this happen.

@yaooqinn yaooqinn pinned this issue Nov 2, 2021
@ulysses-you
Copy link
Contributor

great work

This was referenced Nov 4, 2021
Closed
Closed
Closed
This was referenced Dec 3, 2021
Closed
Closed
Closed
Closed
This was referenced Dec 16, 2021
Closed
Closed
Closed
@SteNicholas
Copy link
Member

@yanghua @yaooqinn , any other subtasks? In above task list, the tasks are completed.

@pan3793
Copy link
Member

pan3793 commented Dec 29, 2021

In above task list, the tasks are completed.

Not yet. The current implementation should be considered as a PoC version, and we do find some disadvantages of this approach, the next step is to investigate Flink application mode.

@yaooqinn
Copy link
Member

@yanghua @yaooqinn , any other subtasks? In above task list, the tasks are completed.

@yanghua could you add more uncompleted tasks to the list, like metaapis, operation logs etc

@yanghua
Copy link
Contributor Author

yanghua commented Dec 29, 2021

@yanghua @yaooqinn , any other subtasks? In above task list, the tasks are completed.

@yanghua could you add more uncompleted tasks to the list, like metaapis, operation logs etc

fine, there are still many operations that did not implement @SteNicholas . Will file more subtask issues.

@SteNicholas
Copy link
Member

@yanghua @yaooqinn @pan3793 , thanks for your reply. IMO, the current implementation doesn't affect the interface definition. Therefore, I could start to add the operations based on the current implementation. Right?

@yanghua
Copy link
Contributor Author

yanghua commented Dec 29, 2021

@yanghua @yaooqinn @pan3793 , thanks for your reply. IMO, the current implementation doesn't affect the interface definition. Therefore, I could start to add the operations based on the current implementation. Right?

yes, brother

@link3280
Copy link
Contributor

@yanghua Thanks for your great work! The proposal seems to be sort of lagged behind. It would be great if we can update it as well. I'll be happy to take some tasks if possible.

This was referenced Dec 29, 2021
Closed
Closed
Closed
Closed
@yanghua
Copy link
Contributor Author

yanghua commented Dec 29, 2021

@yanghua Thanks for your great work! The proposal seems to be sort of lagged behind. It would be great if we can update it as well. I'll be happy to take some tasks if possible.

You are right. The design doc does not match the implementation. Will update it later.

This was referenced Dec 29, 2021
Closed
Closed
pan3793 pushed a commit that referenced this issue Jan 27, 2022
@link3280 @pan3793
[KYUUBI #1839] Minor refactor OperationUtil
65d50e2 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This is a sub-task of KPIP-2 #1322.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1843 from link3280/improvemnt/KYUUBI-1839.

Closes #1839

b870427 [Paul Lin] [KYUUBI #1839] Improve docs and comments
a187a53 [Paul Lin] [KYUUBI #1839] Improve docs and comments
7c36c5c [Paul Lin] [KYUUBI #1839] Remove unused imports
48d611d [Paul Lin] [KYUUBI #1839] Rewrite ResultSetUtil using scala
92a2e3b [Paul Lin] [KYUUBI #1839] Rename OperationUtil to ResultSetUtil
811d3a7 [Paul Lin] [KYUUBI #1839] Minor refactor OperationUtil

Authored-by: Paul Lin <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
@link3280 link3280 mentioned this issue Jan 30, 2022
Closed
3 tasks
This was referenced Feb 7, 2022
Open
Closed
Closed
yanghua pushed a commit that referenced this issue Feb 9, 2022
@link3280 @yanghua
[KYUUBI #1838] Clean up query results after query operations finish
03c84f8 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

Query results would be cached in memory, and we should clean it up when all rows are fetched.

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This is a sub-task of KPIP-2 #1322.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1859 from link3280/feature/KYUUBI-1838.

Closes #1838

3cef076 [Paul Lin] [KYUUBI #1838] Improve logging message syntax
c6253de [Paul Lin] [KYUUBI #1838] Log cleanup exceptions
e0b1866 [Paul Lin] [KYUUBI #1838] Clean up query results after query operations finish

Authored-by: Paul Lin <[email protected]>
Signed-off-by: yanghua <[email protected]>
@link3280 link3280 mentioned this issue Feb 9, 2022
Closed
3 tasks
This was referenced Feb 16, 2022
Closed
Closed
@link3280 link3280 mentioned this issue Feb 20, 2022
Closed
3 tasks
yaooqinn pushed a commit that referenced this issue Feb 22, 2022
@link3280 @yaooqinn
[KYUUBI #1883] Support max result rows for Flink queries
e54d431 
### _Why are the changes needed?_

Currently, Flink engine would pull all result rows into memory before returning it to the client. This would be problematic for large result sets and infinite result sets.

This is a sub-task of KPIP-2 #1322.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1938 from link3280/feature/FLINK-1883.

Closes #1883

80020ce [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala
1b95822 [Paul Lin] [KYUUBI #1883] Avoid allocating too much buffer space
5be7535 [Paul Lin] [KYUUBI #1883] Support max result rows for Flink queries

Authored-by: Paul Lin <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
@link3280 link3280 mentioned this issue Mar 3, 2022
Closed
3 tasks
@yaooqinn yaooqinn mentioned this issue Mar 11, 2022
Open
9 tasks
@yaooqinn
Copy link
Member

retargeting improvement PR to v1.6 and collection to another ticket #2100 to track

@yaooqinn yaooqinn unpinned this issue Mar 11, 2022
@link3280 link3280 mentioned this issue Mar 13, 2022
Closed
3 tasks
yaooqinn pushed a commit that referenced this issue Mar 14, 2022
@link3280 @yaooqinn
[KYUUBI #2118] [SUB-TASK][KPIP-2] Support session jars management
537a0f8 
### _Why are the changes needed?_

Adding and removing jars for a session at runtime is crucial for using UDFs and connectors, which we should support.

This is a sub-task of KPIP-2 #1322.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2118 from link3280/KYUUBI-2002.

Closes #2118

fd3a4b4 [Paul Lin] [KYUUBI #2002] Support show jars operations in PlanOnlyStatement
56cdd12 [Paul Lin] [KYUUBI #2002] Support show jars operations
fbdb009 [Paul Lin] [KYUUBI #2002] Support add/remove jar operations

Authored-by: Paul Lin <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
@SteNicholas SteNicholas mentioned this issue Mar 14, 2022
Closed
3 tasks
@pan3793
Copy link
Member

pan3793 commented Mar 16, 2022

Thanks to all contributors who participated, I'm going to close this ticket, and please go ahead in #2100

@pan3793 pan3793 closed this as completed Mar 16, 2022
@cxzl25 cxzl25 mentioned this issue Sep 19, 2022
Closed
3 tasks
pan3793 pushed a commit that referenced this issue Sep 28, 2022
@cxzl25 @pan3793
[KYUUBI #3519] Flink SQL Engine - GetColumns Operation
8419b7b 
### _Why are the changes needed?_
#1322
#2129

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3519 from cxzl25/flink_get_columns.

Closes #3519

ab81776 [sychen] add column size
efdf1b9 [sychen] indent
630e907 [sychen] refactor
48a79d5 [sychen] add ut
69763bd [sychen] GetColumns
8e5e6c5 [sychen] GetColumns

Authored-by: sychen <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
pan3793 pushed a commit that referenced this issue Sep 28, 2022
@cxzl25 @pan3793
[KYUUBI #3519] Flink SQL Engine - GetColumns Operation
b87cf97 
### _Why are the changes needed?_
#1322
#2129

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3519 from cxzl25/flink_get_columns.

Closes #3519

ab81776 [sychen] add column size
efdf1b9 [sychen] indent
630e907 [sychen] refactor
48a79d5 [sychen] add ut
69763bd [sychen] GetColumns
8e5e6c5 [sychen] GetColumns

Authored-by: sychen <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 8419b7b)
Signed-off-by: Cheng Pan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yanghua yanghua

@SteNicholas SteNicholas

@link3280 link3280

@pan3793 pan3793

Labels
kind:feature Feature request kind:umbrella This a umbrella ticket module:flink priority:major release highlight
Projects
None yet
Milestone
v1.5.0
Development

No branches or pull requests
6 participants
@yanghua @yaooqinn @SteNicholas @ulysses-you @link3280 @pan3793