Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KYUUBI #6411] Grpc common support #6412

Closed
wants to merge 37 commits into from
Closed

[KYUUBI #6411] Grpc common support #6412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
dd39efd
fix-#4057 info:
davidyuan1223 Jan 13, 2023
86e4e1c
fix-#4057 info: modify the shellcheck errors file in ./bin
davidyuan1223 Jan 13, 2023
cb11935
Merge remote-tracking branch 'origin/fix-#4057' into fix-#4057
davidyuan1223 Jan 13, 2023
55a0a43
Merge pull request #10 from xiaoyuandajian/fix-#4057
davidyuan1223 Jan 13, 2023
c48ad38
remove the used blank lines
davidyuan1223 Mar 5, 2023
16237c2
Merge branch 'apache:master' into master
davidyuan1223 Apr 2, 2023
bfa6cbf
Merge branch 'apache:master' into master
davidyuan1223 Sep 22, 2023
e244029
Merge branch 'apache:master' into master
davidyuan1223 Sep 26, 2023
b616044
fix_4186
davidyuan1223 Sep 26, 2023
360d183
Merge branch 'master' into fix_4186
davidyuan1223 Sep 26, 2023
c83836b
Merge pull request #11 from davidyuan1223/fix_4186
davidyuan1223 Sep 26, 2023
40e80d9
Revert "fix_4186"
davidyuan1223 Sep 27, 2023
0925a4b
Merge pull request #12 from davidyuan1223/revert-11-fix_4186
davidyuan1223 Sep 27, 2023
2beccb6
Merge branch 'apache:master' into master
davidyuan1223 Sep 28, 2023
c8eb9a2
Merge branch 'apache:master' into master
davidyuan1223 Oct 10, 2023
56b91a3
fix_4186
davidyuan1223 Oct 10, 2023
57ec746
Merge pull request #13 from davidyuan1223/fix
davidyuan1223 Oct 10, 2023
72e7aea
Merge branch 'apache:master' into master
davidyuan1223 Oct 20, 2023
bcb0cf3
Merge remote-tracking branch 'origin/master'
davidyuan1223 Oct 26, 2023
8b51840
add common method to get session level config
davidyuan1223 Oct 26, 2023
6376466
Merge branch 'apache:master' into master
davidyuan1223 Dec 23, 2023
46001d4
Merge branch 'apache:master' into master
davidyuan1223 Jan 26, 2024
3d6c53b
add new module common-grpc
davidyuan1223 May 23, 2024
e16fd7f
fix_4186
davidyuan1223 Oct 10, 2023
a3a4ebd
add common method to get session level config
davidyuan1223 Oct 26, 2023
631ac9a
add new module common-grpc
davidyuan1223 May 23, 2024
0a52c2a
fork master new version
davidyuan1223 May 23, 2024
d92fdb6
Merge remote-tracking branch 'origin/grpc-common-support' into grpc-c…
davidyuan1223 May 23, 2024
4c5a4ae
fix format error
davidyuan1223 May 23, 2024
3e302aa
fix format error
davidyuan1223 May 23, 2024
c6af8ed
update settings.md
davidyuan1223 May 23, 2024
a1e0447
update
davidyuan1223 May 28, 2024
5bf1814
update
davidyuan1223 Jun 4, 2024
e031c8c
update
davidyuan1223 Jun 4, 2024
55cbae3
update
davidyuan1223 Jul 10, 2024
0c17121
update
davidyuan1223 Jul 15, 2024
2b959f9
add new module kyuubi-grpc
davidyuan1223 Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 8 additions & 21 deletions .idea/vcs.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

207 changes: 105 additions & 102 deletions docs/configuration/settings.md
View file
Open in desktop

Large diffs are not rendered by default.

37 changes: 37 additions & 0 deletions kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,6 +1064,21 @@ object KyuubiConf {
.transformToLowerCase
.createWithDefault(SaslQOP.AUTH.toString)

val GRPC_FRONTEND_BIND_HOST: OptionalConfigEntry[String] =
buildConf("kyuubi.grpc.frontend.bind.host")
.doc("Hostname or IP of the machine on which to run the grpc frontend services.")
.version("1.0.0")
.serverOnly
.stringConf
.createOptional

val GRPC_FRONTEND_SPARK_CONNECT_HOST: ConfigEntry[Option[String]] =
buildConf("kyuubi.grpc.frontend.spark.connect.host")
.doc("Hostname or IP of the machine on which to run the Spark Connect Grpc frontend service.")
.version("1.4.0")
.serverOnly
.fallbackConf(FRONTEND_BIND_HOST)

val FRONTEND_REST_BIND_HOST: ConfigEntry[Option[String]] =
buildConf("kyuubi.frontend.rest.bind.host")
.doc("Hostname or IP of the machine on which to run the REST frontend service.")
Expand Down Expand Up @@ -2731,6 +2746,28 @@ object KyuubiConf {
.stringConf
.createWithDefault("ENGINE")

val ENGINE_SPARK_CONNECT_GRPC_BINDING_PORT: ConfigEntry[Int] =
buildConf("kyuubi.engine.spark.connect.grpc.bind.port")
.doc("The port is used in spark connect frontendService start GrpcServer")
.version("1.9.0")
.intConf
.createWithDefault(15002)

val ENGINE_SPARK_CONNECT_GRPC_MAX_INBOUND_MESSAGE_SIZE: ConfigEntry[Int] =
buildConf("kyuubi.engine.spark.connect.grpc.max.inbound.message.size")
.doc("Sets the maximum inbound message in bytes size for the gRPC requests." +
"Requests with a larger payload will fail.")
.version("1.9.0")
.intConf
.createWithDefault(128 * 1024 * 1024)

val ENGINE_SPARK_CONNECT_GRPC_BINDING_HOST: ConfigEntry[Option[String]] =
buildConf("kyuubi.engine.spark.connect.grpc.bind.host")
.doc("Hostname or IP of the machine on which to run the grpc server in frontend service ")
.version("1.9.0")
.serverOnly
.fallbackConf(FRONTEND_BIND_HOST)

val ENGINE_SPARK_SHOW_PROGRESS: ConfigEntry[Boolean] =
buildConf("kyuubi.session.engine.spark.showProgress")
.doc("When true, show the progress bar in the Spark's engine log.")
Expand Down
243 changes: 243 additions & 0 deletions kyuubi-grpc-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,243 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-parent</artifactId>
<version>1.10.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>kyuubi-grpc-server_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<name>Kyuubi Project Grpc Server</name>
<url>https://kyuubi.apache.org/</url>

<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-connect -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect_2.13</artifactId>
<version>3.5.1</version>
<exclusions>
<exclusion>
<groupId>com.google.common</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>

<!-- https://mvnrepository.com/artifact/com.google.guava/failureaccess -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>failureaccess</artifactId>
<version>1.0.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-network-common -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-common_2.13</artifactId>
<version>3.5.1</version>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-grpc_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
</dependency>

<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-core</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
</dependency>

<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-grpc-shade</artifactId>
<version>${project.version}</version>
</dependency>

</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-patch-plugin</artifactId>
<version>1.2</version>
<configuration>
<patches>
<patch>mylittle.patch</patch>
</patches>
</configuration>
<executions>
<execution>
<id>patch</id>
<phase>process-sources</phase>
<goals>
<goal>apply</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Include a source dir depending on the Scala version -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-scala-sources</id>
<goals>
<goal>add-source</goal>
</goals>
<phase>generate-sources</phase>
<configuration>
<sources>
<source>src/main/scala-${scala.binary.version}</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-scala-test-sources</id>
<goals>
<goal>add-test-source</goal>
</goals>
<phase>generate-test-sources</phase>
<configuration>
<sources>
<source>src/test/scala-${scala.binary.version}</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>prepare-test-jar</id>
<goals>
<goal>test-jar</goal>
</goals>
<phase>test-compile</phase>
</execution>
</executions>
</plugin>
</plugins>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.6.2</version>
</extension>
</extensions>
</build>
</project>
29 changes: 29 additions & 0 deletions kyuubi-grpc-server/src/main/resources/log4j2.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
Loading