Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUILD] Reduce unnecessary maven profile #1257

Closed
wants to merge 15 commits into from
Closed

[BUILD] Reduce unnecessary maven profile #1257

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ced8d98
Reduce unnecessary maven profile
pan3793 Oct 19, 2021
8d40079
Reduce unnecessary maven profile
pan3793 Oct 19, 2021
49a41e8
nit
pan3793 Oct 19, 2021
3055c2a
nit
pan3793 Oct 19, 2021
23b81dc
Address comments
Oct 20, 2021
ccff63c
update GA
pan3793 Oct 20, 2021
087d3fd
Add spark 3.2 binary test
pan3793 Oct 20, 2021
6615d7c
Exclude HudiTest in cross test
pan3793 Oct 20, 2021
98347c8
Update doc
pan3793 Oct 20, 2021
e85116c
nit
pan3793 Oct 20, 2021
bd96097
nit
pan3793 Oct 20, 2021
445666e
typo
pan3793 Oct 20, 2021
00a32b2
nit
pan3793 Oct 20, 2021
874f30c
Address comments
pan3793 Oct 22, 2021
1d871b6
nit
pan3793 Oct 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,10 @@ jobs:
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-1'
codecov: 'true'
- java: 8
spark: '3.2'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-2'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
matrix:
profiles:
- '-Pspark-master -pl :kyuubi-spark-sql-engine_2.12,:kyuubi-common_2.12,:kyuubi-ha_2.12,:kyuubi-zookeeper_2.12,:kyuubi-spark-monitor_2.12'
- '-Pspark-master -pl externals/kyuubi-spark-sql-engine -am'
env:
SPARK_LOCAL_IP: localhost
steps:
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/publish-snapshot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,6 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -DskipTests -Pkyuubi-extension-spark-3-1,spark-provided -s ./build/release/asf-settings.xml
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-3.1 -s ./build/release/asf-settings.xml -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-3.2 -s ./build/release/asf-settings.xml -pl dev/kyuubi-extension-spark-3-2 -am
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L39: build against the default spark version, publish no kyuubi spark extension modules
L40: only build kyuubi-extension-spark-3-1 and dependent modules against spark 3.1, publish kyuubi-extension-spark-3-1
L41: only build kyuubi-extension-spark-3-2 and dependent modules against spark 3.2, publish kyuubi-extension-spark-3-2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it allowed if we release the binary that contains Spark binary ? before we only release Kyuubi without Spark.

2 changes: 1 addition & 1 deletion build/release/create-package.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ package_binary() {

echo "Creating binary release tarball ${BIN_TGZ_FILE}"

${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1
${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pspark-3.1

cp "${BIN_TGZ_FILE}" "${RELEASE_DIR}"

Expand Down
6 changes: 6 additions & 0 deletions build/release/release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,12 @@ upload_svn_staging() {
upload_nexus_staging() {
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml"
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-3.1 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-1 -am
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-3.2 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-2 -am
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as GA

}

finalize_svn() {
Expand Down
9 changes: 0 additions & 9 deletions dev/kyuubi-extension-spark-3-1/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,13 +155,4 @@
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>kyuubi-extension-spark-3-1</id>
<properties>
<spark.version>3.1.2</spark.version>
</properties>
</profile>
</profiles>
</project>
10 changes: 0 additions & 10 deletions dev/kyuubi-extension-spark-3-2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,14 +160,4 @@
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>kyuubi-extension-spark-3-2</id>
<properties>
<spark.version>3.2.0</spark.version>
</properties>
</profile>
</profiles>

</project>
32 changes: 5 additions & 27 deletions dev/kyuubi-extension-spark-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,13 +135,11 @@
</plugin>

<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>

<plugin>
Expand All @@ -157,26 +155,6 @@
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

<profiles>
<profile>
<id>kyuubi-extension-spark-3-1</id>
<properties>
<spark.version>3.1.2</spark.version>
</properties>
</profile>
<profile>
<id>kyuubi-extension-spark-3-2</id>
<properties>
<spark.version>3.2.0</spark.version>
</properties>
</profile>
</profiles>
</project>
2 changes: 1 addition & 1 deletion docs/community/release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ export RELEASE_RC_NO=<RC number, e.g. 0>
```shell
build/mvn versions:set -DgenerateBackupPoms=false \
-DnewVersion="${RELEASE_VERSION}" \
-Pkubernetes,kyuubi-extension-spark-3-1,spark-block-cleaner,tpcds
-Pspark-3.1,spark-block-cleaner

git commit -am "[RELEASE] Bump ${RELEASE_VERSION}"
```
Expand Down
11 changes: 7 additions & 4 deletions docs/sql/rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,13 @@ And don't worry, Kyuubi will support the new Apache Spark version in future. Tha

## How to use Kyuubi SQL extension

1. you need to choose Apache Spark branch-3.1 or higher version with Kyuubi binary tgz.
2. if you want to compile Kyuubi by yourself, the maven opt should add `-Pkyuubi-extension-spark-3-1`
3. move the jar(`kyuubi-extension-spark-*.jar`) which is in `$KYUUBI_HOME/extension` into `$SPARK_HOME/jars`
4. add a config into `spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
1. Since v1.3.0-incubating, Kyuubi provides `kyuubi-extension-spark-3-1-*.jar` for Apache Spark 3.1.x
2. Since v1.4.0-incubating, Kyuubi provides `kyuubi-extension-spark-3-2-*.jar` for Apache Spark 3.2.x
3. You need to choose the supported Spark version with cooperated Kyuubi Spark SQL Extension Jar
5. The Kyuubi binary release tarball only contains one Kyuubi Spark SQL Extension jar which is Kyuubi the current default build Spark version, you can find all supported versions jars in [Maven Central](https://search.maven.org/search?q=kyuubi-extension-spark), or build by yourself
4. If you want to compile Kyuubi Spark SQL Extension by yourself, the supported Spark version profile will activate the cooperated Kyuubi Spark SQL Extension, i.e. you can get `kyuubi-extension-spark-3-1` when `-Pspark-3.1` is activated
6. Put the jar(`kyuubi-extension-spark-*.jar`) into `$SPARK_HOME/jars`
7. Add a config into `$SPARK_HOME/conf/spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`

Now, you can enjoy the Kyuubi SQL Extension, and also Kyuubi provides some configs to make these feature easy to use.

Expand Down
24 changes: 8 additions & 16 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1775,6 +1775,10 @@
<delta.version>1.0.0</delta.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-1</module>
</modules>
</profile>

<profile>
Expand All @@ -1783,6 +1787,10 @@
<spark.version>3.2.0</spark.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-2</module>
</modules>
</profile>

<profile>
Expand Down Expand Up @@ -1826,22 +1834,6 @@
</properties>
</profile>

<profile>
<id>kyuubi-extension-spark-3-1</id>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-1</module>
</modules>
</profile>

<profile>
<id>kyuubi-extension-spark-3-2</id>
<modules>
<module>dev/kyuubi-extension-spark-common</module>
<module>dev/kyuubi-extension-spark-3-2</module>
</modules>
</profile>

<profile>
<id>spark-block-cleaner</id>
<modules>
Expand Down
5 changes: 0 additions & 5 deletions tools/spark-block-cleaner/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,5 @@
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<resources>
<resource>
<directory>${project.basedir}/src/main/resources</directory>
</resource>
</resources>
</build>
</project>