Skip to content

Commit

Permalink
Reduce unnecessary maven profile
Browse files Browse the repository at this point in the history
  • Loading branch information
pan3793 committed Oct 20, 2021
1 parent ced8d98 commit 8d40079
Show file tree
Hide file tree
Showing 7 changed files with 20 additions and 11 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@ jobs:
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-1'
profiles: '-Pspark-3.1'
codecov: 'true'
- java: 8
spark: '3.2'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark-3-2'
profiles: '-Pspark-3.2'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
matrix:
profiles:
- '-Pspark-master -pl :kyuubi-spark-sql-engine_2.12,:kyuubi-common_2.12,:kyuubi-ha_2.12,:kyuubi-zookeeper_2.12,:kyuubi-spark-monitor_2.12'
- '-Pspark-master -pl externals/kyuubi-spark-sql-engine,kyuubi-common,kyuubi-ha,kyuubi-zookeeper'
env:
SPARK_LOCAL_IP: localhost
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-snapshot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -DskipTests -Pkyuubi-extension-spark-3-1,spark-provided -s ./build/release/asf-settings.xml
./build/mvn clean deploy -DskipTests -Pspark-3.1,spark-provided -s ./build/release/asf-settings.xml
2 changes: 1 addition & 1 deletion build/release/create-package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ package_binary() {

echo "Creating binary release tarball ${BIN_TGZ_FILE}"

${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1
${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pspark-3.1

cp "${BIN_TGZ_FILE}" "${RELEASE_DIR}"

Expand Down
2 changes: 1 addition & 1 deletion docs/community/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ export RELEASE_RC_NO=<RC number, e.g. 0>
```shell
build/mvn versions:set -DgenerateBackupPoms=false \
-DnewVersion="${RELEASE_VERSION}" \
-Pkubernetes,kyuubi-extension-spark-3-1,spark-block-cleaner,tpcds
-Pspark-3.1,spark-block-cleaner

git commit -am "[RELEASE] Bump ${RELEASE_VERSION}"
```
Expand Down
11 changes: 7 additions & 4 deletions docs/sql/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,13 @@ And don't worry, Kyuubi will support the new Apache Spark version in future. Tha

## How to use Kyuubi SQL extension

1. you need to choose Apache Spark branch-3.1 or higher version with Kyuubi binary tgz.
2. if you want to compile Kyuubi by yourself, the maven opt should add `-Pkyuubi-extension-spark-3-1`
3. move the jar(`kyuubi-extension-spark-*.jar`) which is in `$KYUUBI_HOME/extension` into `$SPARK_HOME/jars`
4. add a config into `spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
1. Since v1.3.0-incubating, Kyuubi provides `kyuubi-extension-spark-3-1-*.jar` for Apache Spark 3.1.x
2. Since v1.4.0-incubating, Kyuubi provides `kyuubi-extension-spark-3-2-*.jar` for Apache Spark 3.2.x
3. You need to choose the supported Spark version with cooperated Kyuubi Spark SQL Extension Jar
5. The Kyuubi binary release tarball only contains one Kyuubi Spark SQL Extension jar which is Kyuubi the current default build Spark version, you can find all supported versions jars in Maven Central, or build by yourself
4. If you want to compile Kyuubi Spark SQL Extension by yourself, the supported Spark version profile will activate the cooperated Kyuubi Spark SQL Extension, i.e. you can get `kyuubi-extension-spark-3-1` when `-Pspark-3.1` is activated
6. Put the jar(`kyuubi-extension-spark-*.jar`) into `$SPARK_HOME/jars`
7. Add a config into `$SPARK_HOME/conf/spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`

Now, you can enjoy the Kyuubi SQL Extension, and also Kyuubi provides some configs to make these feature easy to use.

Expand Down
8 changes: 7 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1834,11 +1834,17 @@
</properties>
</profile>

<profile>
<id>spark-block-cleaner</id>
<modules>
<module>tools/spark-block-cleaner</module>
</modules>
</profile>

<profile>
<id>kubernetes</id>
<modules>
<module>kubernetes/integration-tests</module>
<module>tools/spark-block-cleaner</module>
</modules>
</profile>

Expand Down

0 comments on commit 8d40079

Please sign in to comment.