Skip to content

Commit

Permalink
[KYUUBI #1947] Revamp Github Action settings for CI
Browse files Browse the repository at this point in the history
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. Add missing asf header
2. Disable fail-fast to let jobs run individually
3. Group defaults maven builds with java & spark version matrix
    - so this PR brings fully tests with JDK 11 back
4. Group others(now cross-version check only) in `include`
5. replace `actions/cachev2` with `actions/setup-javav2` - `cache: 'maven'`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Need to pass all CI jobs

Closes #1947 from yaooqinn/ga.

Closes #1947

7d5d247 [Kent Yao] address comment
d8d347b [Kent Yao] address comment
8c3cba9 [Kent Yao] address comment
7400cdc [Kent Yao] checkout
20eeb7b [Kent Yao] test
b1c70e4 [Kent Yao] test
4ca4f23 [Kent Yao] test
b588cd3 [Kent Yao] test
da4f6ae [Kent Yao] test
057790d [Kent Yao] test
01b07b2 [Kent Yao] test
2906ec6 [Kent Yao] test
4d97bf1 [Kent Yao] test
cf16f16 [Kent Yao] test
8c12b8e [Kent Yao] test
3b84e45 [Kent Yao] test
6e21d97 [Kent Yao] test
dc2d900 [Kent Yao] test
1cff392 [Kent Yao] test
a4b4747 [Kent Yao] test
c471802 [Kent Yao] test
d264d33 [Kent Yao] test
ee293c1 [Kent Yao] test
903cb0b [Kent Yao] test
f74fa1a [Kent Yao] test

Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
  • Loading branch information
yaooqinn committed Feb 22, 2022
1 parent 74abe85 commit 5836db6
Show file tree
Hide file tree
Showing 7 changed files with 131 additions and 88 deletions.
19 changes: 18 additions & 1 deletion .github/workflows/docker-image.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Publish Docker image

on:
Expand All @@ -16,7 +33,7 @@ jobs:
cancel-in-progress: true
steps:
- name: Checkout
uses: actions/checkout@v2.3.5
uses: actions/checkout@v2
- name: Login to Docker Hub
uses: docker/login-action@v1
with:
Expand Down
17 changes: 17 additions & 0 deletions .github/workflows/greetings.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Greetings

# on: [pull_request, issues]
Expand Down
7 changes: 3 additions & 4 deletions .github/workflows/license.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
name: License check

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
Expand All @@ -17,8 +15,9 @@ name: License check
# limitations under the License.
#

# This GitHub workflow checks Apache License v2.0 Header is added if needed
name: License check

# This GitHub workflow checks Apache License v2.0 Header is added if needed

on:
pull_request:
Expand All @@ -31,7 +30,7 @@ jobs:
name: License
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2.3.5
- uses: actions/checkout@v2
- name: Setup JDK 8
uses: actions/setup-java@v2
with:
Expand Down
131 changes: 54 additions & 77 deletions .github/workflows/master.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,21 @@
name: Kyuubi
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: CI

on:
push:
Expand All @@ -13,76 +30,58 @@ on:
jobs:

build:
name: Build
name: Test
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
java:
- 8
- 11
spark:
- '3.0'
- '3.1'
- '3.2'
spark-archive: [""]
exclude-tags: [""]
include:
- java: 8
spark: '3.0'
profiles: -Pspark-hadoop-2.7
- java: 8
spark: '3.1'
- java: 8
spark: '3.2'
profiles: >-
-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3
-Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz
-Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest
- java: 8
spark: '3.2'
profiles: >-
-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.2
-Dspark.archive.name=spark-3.1.2-bin-hadoop3.2.tgz
-Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest
spark-archive: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz'
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.2'
codecov: 'true'
- java: 11
spark: '3.2'
profiles: -DskipTests -Pflink-provided,spark-provided

spark-archive: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.2 -Dspark.archive.name=spark-3.1.2-bin-hadoop3.2.tgz'
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2.3.5
- uses: actions/checkout@v2
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
uses: actions/setup-java@v2
with:
distribution: zulu
java-version: ${{ matrix.java }}
- uses: actions/cache@v2
with:
path: ~/.m2/repository/com
key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-com-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/org
key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-org-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/net
key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-net-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/io
key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-io-
- name: Build with Maven
cache: 'maven'
check-latest: false
- name: Build and test with maven w/o linters
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -Drat.skip=true -Dscalastyle.skip=true -Dspotless.check.skip -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -V
-Pspark-${{ matrix.spark }} ${{ matrix.profiles }}
./build/mvn clean install
-Dmaven.javadoc.skip=true
-Drat.skip=true
-Dscalastyle.skip=true
-Dspotless.check.skip
-Dorg.slf4j.simpleLogger.defaultLogLevel=warn
-Pspark-${{ matrix.spark }}
${{ matrix.spark-archive }}
${{ matrix.exclude-tags }}
- name: Code coverage
if: ${{ matrix.codecov == 'true' }}
if: |
matrix.java == 8 &&
matrix.spark == '3.2' &&
matrix.spark-archive == ''
uses: codecov/codecov-action@v2
with:
verbose: true
Expand All @@ -103,38 +102,16 @@ jobs:
env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2.3.5
- uses: actions/checkout@v2
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK 8
uses: actions/setup-java@v2
with:
distribution: zulu
java-version: 8
- uses: actions/cache@v2
with:
path: ~/.m2/repository/com
key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-com-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/org
key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-org-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/net
key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-net-
- uses: actions/cache@v2
with:
path: ~/.m2/repository/io
key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-io-
cache: 'maven'
check-latest: false
- name: Run TPC-DS Tests
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -Drat.skip=true -Dscalastyle.skip=true -Dspotless.check.skip -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -V
Expand All @@ -148,7 +125,7 @@ jobs:
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2.3.5
uses: actions/checkout@v2
# https://github.com/docker/build-push-action
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
Expand Down
19 changes: 18 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Kyuubi Nightly

on:
Expand All @@ -16,7 +33,7 @@ jobs:
env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2.3.5
- uses: actions/checkout@v2
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK 8
Expand Down
19 changes: 18 additions & 1 deletion .github/workflows/publish-snapshot.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: Publish Snapshot

on:
Expand All @@ -16,7 +33,7 @@ jobs:
- branch-1.3
steps:
- name: Checkout repository
uses: actions/checkout@v2.3.5
uses: actions/checkout@v2
with:
ref: ${{ matrix.branch }}
- name: Cache Maven local repository
Expand Down
7 changes: 3 additions & 4 deletions .github/workflows/style.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
name: Style check

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
Expand All @@ -17,8 +15,9 @@ name: Style check
# limitations under the License.
#

# This GitHub workflow checks style & dependency issues.
name: Style check

# This GitHub workflow checks style & dependency issues.

on:
pull_request:
Expand All @@ -35,7 +34,7 @@ jobs:
profiles:
- '-Ptpcds -Pspark-block-cleaner -Pkubernetes-deployment-it -Pspark-3.1 -Pspark-3.2'
steps:
- uses: actions/checkout@v2.3.5
- uses: actions/checkout@v2
- name: Setup JDK 8
uses: actions/setup-java@v2
with:
Expand Down

0 comments on commit 5836db6

Please sign in to comment.