RUM-4079 chore: Migrate tools and smoke tests to GitLab #1921

ncreated · 2024-06-24T09:39:08Z

What and why?

📦 🧰 Following #1910, this PR migrates next portion of CI setup to GitLab:

✅ Smoke tests:
- whole SDK integration (iOS + tvOS) with Cocoapods, Carthage, XCFrameworks and SPM
- SPM build for iOS, tvOS, macOS, macOS through Mac Catalyst and visionOS
✅ Tooling tests
- unit tests for http-server-mock, rum-models-generator, sr-snapshots and distribution package (python)

The following integrations are still running on Bitrise:

❌ SR snapshot tests - these will be moved to GitLab after upgrading our macOS runner to Xcode 15.4 and installing the iOS 17.5 runtime.
❌ Dogfooding automation
❌ Release automation
❌ E2E tests app upload to s8s

I will address the remaining items in subsequent PR(s).

How?

General remarks:

The new smoke-test stage uses macos:ventura runner (lower spec). This is to not consume parallelisation budget of macos:sonoma which can run test and test-ui for other branches while smoke tests are still performed on the current branch.
Xcode 15 is not activated by default in macos:ventura, so I added an option to runner-setup.sh to activate it. Ultimately, this will be done in the AMI (runner image).
I renamed dependency-manager-tests to SmokeTests for clarity and to better fit our conventions.
All smoke tests are ran via distinct Makefiles stored at SmokeTests/<dep manager>/Makefile. This is not different from what we had before. To better streamline these tests, each smoke Makefile now distinguishes make clean install test commands.
I also moved instrumented-tests/http-server-mock to tools/http-server-mock to better fit repo layout.

Makefile

Same as in #1910, smoke and tool tests are ran from the main Makefile. List of new key commands:

Tool Tests:
- make tools-test - Runs tests for all repo tools.
Smoke Tests:
- make smoke-test - Runs smoke tests in specified TEST_DIRECTORY (using OS, PLATFORM and DEVICE)
- make smoke-test-ios-all - Runs all smoke tests adequate for iOS.
- make smoke-test-tvos-all - Runs all smoke tests adequate for tvOS.
- make spm-build - Builds Package.swift for specified SCHEME and DESTINATION.
- make spm-build-ios - Builds Package.swift for iOS.
- make spm-build-tvos - Builds Package.swift for tvOS.
- make spm-build-visionos - Builds Package.swift for visionOS.
- make spm-build-macos - Builds Package.swift for macOS and Mac Catalyst.

Ultimately, the whole workflow for integrating branches will look this:

Review checklist

Feature or bugfix MUST have appropriate tests (unit, integration)
Make sure each commit and the PR mention the Issue number or JIRA reference
Add CHANGELOG entry for user facing changes

Custom CI job configuration (optional)

Run unit tests for Session Replay

as we now test macOS support through bare spm build

now moved to GitLab

…ests')

ncreated · 2024-06-24T09:41:53Z

Makefile

-
-# Helpers
-
-ECHO_TITLE=./tools/utils/echo_color.sh --title
-ECHO_ERROR=./tools/utils/echo_color.sh --err
-ECHO_WARNING=./tools/utils/echo_color.sh --warn
-ECHO_SUCCESS=./tools/utils/echo_color.sh --succ
-
-define require_param
-    if [ -z "$${$(1)}" ]; then \
-        $(ECHO_ERROR) "Error:" "$(1) parameter is required but not provided."; \
-        exit 1; \
-    fi
-endef


💡 Moved to tools/utils/common.mk as now being shared with Makefiles used for smoke testing.

rtrieu

Looks good!

ganeshnj

Looks great, a couple of questions nothing blocking.

ganeshnj · 2024-06-25T08:11:40Z

tools/utils/current_git.sh

+    fi
+}
+
+# Prints current git branch


/question

what happens when HEAD doesn't point to a branch?

Like in detached HEAD configuration? It will print just "HEAD" in such case. Although, this code isn't new - we've been using it for years in existing smoke tests. I don't think GitLab setup changes anything significant to break it 🤞.

ganeshnj · 2024-06-25T08:13:29Z

tools/utils/current_git.sh

+
+# Prints current git tag (if any)
+function current_git_tag() {
+    if [[ -n "$CI_COMMIT_TAG" ]]; then


/question

are these env variable populated via gitlab?

Exactly 👍

ganeshnj · 2024-06-25T08:15:42Z

tools/runner-setup.sh

+        echo_succ "Already using Xcode version '$version'."
+    elif [[ -d "$XCODE_PATH" ]]; then
+        echo "Found Xcode at '$XCODE_PATH'."
+        if sudo xcode-select -s "$XCODE_PATH"; then


/question

Is the runner user already in elevated/sudo mode?

Yes it is 👌 - tested, it works 👍

ganeshnj · 2024-06-25T08:32:44Z

.github/PULL_REQUEST_TEMPLATE.md

@@ -13,5 +13,3 @@ A brief description of implementation details of this PR.

 ### Custom CI job configuration (optional)
 - [ ] Run unit tests for Session Replay
- [ ] Run smoke tests


ganeshnj · 2024-06-25T08:38:58Z

SmokeTests/spm/Makefile

+REPO_ROOT := ../../
+include ../../tools/utils/common.mk
+
+ifeq ($(PLATFORM), iOS Simulator)


/comment

IIRC, we updated SPM tests with macOS support, it not then we should include macOS and visionOS.

If you can do along with this PR :great else we can do as separate PT too.

It is already there, in this PR 👍. We distinguish:

smoke tests - integrate SDK and run basic UI test (launch the app → check no crash)

spm build tests - build SDK for all supported platforms

The SmokeTest/spm/Makefile in question performs smoke testing. It integrates SDK into SPMProject via Xcode's SPM (using current branch), then launches it to assert there is no runtime crash. We do it for iOS and tvOS.

The spm.sh added earlier is renamed to spm-build.sh and ran from .gitlab-ci.yml:

make spm-build-ios make spm-build-tvos make spm-build-visionos make spm-build-macos

It validates that Package.swift builds for all these platforms. Both iOS and tvOS builds might be considered redundant considering SmokeTest/spm/Makefile, but I wanted to avoid changing existing concepts.

ncreated added 11 commits June 24, 2024 10:52

RUM-4079 Migrate tools tests to GitLab

76b60b2

RUM-4079 Move http-server-mock to tools/

a79c8de

RUM-4079 Prepare macos:ventura runner for running smoke tests

955b435

RUM-4079 Run smoke tests for SPM and Carthage

ddc00ea

RUM-4079 Run smoke tests for Cocoapods

3bc3a7b

RUM-4079 Run smoke tests for XCFrameworks and SPM build

2e0d945

RUM-4079 Simplify building SPM package schemes

3817b79

RUM-4079 Cleanup: delete macOS app scheme from SPMProject

2eb8eb6

as we now test macOS support through bare spm build

RUM-4079 Cleanup: remove remaining tasks from bitrise.yml

fc8d367

now moved to GitLab

RUM-4079 Update smoke tests path ('dependency-manager-test' → 'SmokeT…

a8297aa

…ests')

RUM-4079 Cleanup

facf8f9

ncreated self-assigned this Jun 24, 2024

ncreated commented Jun 24, 2024

View reviewed changes

RUM-4079 Update PR template

634f017

ncreated marked this pull request as ready for review June 24, 2024 15:12

ncreated requested review from a team as code owners June 24, 2024 15:12

RUM-4079 Fix release automation for Bitrise

ef7ead5

rtrieu approved these changes Jun 24, 2024

View reviewed changes

ganeshnj approved these changes Jun 25, 2024

View reviewed changes

maxep approved these changes Jun 25, 2024

View reviewed changes

ncreated merged commit 85dcd0f into develop Jun 25, 2024
15 checks passed

ncreated deleted the ncreated/RUM-4079/migrate-smoke-and-tool-tests branch June 25, 2024 13:33

This was referenced Jul 5, 2024

RUM-4079 chore: Migrate release automation to GitLab #1945

Merged

RUM-4079 chore: Migrate SR Snapshot Tests automation to GitLab #1953

Merged

RUM-4079 chore: Migrate E2E Tests app workflow to GitLab #1957

Merged

ncreated mentioned this pull request Jul 18, 2024

RUM-4079 chore: Migrate dogfooding automation to GitLab #1961

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RUM-4079 chore: Migrate tools and smoke tests to GitLab #1921

RUM-4079 chore: Migrate tools and smoke tests to GitLab #1921

ncreated commented Jun 24, 2024 •

edited

Loading

ncreated Jun 24, 2024

rtrieu left a comment

ganeshnj left a comment

ganeshnj Jun 25, 2024

ncreated Jun 25, 2024

ganeshnj Jun 25, 2024

ncreated Jun 25, 2024

ganeshnj Jun 25, 2024

ncreated Jun 25, 2024

ganeshnj Jun 25, 2024

ganeshnj Jun 25, 2024

ncreated Jun 25, 2024

RUM-4079 chore: Migrate tools and smoke tests to GitLab #1921

RUM-4079 chore: Migrate tools and smoke tests to GitLab #1921

Conversation

ncreated commented Jun 24, 2024 • edited Loading

What and why?

How?

Makefile

Review checklist

Custom CI job configuration (optional)

Choose a reason for hiding this comment

rtrieu left a comment

Choose a reason for hiding this comment

ganeshnj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncreated commented Jun 24, 2024 •

edited

Loading