Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scorecard e2e Testing #3507

Merged
merged 12 commits into from
Jul 31, 2020
Merged

Scorecard e2e Testing #3507

Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
08fa066
add basic cli call
Jul 17, 2020
bba255b
parse json output
Jul 23, 2020
c37ffab
check testlist to have 1 result
Jul 23, 2020
a5255db
verify test passing
Jul 23, 2020
720592e
olm scorecard added
Jul 23, 2020
64bd778
olm scorecard flag
Jul 23, 2020
a7d1cac
run tests before cleanup
Jul 24, 2020
1d21262
fix olm test
Jul 24, 2020
f9f1aed
olm expected result table
Jul 24, 2020
a29956e
init map
Jul 24, 2020
30350ce
linter fix
Jul 29, 2020
940dc26
rm skip cleanup, add const test names
Jul 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions test/e2e/e2e_suite.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ package e2e

import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"os"
Expand All @@ -31,6 +32,7 @@ import (
. "github.com/onsi/gomega" //nolint:golint
kbtestutils "sigs.k8s.io/kubebuilder/test/e2e/utils"

"github.com/operator-framework/operator-sdk/pkg/apis/scorecard/v1alpha3"
testutils "github.com/operator-framework/operator-sdk/test/internal"
)

Expand Down Expand Up @@ -204,6 +206,42 @@ var _ = Describe("operator-sdk", func() {
_, err = tc.Run(runPkgManCmd)
Expect(err).NotTo(HaveOccurred())

By("running basic scorecard tests")
camilamacedo86 marked this conversation as resolved.
Show resolved Hide resolved
var scorecardOutput v1alpha3.TestList
runScorecardCmd := exec.Command(tc.BinaryName, "scorecard", "bundle",
"--selector=suite=basic",
"--output=json",
"--skip-cleanup=true",
"--wait-time=40s")
scorecardOutputBytes, err := tc.Run(runScorecardCmd)
Expect(err).NotTo(HaveOccurred())
err = json.Unmarshal(scorecardOutputBytes, &scorecardOutput)
camilamacedo86 marked this conversation as resolved.
Show resolved Hide resolved
Expect(err).NotTo(HaveOccurred())
Expect(len(scorecardOutput.Items)).To(Equal(1))
Expect(scorecardOutput.Items[0].Status.Results[0].State).To(Equal(v1alpha3.PassState))

By("running olm scorecard tests")
runOLMScorecardCmd := exec.Command(tc.BinaryName, "scorecard", "bundle",
"--selector=suite=olm",
"--output=json",
"--skip-cleanup=true",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason we're skipping cleanup?

"--wait-time=40s")
scorecardOutputBytes, err = tc.Run(runOLMScorecardCmd)
Expect(err).To(HaveOccurred())
err = json.Unmarshal(scorecardOutputBytes, &scorecardOutput)
Expect(err).NotTo(HaveOccurred())

resultTable := make(map[string]v1alpha3.State)
resultTable["olm-status-descriptors"] = v1alpha3.FailState
resultTable["olm-crds-have-resources"] = v1alpha3.FailState
resultTable["olm-bundle-validation"] = v1alpha3.PassState
resultTable["olm-spec-descriptors"] = v1alpha3.FailState
resultTable["olm-crds-have-validation"] = v1alpha3.PassState
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use these constants.


for a := 0; a < len(scorecardOutput.Items); a++ {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use test names to index into the resultTable map to make sure each test is run.

Expect(scorecardOutput.Items[a].Status.Results[0].State).To(Equal(resultTable[scorecardOutput.Items[a].Status.Results[0].Name]))
}

By("destroying the deployed package manifests-formatted operator")
cleanupPkgManCmd := exec.Command(tc.BinaryName, "cleanup", "packagemanifests",
"--operator-namespace", tc.Kubectl.Namespace,
Expand Down