Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci-operator/step-registry/gather/aws-console: Use LC_ALL instead of PYTHONIOENCODING #19836

Merged
merged 3 commits into from
Jun 30, 2021
Merged

ci-operator/step-registry/gather/aws-console: Use LC_ALL instead of PYTHONIOENCODING #19836

merged 3 commits into from
Jun 30, 2021

Conversation

wking
Copy link
Member

@wking wking commented Jun 30, 2021 •
edited
Loading

For some reason I don't understand, 24b834c (#19827) is not doing well in the wild (although I have been unable to reproduce locally). For example:

INFO[2021-06-29T20:14:28Z] Gathering console logs for i-008f73fde1ef3f0dc

'ascii' codec can't encode character '\u2026' in position 13999: ordinal not in range(128)
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-06-29T20:14:27Z"}
error: failed to execute wrapped command: exit status 1

I'd rather not revert, because I want these console logs. Instead, I'm moving to JSON. It will be harder to read, but might have the non-ASCII output escaped. Edit: now I'm trying LC_ALL.

Wait for rehearsals this time:

/hold

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2021
@openshift-ci openshift-ci bot requested review from enxebre and vrutkovs June 30, 2021 01:56
@wking
Copy link
Member Author

wking commented Jun 30, 2021

from 4.9-e2e-aws-disruptive:

aws-cli/2.2.5 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.7 prompt/off
Gathering console logs for 
/bin/bash: line 41: /logs/artifacts/: Is a directory
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-06-30T03:26:32Z"}
error: failed to execute wrapped command: exit status 1

I've added b2fecba6ac, following GCP's #14467, to exclude empty-string "instance IDs".

@wking
Copy link
Member Author

wking commented Jun 30, 2021

/hold

For some reason my earlier request doesn't seem to have set the label.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 30, 2021
wking added 3 commits June 29, 2021 21:32
@wking
ci-operator/step-registry/gather/aws-console: Add 'aws --version'
22bc15b 
For debugging, so we don't have to guess the AWS CLI version when
trying to decide why it's breaking.
@wking
ci-operator/step-registry/gather/aws-console: Exclude empty-string "i…
0a0e1ab 
…nstance IDs"

Like we did for GCP in 7e1a515 (ci-operator/step-registry/gather:
Exclude empty-string "instance IDs", 2020-12-19, openshift#14467).  In this
case, the empty-string example was [1]:

  Gathering console logs for
  /bin/bash: line 41: /logs/artifacts/: Is a directory

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/19836/rehearse-19836-pull-ci-openshift-origin-release-4.9-e2e-aws-disruptive
@wking
ci-operator/step-registry/gather/aws-console: Use LC_ALL instead of P…
1b472c0 
…YTHONIOENCODING

24b834c (ci-operator/step-registry/gather/aws-console: UTF-8 for
output, 2021-06-29, openshift#19827) is not doing well in the wild.  For
example, [1]:

  INFO[2021-06-29T20:14:28Z] Gathering console logs for i-008f73fde1ef3f0dc

  'ascii' codec can't encode character '\u2026' in position 13999: ordinal not in range(128)
  {"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-06-29T20:14:27Z"}
  error: failed to execute wrapped command: exit status 1

Apparently the AWS CLI is not using Python's default stream encoding
here.  But from [2,3,4] there is the suggestion that
LC_ALL=en_US.UTF-8 will help.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-e2e-aws/1409949910362492928
[2]: https://awscli.amazonaws.com/v2/documentation/api/latest/topic/config-vars.html#locale
[3]: https://github.com/aws/aws-cli/blob/2.2.5/awscli/compat.py#L121-L146
[4]: https://docs.python.org/3/library/locale.html#locale.getpreferredencoding
@wking wking force-pushed the gather-aws-console-json branch from b2fecba to 1b472c0 Compare June 30, 2021 04:45
@wking wking changed the title ci-operator/step-registry/gather/aws-console: Temporarily encode to JSON ci-operator/step-registry/gather/aws-console: Use LC_ALL instead of PYTHONIOENCODING Jun 30, 2021
@wking
Copy link
Member Author

wking commented Jun 30, 2021

This:

INFO[2021-06-30T02:18:40Z] Resolved source https://github.com/openshift/release to master@a831e974, merging: #19836 25d3409a @wking
...
INFO[2021-06-30T03:27:13Z] aws-cli/2.2.5 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.7 prompt/off
Gathering console logs for i-00a3250b51e067bea
'ascii' codec can't encode character '\u2026' in position 6669: ordinal not in range(128)

suggests encode-to-JSON is not going to work. I've pushed b2fecba6ac -> 1b472c0, pivoting from JSON to using LC_ALL.

@wking
Copy link
Member Author

wking commented Jun 30, 2021

This looks good:

$ wget https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_release/19836/rehearse-19836-pull-ci-openshift-cluster-logging-operator-tech-preview-e2e-operator/1410097682520739840/artifacts/e2e-operator/gather-aws-console/artifacts/i-000bbb4e6d1919ca7
$ file i-000bbb4e6d1919ca7
i-000bbb4e6d1919ca7: UTF-8 Unicode text, with very long lines, with CRLF, CR, LF line terminators, with escape sequences

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 30, 2021

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/rehearse/openshift/installer/release-4.9/e2e-aws-shared-vpc 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/origin/release-4.1/e2e-aws-builds 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/origin/release-4.1/e2e-aws-image-ecosystem 1b472c0 link /test pj-rehearse
ci/rehearse/redhat-developer/jenkins-operator/main/e2e 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/cluster-logging-operator/tech-preview/e2e-operator 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.9/e2e-aws-ccm-install 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/origin/release-4.9/e2e-aws-image-registry 1b472c0 link /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-ci-4.9-e2e-aws-upgrade-single-node 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/origin/release-4.2/e2e-cmd 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/release-4.9/e2e-aws-manual-oidc 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.9/e2e-aws-ccm 1b472c0 link /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-stable-4.8-upgrade-from-stable-4.6-e2e-aws-upgrade-paused 1b472c0 link /test pj-rehearse
ci/rehearse/openshift/origin/release-4.9/e2e-aws-disruptive 1b472c0 link /test pj-rehearse

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vrutkovs
vrutkovs approved these changes Jun 30, 2021
View reviewed changes
Copy link
Member

@vrutkovs vrutkovs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 30, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2021
@openshift-merge-robot openshift-merge-robot merged commit 55ff4b2 into openshift:master Jun 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 30, 2021

@wking: Updated the step-registry configmap in namespace ci at cluster app.ci using the following files:

  • key gather-aws-console-commands.sh using file ci-operator/step-registry/gather/aws-console/gather-aws-console-commands.sh

In response to this:

For some reason I don't understand, 24b834c (#19827) is not doing well in the wild (although I have been unable to reproduce locally). For example:

INFO[2021-06-29T20:14:28Z] Gathering console logs for i-008f73fde1ef3f0dc

'ascii' codec can't encode character '\u2026' in position 13999: ordinal not in range(128)
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-06-29T20:14:27Z"}
error: failed to execute wrapped command: exit status 1

I'd rather not revert, because I want these console logs. Instead, I'm moving to JSON. It will be harder to read, but might have the non-ASCII output escaped. Edit: now I'm trying LC_ALL.

Wait for rehearsals this time:

/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@vrutkovs vrutkovs mentioned this pull request Jun 30, 2021
Merged
@wking wking deleted the gather-aws-console-json branch June 30, 2021 13:27
@sdodson sdodson mentioned this pull request Jun 30, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vrutkovs vrutkovs vrutkovs approved these changes

@enxebre enxebre Awaiting requested review from enxebre

Assignees

@vrutkovs vrutkovs

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wking @vrutkovs @openshift-merge-robot