ci-operator/step-registry/gather/aws-console: UTF-8 for output #19827
Conversation
In recent CI, we have failed console gathers like [1]: Gathering console logs for i-026ba0637214cd027 'ascii' codec can't encode character '\u2026' in position 14093: ordinal not in range(128) Failed to gather console logs This is descended from d060927 (gather/aws-console: Make log gathering non-fatal, 2021-04-14, openshift#17735). But we want to gather these logs, not silently fail to gather the logs. This commit restores our old "fail the step when we fail to gather" behavior, and sets PYTHONIOENCODING [2] to ask for UTF-8 instead of ASCII output. [1]: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-[…]e2e-aws-serial/gather-aws-console/build-log.txt [2]: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING
openshift-ci
bot
added
the
approved
|
Example of the fix working: $ AWS_PROFILE=ci PYTHONIOENCODING=ascii:strict aws --region us-east-1 ec2 get-console-output --instance-id i-024c99a8f35bdc94d --output text
'ascii' codec can't encode character '\u2026' in position 19168: ordinal not in range(128)
$ AWS_PROFILE=ci PYTHONIOENCODING=UTF-8:backslashreplace aws --region us-east-1 ec2 get-console-output --instance-id i-024c99a8f35bdc94d --output text | head -n3
i-024c99a8f35bdc94d ated 151 pages with 5 groups
[ 0.001000] rcu: Hierarchical RCU implementation.
[ 0.001000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cgwalters, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
In the random instance I picked out of CI for testing $ AWS_PROFILE=ci PYTHONIOENCODING=ascii:strict aws --region us-east-1 ec2 get-console-output --instance-id i-024c99a8f35bdc94d --output text
'ascii' codec can't encode character '\u2026' in position 19168: ordinal not in range(128)
$ AWS_PROFILE=ci PYTHONIOENCODING=UTF-8:strict aws --region us-east-1 ec2 get-console-output --instance-id i-024c99a8f35bdc94d --output text | head -n3
i-024c99a8f35bdc94d ated 151 pages with 5 groups
[ 0.001000] rcu: Hierarchical RCU implementation.
[ 0.001000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.The only issues with that instance were U+2026 $ AWS_PROFILE=ci PYTHONIOENCODING=ascii:backslashreplace aws --region us-east-1 ec2 get-console-output --instance-id i-024c99a8f35bdc94d --output text | grep -o '\\u[0-9]*' | sort | uniq -c
13 \u2026 |
|
@wking: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@wking: Updated the
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…YTHONIOENCODING 24b834c (ci-operator/step-registry/gather/aws-console: UTF-8 for output, 2021-06-29, openshift#19827) is not doing well in the wild. For example, [1]: INFO[2021-06-29T20:14:28Z] Gathering console logs for i-008f73fde1ef3f0dc 'ascii' codec can't encode character '\u2026' in position 13999: ordinal not in range(128) {"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-06-29T20:14:27Z"} error: failed to execute wrapped command: exit status 1 Apparently the AWS CLI is not using Python's default stream encoding here. But from [2,3,4] there is the suggestion that LC_ALL=en_US.UTF-8 will help. [1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.9-e2e-aws/1409949910362492928 [2]: https://awscli.amazonaws.com/v2/documentation/api/latest/topic/config-vars.html#locale [3]: https://github.com/aws/aws-cli/blob/2.2.5/awscli/compat.py#L121-L146 [4]: https://docs.python.org/3/library/locale.html#locale.getpreferredencoding
In recent CI, we have failed console gathers like:
This is descended from d060927 (#17735). But we want to gather these logs, not silently fail to gather the logs. This commit restores our old "fail the step when we fail to gather" behavior, and sets
PYTHONIOENCODINGto ask for UTF-8 instead of ASCII output.