Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MANOPD-84031 Input files encoding rework and non-ASCII upload #346

Merged
merged 5 commits into from
Feb 16, 2023

Conversation

ilia1243
Copy link
Contributor

@ilia1243 ilia1243 commented Jan 27, 2023

Description

There are many places in KubeMarine where there can be an attempt to upload non-ASCII characters. This does not work because of issue paramiko/paramiko#1133
The problem is especially relevant for custom template plugins.

Fixes #342

Solution

  • NodeGroup.put now uploads data only as stream of bytes, not characters.
  • New requirement for all input text files to be strictly in utf-8 encoding.
    • This is reflected in new utils.py methods which now always open in utf-8.
    • Any internal or external text file should be opened using new methods.
  • New approach automatically leads to CRLF -> LF transformation of line endings on Windows deployer for any input text file (internal or external) except external config files for plugins with do_render=False.
  • Removed binary parameter for thirdparties as uploading from filesystem is not supported for external thirdparties.
  • Minor fix in services.packages.package_manager.repositories: now it allows input as path to file for both apt and yum package managers.

Breaking changes

If services.packages.package_manager.repositories for Debian-based cluster was passed as string, it will stop working as the string is now considered as path to file. No longer working configuration:

services:
  packages:
    package_manager:
      repositories: |
        deb http://archive.ubuntu.com/ubuntu focal main restricted
        deb http://archive.ubuntu.com/ubuntu focal-updates main restricted

New working configuration:

services:
  packages:
    package_manager:
      repositories: files/predefined.list

where files/predefined.list contains the repositories.

Test Cases

See #342

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Integration CI passed
  • Unit tests. If Yes list of new/changed tests with brief description
  • There is no merge conflicts

Unit tests

test_demo.py and demo.py to adopt to new upload behaviour.

@ilia1243 ilia1243 added bug Something isn't working documentation Improvements or additions to documentation labels Jan 27, 2023
@ilia1243 ilia1243 requested a review from shmo1218 January 27, 2023 11:36
@shmo1218
Copy link
Contributor

shmo1218 commented Feb 8, 2023

The doc review is done.

@ilia1243 ilia1243 force-pushed the bugfix/encoding_and_normalization branch from 53effcb to 5f720d4 Compare February 10, 2023 09:08
@koryaga koryaga added this to the 0.13.0 milestone Feb 14, 2023
@ilia1243 ilia1243 force-pushed the bugfix/encoding_and_normalization branch from 5f720d4 to 709c350 Compare February 15, 2023 08:27
@koryaga koryaga requested a review from alku89 February 15, 2023 10:06
@koryaga
Copy link
Contributor

koryaga commented Feb 15, 2023

@alku89 pls note about Breaking changes

@koryaga koryaga merged commit e0dc4af into main Feb 16, 2023
@koryaga koryaga deleted the bugfix/encoding_and_normalization branch February 16, 2023 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot upload io.StringIO with non-ASCII characters
4 participants