Crypto issues

[felixfontein]

I've tried to go through all crypto module issues and see which ones are more urgent (and should really be fixed for Ansible 2.8 - remember, community freeze is on March 21st!), either because they report bugs (most of them), or features which seem to be important. Is anyone interested in working on some of these? If you have specific comments on the issues, or want to work on them, please state so in the issue itself (so this issue won't get too confusing :) ). If you have general remarks, other other things which you think would be good to get done, please post them here.

openssl_certificate bugs:

• openssl_certificate traceback with empty file ansible#36738: module crashes when encounters invalid PEM file (PR in progress)
• openssl_certificate only asserts present extensions, ignoring absent ones ansible#38733: assertonly provider is not working properly when extensions are not present (openssl_certificate: make sure extensions are present when they are queried by assertonly ansible#53207 merged)
• openssl_certificate fail with acme provider ansible#41396: acme provider seems to be broken
• openssl_certificate - selfsigned_not_after => no attribute 'ASN1_TIME_set_string' ansible#46196: selfsigned_not_after seems to be broken (bug in library/setup)
• openssl_certificate: has_expired=no does nothing ansible#51267: has_expired seems to be broken (openssl_certificate, fixed has_expired to check the cert expiration date ansible#53168 merged)

openssl_certificate features:

• openssl_certificate should return assertion result ansible#33457: improve return values for assertonly provider
• module: openssl_certificate unusable on RHEL 7 ansible#34054: make module work with RHEL7 or other distributions with stone-age PyOpenSSL (idea: add cryptography backend?) (New cryptography backend for openssl_certificate ansible#53924 merged)

openssl_privatekey:

• openssl_privatekey can't create files which only have read permissions ansible#48656: cannot create read-only keys (will be fixed by openssl_privatekey: fix broken backup option ansible#54290)

openssl_pkcs12:

• openssl_pkcs12 error: name must be a byte string or None ansible#46047: friendly_name is required (see Issue 1 here)
• openssl_pkcs12: cannot create archive without a private key ansible#51703: cannot create archive without a private key
• openssl_pkcs12: idempotency check is not working ansible#53221: idempotency checking does not work (fixed in openssl_pkcs12: Add idempotency checks ansible#54633)

openssh_* modules:

• openssl_*: private key handling with passphrases? ansible#53535: input/output handling needs work

cc @MarkusTeufelberger @puiterwijk @resmo @Spredzy @Shaps @Xyon @dagwieers

[MarkusTeufelberger]

By the way - I'd be more than happy to completely eliminate pyOpenSSL from all crypto modules once they work reliably with cryptography, the only thing that I really need is compatibility with Debian Jessie (including backports - so cryptography 1.7.1 is good enough, see https://packages.debian.org/jessie-backports/python-cryptography, RHEL-6 seems to have 2.0.1).

Maybe worth a shot for 2.9?

[felixfontein]

I guess we should keep PyOpenSSL support for modules which supported it before (if it doesn't become extremely hard to maintain both), but yeah, that's one of the goals I have. For 2.8, I hope we'll have openssl_certificate as well (next to openssl_privatekey and openssl_csr, and the ones which never used PyOpenSSL in the first place). Having all others for 2.9 is hopefully more doable.

At least if some more people want to help with this :)

[MarkusTeufelberger]

pyOpenSSL is deprecated for years by now, see also the README in https://github.com/pyca/pyopenssl.

openssl_dhparams calls out to the openssl binary directly and get_certificate, openssl_pkcs12 as well as openssl_publickey only use pyOpenSSL so far, an easy first one for anyone interested would be the publickey module.

[felixfontein]

Yep, openssl_publickey is a good start. If someone's interested in openssl_pkcs12, it's probably better to look at the current issues first, they should give a good overview of how the module works internally.

Talking about stuff: there are two crypto PRs which need a review (openssl_privatekey and openssl_csr):
ansible/ansible#53593
ansible/ansible#53927
Anyone interested?

[felixfontein]

I've updated the above issues lists and hid all comments which are no longer relevant.

Please note that the freeze dates for Ansible 2.8 have been moved again (#346 (comment)). (You can subscribe to #346 to be updated about important changes for contributors.)

Also, there are a couple of issues for openssl_pkcs12 and the openssh_* modules (the latter are new for 2.8) which would be good if they could be resolved. Is anyone interested in working on them?

[felixfontein]

There are some non-openssl_* modules which need attention:

• openssh_cert and openssh_keypair:
  • openssh_* modules: input/output handling problematic ansible#54291: input/output handling problematic (crashes/hanging on bad input), avoiding trashed files when problems during regeneration appears
  • openssh_cert: idempotency check ignores signing_key, public_key, options and identifier ansible#54655: openssh_cert's idempotency check ignores changes to signing_key, public_key, options and identifier.
• luks_device:
  • needs rudimentary check mode support; i.e. PR luks_device: add basic check mode ansible#54477 needs review :)

Also, more input for the general discussion in ansible/ansible#54635 about potential futures of the assertonly provider for openssl_certificate would be appreciated :)

[felixfontein]

I've created some issues for tracking adding cryptography backends to modules which still rely on PyOpenSSL: ansible/ansible#59904, ansible/ansible#59905, ansible/ansible#59906

I've also created a PR (ansible/ansible#59907) which will deprecate the PyOpenSSL backend for openssl_certificate, openssl_certificate_info, openssl_csr, openssl_csr_info, openssl_privatekey, openssl_privatekey_info and announce its removal in Ansible 2.13.

[felixfontein]

Since there are currently a lot of open crypto-related PRs, here's an overview:

General PRs:

• openssl_*: deprecate PyOpenSSL backends ansible#59907: openssl_*: deprecate PyOpenSSL backends
• crypto modules: fix sanity errors ansible#60046: crypto modules: fix sanity errors

New cryptography backends:

• openssl_publickey: add cryptography backend ansible#60387: openssl_publickey: add cryptography backend
• get_certificate: add cryptography backend ansible#60599: get_certificate: add cryptography backend

Certificates:

• Entrust Datacard - Support for "entrust" provider in openssl_certificate module ansible#59272: openssl_certificate: support for entrust provider (Entrust Datacard)
• openssl_certificate: deprecate assertonly provider ansible#60623: openssl_certificate: deprecate assertonly provider
• openssl_certificate: various assertonly bugfixes ansible#60658: openssl_certificate: various assertonly bugfixes
• openssl_certificate: check for existence before loading CSR / private keys / ... ansible#60740: openssl_certificate: check for existence before loading CSR / private keys / ...
• openssl_certificate/csr(_info): add support for SubjectKeyIdentifier and AuthorityKeyIdentifier ansible#60741: openssl_certificate/csr(_info): add support for SubjectKeyIdentifier and AuthorityKeyIdentifier extensions
• openssl_certificate_info: add ocsp_uri return value ansible#60393: openssl_certificate_info: add ocsp_uri return value
• acme_certificate: make compatible to Buypass' ACME v2 testing endpoint ansible#60727: acme_certificate: make compatible to Buypass' ACME v2 testing endpoint
• acme_account_info: retrieve orders ansible#59697: acme_account_info: retrieve orders
• acme_certificate: only return challenges that need to be satisfied ansible#61191: acme_certificate: only return challenges that need to be satisfied
• acme_certificate: improve alternate chain handling ansible#60742: acme_certificate: improve alternate chain handling
• Addition of ECS_Certificate Module ansible#60883: Addition of ECS_Certificate Module

WIPs:

• openssl_privatekey: add support for format option ansible#60388: openssl_privatekey: add support for format option
• acme_certificate: add select_chain option ansible#60710: acme_certificate: add select_alternate_chain option

[felixfontein]

Reminder: the Feature Freeze for Ansible 2.9 is coming up on August 29th, i.e. in less than two weeks! (And I guess/somewhat hope that this time it won't be delayed as much as last time.) So if you want to add something like a new feature, please hurry up :)

[MarkusTeufelberger]

As an overview, here's the current versions of pyOpenSSL and cryptography of major distros as of today (2019-08-24):

Alpine Linux (https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases)

edge (rolling)

pyOpenSSL: 19.0.0 https://pkgs.alpinelinux.org/package/edge/main/x86_64/py-openssl
cryptography: 2.7 https://pkgs.alpinelinux.org/package/edge/main/x86_64/py-cryptography

3.7 (the oldest still supported release)

pyOpenSSL: 17.5.0 https://pkgs.alpinelinux.org/package/v3.7/main/x86_64/py-openssl
cryptography: 2.1.4 https://pkgs.alpinelinux.org/package/v3.7/main/x86_64/py-cryptography

Arch Linux (rolling)

pyOpenSSL: 19.0.0 https://www.archlinux.org/packages/extra/any/python-pyopenssl/
cryptography: 2.7 https://www.archlinux.org/packages/extra/x86_64/python-cryptography/

CentOS (supported similar to https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates)

8:

Not out yet (soon!)

7:

pyOpenSSL: 0.13.1 (EPEL: 17.3.0) https://pkgs.org/download/pyOpenSSL (No good online package version overview/search...)
cryptography: 1.7.2 (EPEL: 2.3) https://pkgs.org/download/cryptography

6:

pyOpenSSL: 0.13.1 https://pkgs.org/download/pyOpenSSL
cryptography: nope

Debian (https://wiki.debian.org/DebianReleases#Production_Releases)

Sid (rolling):

pyOpenSSL: 19.0.0 https://packages.debian.org/sid/python-openssl
cryptography: 2.6.1 https://packages.debian.org/sid/python-cryptography

Buster (10):

pyOpenSSL: 19.0.0 https://packages.debian.org/buster/python-openssl
cryptography: 2.6.1 https://packages.debian.org/buster/python-cryptography

Stretch (9):

pyOpenSSL: 16.2.0 https://packages.debian.org/stretch/python-openssl
cryptography: 1.7.1 (Backports: 2.3) https://packages.debian.org/stretch/python-cryptography

Jessie (8):

pyOpenSSL: 0.14 https://packages.debian.org/jessie/python-openssl
cryptography: 0.6.1 https://packages.debian.org/jessie/python-cryptography

Fedora (https://fedoraproject.org/wiki/Releases#Current_Supported_Releases)

Rawhide (rolling):

pyOpenSSL: 19.0.0 https://apps.fedoraproject.org/packages/pyOpenSSL
cryptography: 2.6.1 https://apps.fedoraproject.org/packages/python-cryptography

30:

pyOpenSSL: 19.0.0 https://apps.fedoraproject.org/packages/pyOpenSSL
cryptography: 2.6.1 https://apps.fedoraproject.org/packages/python-cryptography

29:

pyOpenSSL: 19.0.0 https://apps.fedoraproject.org/packages/pyOpenSSL
cryptography: 2.3 https://apps.fedoraproject.org/packages/python-cryptography

OpenSUSE (https://en.opensuse.org/Lifetime)

Tumbleweed (rolling):

pyOpenSSL: 19.0.0 https://software.opensuse.org/package/python-pyOpenSSL
cryptography: 2.7 https://software.opensuse.org/package/python-cryptography

Leap 15.1:

pyOpenSSL: 19.0.0 https://software.opensuse.org/package/python-pyOpenSSL
cryptography: 2.1.4 https://software.opensuse.org/package/python-cryptography

Leap 15.0:

pyOpenSSL: 17.5.0 https://software.opensuse.org/package/python-pyOpenSSL
cryptography: 2.1.4 https://software.opensuse.org/package/python-cryptography

Ubuntu (https://wiki.ubuntu.com/Releases)

Disco Dingo (19.04):

pyOpenSSL: 19.0.0 https://packages.ubuntu.com/disco/python-openssl
cryptography: 2.3 https://packages.ubuntu.com/disco/python-cryptography

Bionic Beaver (18.04):

pyOpenSSL: 17.5.0 https://packages.ubuntu.com/bionic/python-openssl
cryptography: 2.1.4 (updates: same) https://packages.ubuntu.com/bionic/python-cryptography

Xenial Xerus (16.04):

pyOpenSSL: 0.15.1 (updates: same) https://packages.ubuntu.com/xenial/python-openssl
cryptography: 1.2.3 (updates: same) https://packages.ubuntu.com/xenial/python-cryptography

Both Xenial and CentOS 7 should be out of support by mid-2021, so realistically cryptography 2.1.4 (or even newer!) should be a sane minimal version by then. Until April 2021 however, Ubuntu Xenial looks like the major pain in the butt to support. Not sure how/if someone could get newer versions (especially for cryptography) made available in their "updates" repo.

[felixfontein]

@MarkusTeufelberger thanks a lot for collecting this info! The deprecation cycle of 4 Ansible versions should cover two years, at least I hope so - if not, we can also extend the deprecation deadline a bit.

[felixfontein]

There are two PRs which need reviews:

• Add x509_crl module ansible#63435 Add x509_crl module
• openssl_* modules: allow direct input and output for some files ansible#65400 allow direct input and output for some files

Also, there's a "discussion" PR about renaming the openssl_* modules. If you have any ideas/preferences/..., please write something there!

• [WIP] OpenSSL modules: rename to x509_* ansible#63801 [WIP] OpenSSL modules: rename to x509_*

Finally, there's a discussion triggered by a core team decision on openssh_keypair and openssl_privatekey regenerating keys when passphrase does not match, keys are invalid, or more generally when potentially unexpected changes happen. Please write your opinions/ideas/wishes!

• openssh_keypair and openssl_privatekey: make idempotence configurable ansible#65639 openssh_keypair and openssl_privatekey: make idempotence configurable

[felixfontein]

Here's another PR:

• openssh_keypair and openssl_privatekey: add regenerate option ansible#67038

BTW, most modules will probably be moved out to other repositories soon, so it would be really cool if someone could review this and ansible/ansible#63435 so they have a chance of being merged before the move. :)

[felixfontein]

FYI: there's a schedule for the collection migration, see https://github.com/ansible-collections/overview/pull/3/files#diff-88b99bb28683bd5b7e3a204826ead112R112-R116 and https://github.com/ansible-collections/overview/pull/3/files#diff-88b99bb28683bd5b7e3a204826ead112R136-R139

This means that all new modules should be merged by March 2nd (even better earlier than that :) ). (Modules existing by then can be used in Ansible 2.10 without using FQCN, so there is some advantage in having them included by then.)

I'd be really happy if someone can give ansible/ansible#63435 (x509_crl) a review soon, so I can merge it (and add a basic x509_crl_info, whose tests will rely on x509_crl).

[MarkusTeufelberger]

Personally I'm slightly in favor of having a separate community.crypto collection by the way instead of being bunched together in the catch-all community.general one. Any other opinions?

[felixfontein]

@MarkusTeufelberger thanks for bringing this up! (I completely forgot about it...)

I'm also in favor of this, for two reasons:

1. it makes development easier since we don't need something like ansibot, and we can use some more of GitHub's features (like code ownership);
2. we can release LTS versions if we want to (it is not clear whether community.general will have that; I think it would be good to have something like that at least for crypto-related stuff).

Also, according to @gundalow we can have CI provided by Ansible (assuming we're staying in https://github.com/ansible-collections/).

Any other opinions on this? Are you interested in helping out with a community.crypto collection as well? (I'm explicitly pinging all of you not already involved: @Spredzy @Xyon @Shaps @puiterwijk)

[puiterwijk]

I think I have a slight preference for ‘community.crypto’.
Perhaps with that, it might also get easier to see why I got pinged for a specific issue, and I’d probably be more involved (I would like to remain involved).

I’m not sure I’m entirely convinced an LTE version is a great idea (though I’m not against it either), what would be the level of LTS you were thinking of?
At the underlying library level, or on the module level itself wrt its arguments etc?

[felixfontein]

I’m not sure I’m entirely convinced an LTE version is a great idea (though I’m not against it either), what would be the level of LTS you were thinking of?
At the underlying library level, or on the module level itself wrt its arguments etc?

Only at the module level w.r.t arguments, default values, features, deprecations. I think having something similar to what Ansible currently does (minor releases of the current three major versions for a certain amount of time which only contain backports of bugfixes, i.e. no new features, deprecations, feature removals, etc.) is a good model and makes it easier to rely on the modules in production.

It shouldn't be more work than the current backporting system, and the main drawback is that there will be more branches :-)

[gundalow]

It's too early to tell yet about Long-Term-Support for: Ansible-base (ansible/ansible), community.general, or any other collections yet.

If the repo lives in github.com/ansible-collections/ we can provide CI

Whomever from this group wants direct GitHub powers on the community.crypto repo can have it, so you can use nice things like assignment, GitHub Project Boards, Milestones, etc.

If you want community.crypto please add a 👍 to this issue, otherwise a 👎
Then once you are happy you've got the votes, request via https://github.com/ansible-collections/overview/issues/new/choose

This ideal should be decided in the next few days as we are finalising the scenario files for the migration.

[felixfontein]

I assume "this issue" == @gundalow's comment? Anyway, I'm +1.

[puiterwijk]

I’m not sure I’m entirely convinced an LTE version is a great idea (though I’m not against it either), what would be the level of LTS you were thinking of?
At the underlying library level, or on the module level itself wrt its arguments etc?

Only at the module level w.r.t arguments, default values, features, deprecations. I think having something similar to what Ansible currently does (minor releases of the current three major versions for a certain amount of time which only contain backports of bugfixes, i.e. no new features, deprecations, feature removals, etc.) is a good model and makes it easier to rely on the modules in production.

It shouldn't be more work than the current backporting system, and the main drawback is that there will be more branches :-)

Okay, that makes more sense than what I was at first guessing (and what my reply was about).
We'll see what ends up happening, but that's more sane than I thought, thanks!

[felixfontein]

@ctrufan are you also interested in this? The entrust modules are part of modules/crypto, so they would also move to community.crypto (if you don't want to move them somewhere else).

[ctrufan]

@felixfontein Thanks for pinging me! I don't necessarily have an opinion on the move to collections in general, I haven't been staying sufficiently in the loop, but since it looks like a migration is happening regardless? I would agree that it makes sense to have a crypto grouping.

[felixfontein]

BTW, there are two PRs which need a review:

• openssl_* modules, cryptography backend: parse dirName, RID and otherName names ansible#67669: openssl_* modules, cryptography backend: parse dirName, RID and otherName names;
• Add x509_crl_info module ansible#67539: add x509_crl_info module.

It would be nice if these could get merged before the repository lockdown next week :)

[gundalow]

As of today, I see 5 👍 and zero 👎 so we will create community.crypto

I've created a "scenario file" so that on the day of migration community.crypto will be created.
Once that's created I will give the people here Commit on that repo.

Can you all please carefully review ansible-community/collection_migration#462 and ensure I've not missed any

• inventory scripts/plugins
• any other plugins
• any other module_utils

Once happy, please leave your review on ansible-community/collection_migration#462

Thanks for your continued great work :)

[gundalow]

You can see the created collection here https://github.com/ansible-collection-migration/community.crypto This will be rebuilt (at least) daily until the migration is run for real.

[felixfontein]

FYI, the devel branch is now frozen: https://groups.google.com/d/msg/ansible-devel/6UAfogEDtG8/wMTgxBGWCgAJ

[felixfontein]

FYI: there are currently discussions in ansible-collections/overview#37 (also see ansible/ansible#68594) on collection versioning. The guidelines from ansible/ansible#68594 are mainly for "official" collections (which end up on Automation Hub), but I guess something like that also makes sense for community.crypto. I've sketched out a proposal here which I think would be well-suited for community.crypto: ansible-collections/overview#37 (comment) Any opinions on this?

This would be very close to what we currently have in ansible/ansible, and every active release branch would be similar to a LTS version.

[gundalow]

All,
Since we now have a dedicated repo for community.crypto I'd be worth considering creating a pinned issue in https://github.com/ansible-collections/community.crypto/issues

1. You can see examples of pinned issues at the top of community issues
2. I'd hope that more people would see the new issue, as everybody that goes to create or look at existing Crypto issues would see this new comment thread at the top.
3. I know we have wiki's though they seem to quickly get out of date. Now we have a dedicated repo we can use GitHub project boards or GitHub milestones which give a better visual representation, rather than a list in a wiki

Welcome peoples thoughts on this or anything else we can take advantage of now we have a dedicated repo, with more contributors that have direct GitHub powers.

[felixfontein]

I think this is an excellent idea, and created a pinboard issue at ansible-collections/community.crypto#24. Please subscribe to it!

[felixfontein]

If nobody complains, I will close this in a week.