Crash when signature with invalid JWK is attempted

[samugi]

If an invalid JWK that is missing the private exponent p is passed to pkey.new() for the OKP type, the subsequent call to pkey:sign() will result in a crash and kill the worker.

Can be reproduced with:

local pkey, err = openssl_pkey.new(
  '{"kty" : "OKP", "crv" : "Ed25519","x" : "MKBCTNIcKUSDii11ySs3526iDZ8AiTo7Tu6KPAqv7D4"}',
  {format = "JWK"}
)
pkey:sign("something")

Using the following JWK instead works as expected: {"kty" : "OKP", "crv" : "Ed25519","x" : "MKBCTNIcKUSDii11ySs3526iDZ8AiTo7Tu6KPAqv7D4", "d" : "4Etl6SRW2YiLUrN5vfvVHuhp7x8PxltmWWlbbM4IFyM"}.

[fffonion]

Thanks @samugi , seems to be an bug on openssl, there's no guard before calling sign https://github.com/openssl/openssl/blob/OpenSSL_1_1_1r/crypto/ec/curve25519.c#L5442
then ultimately fail in
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1r/crypto/sha/sha512.c#L307

note in stacktrace private_key (a char[32] is NULL):

#0  SHA512_Update (c=c@entry=0x7fffffffd600, _data=<optimized out>, len=len@entry=32) at crypto/sha/sha512.c:307
#1  0x00007ffff730c565 in ED25519_sign (out_sig=out_sig@entry=0x7ffff7ea6720 "", message=message@entry=0x7ffff7eec948 "something",
    message_len=message_len@entry=9, public_key=0x5555558cb450 "}\243\216\067|\341k(t\005\207q\300N\363>#Gp`d\251\252\332I[\263\223. \fk",
    private_key=<optimized out>) at crypto/ec/curve25519.c:5442
#2  0x00007ffff735b281 in pkey_ecd_digestsign25519 (ctx=<optimized out>, sig=0x7ffff7ea6720 "", siglen=0x7ffff7ea6d80, tbs=0x7ffff7eec948 "something",
    tbslen=9) at crypto/ec/ecx_meth.c:753
#3  0x00007ffff7c0d135 in lj_vm_ffi_call () from /usr/local/openresty/luajit/lib/libluajit-5.1.so.2
...

let me submit an issue at upstream. for now i'm going to do a pre-check on if pkey is private key before sign.
other keys like EC, RSA will fail like pkey:sign: EVP_DigestSign: ec/ecdsa_ossl.c:174:error:100F907D:elliptic curve routines:ossl_ecdsa_sign_sig:missing private key

[fffonion]

openssl/openssl#19524

[fffonion]

It's now fixed in 0.8.15 release. I will keep this issue open for a bit for tracking.

[fffonion]

closing this now as it's fixed in upstream. openssl team decided to not backport this to 1.1.1 as it's not a security issue, so our own check to verify it's a private key should stay.