Remove unwrap from public_key() function

[kevinlewi]

These unwraps have to be removed.

My idea was to use the appropriate SecretKey type instead of storing GenericArrays everywhere. I believe this would be better anyway, because conversion isn't free (it's very cheap though) and any checking is now guaranteed in the type and can now be handled during serialization and creation.

Originally posted by @daxpedda in #250 (comment)

[daxpedda]

RustCrypto/utils#652 is a problem we will have to circumvent if we want to use these types.