Document how to integrate with ORY / Hydra

[isodude]

What would you like to be added

A way to use Hydra as OIDC

Why this is needed

Because it's a great open source project!

[isodude]

I documented my trial and error here: https://github.com/isodude/step-ca-oidc-hydra, it's not working at all right now.

[isodude]

So I had a bug where I sent openid=on instead of grant_scope=openid to consent. A bit embarrassing.
I still get error parsing token: missing payload in JWS message though, which I get even if I do the token flow with the browser.

[mmalone]

The error makes it sound like the identity token being returned from Hyrda is not a properly formatted JWT (a JWT is a subtype of JWS). The general format of a JWS is <header>.<payload>.<signature>. Each part is base64 encoded.

The error makes it sound like the payload portion is missing from the JWS. My best guess is that the token you're getting back is actually an OAuth access token rather than an OpenID Connect (OIDC) identity token. I'm not sure how / why that might be happening, but if you have an example token you could share we can confirm whether or not that's what's happening, which should at least help narrow down what's going wrong.

[isodude]

Thanks, I've tried a couple of options here, like --oidc etc, no go though. Here's the log and token

{
  "access_token": "11WUOTq0w804852v8ABLjCvMBK5zSvoqIEVQ1kfco_I.9gerqFNc-twKlFBjtqEqVvOIb8XSfRy8NhfVyrisyqw",
  "id_token": "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzplNzRlZmEwYy1iMWIwLTQ2YWUtOWQxNy03OTU2NTdkNWRiMmEifQ.eyJhY3IiOiIwIiwiYXRfaGFzaCI6IjRlb09UQnpMdThweDRXcDlHcVZPY0EiLCJhdWQiOlsidXNlciJdLCJhdXRoX3RpbWUiOjE2MzE2MDIzMzMsImV4cCI6MTYzMTYwNTkzNywiaWF0IjoxNjMxNjAyMzM3LCJpc3MiOiJodHRwczovLzEyNy4wLjAuMTo0NDQ0LyIsImp0aSI6Ijc4OGE1ZTFhLWY3OWItNDBjMi05OWRlLWE3MWZmYzAyZDAwMiIsIm5vbmNlIjoiMDNjYjc1MWEwNDFmNmQxOTVlOWE5ZmUzNjU4MDE1ZDgwNDAzNWEwNmMxMDNhNTMzMDUwZjkyNGQwZGQ1NWU3MyIsInJhdCI6MTYzMTYwMjMzMSwic2lkIjoiZDJlYWRhNzQtMjU0OC00OWI0LTg5YjEtNDMyYjQ3NTE3YzMyIiwic3ViIjoiZm9vQGJhci5jb20ifQ.DC4bMB5HJq2UqNXjY3lNzM2Ri3VN1c16Ln_sLzulqpdSaeDgtqabOgLZE7M3QyW6zeLFNTDeESc8nhYvcFfnCi8WueIkg8G9Zi8LLYBkGgVMxV8iOq28vRdJxDl01KCvhdNwbPeM91JLLoKdreixDEccBQSfqdFSa1a-1ZmAQZuHpm7qwOwBL8FhG5AXbR38BEW9cBLdtsO-59eHusfHrKSAJrYVddcn0cF5WxP2-nZDx5dEWTypti4GZoi3gh1SpburDkrXvP8amYiJA3ffnAzvZ8GqHfZ62lxq-On9UAG0XYwCS82cy7BmIUrXkho62zGiaaACnxM_k4lX2XWdpCvRmhg0k5KyQHsdN1r_5iSnsnbqLN8uuWHOCGCU29XTLmNbM-Krgeyn1Mt-D57Jt0zWyFOmZtu-y6IPrmyhbgihV0A9LIFwqkcwbSgnsertzX3GOKRi0DpC0OTMpf3jgUCg-_UYaTNiNlAWzTxL6YY4Eq8pcyFWJgO1k4roGIbdJM8b65_qprG9Cwol1F0_vkWssc0gRSwyxS6HdeGURGPoU35u7ksfoAVV_fPKv5wFXgMcpYZytGDvNR2-FQaIIBZvKCV5xbLDod-kF7dTYLnHLbs-NgUh2RgW53NehxFZpddCZoTcze1UVX4_kpkkVNA0vL5_kyHvTyE8i6nw8vE",
  "refresh_token": "",
  "expires_in": 3600,
  "token_type": "bearer"
}

user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ ./run.sh 
+ docker-compose down -v
Stopping step1-connect-ory-hydra-step-ca_consent_1 ... done
Stopping step1-connect-ory-hydra-step-ca_hydra_1   ... done
Stopping step1-connect-ory-hydra-step-ca_step-ca_1 ... done
Removing step1-connect-ory-hydra-step-ca_curl_run_6aaad83cc060 ... done
Removing step1-connect-ory-hydra-step-ca_curl_run_13be4ede3f48 ... done
Removing step1-connect-ory-hydra-step-ca_consent_1             ... done
Removing step1-connect-ory-hydra-step-ca_hydra_1               ... done
Removing step1-connect-ory-hydra-step-ca_step-ca_1             ... done
Removing volume step1-connect-ory-hydra-step-ca_hydra-sqlite
+ rm -vrf step-ca conf hydra-sqlite
removed 'step-ca/secrets/password'
removed 'step-ca/secrets/provisioner-password'
removed 'step-ca/secrets/intermediate_ca_key'
removed 'step-ca/secrets/root_ca_key'
removed directory 'step-ca/secrets'
removed 'step-ca/db/MANIFEST'
removed 'step-ca/db/KEYREGISTRY'
removed 'step-ca/db/000000.vlog'
removed 'step-ca/db/000004.sst'
removed directory 'step-ca/db'
removed 'step-ca/certs/root_ca.crt'
removed 'step-ca/certs/intermediate_ca.crt'
removed directory 'step-ca/certs'
removed 'step-ca/config/ca.json'
removed 'step-ca/config/defaults.json'
removed directory 'step-ca/config'
removed directory 'step-ca/templates'
removed directory 'step-ca'
removed 'conf/cert.crt'
removed 'conf/hydra.yaml'
removed 'conf/ca-certificates.crt'
removed 'conf/cert.key'
removed directory 'conf'
+ mkdir -pv conf
mkdir: created directory 'conf'
+ cat
++ cat /proc/sys/kernel/random/uuid
++ cat /proc/sys/kernel/random/uuid
+ mkdir -pv step-ca/secrets/
mkdir: created directory 'step-ca'
mkdir: created directory 'step-ca/secrets/'
+ cat /proc/sys/kernel/random/uuid
+ cat /proc/sys/kernel/random/uuid
+ touch conf/ca-certificates.crt
+ docker-compose run --rm --entrypoint step step-ca ca init --name step-ca --dns step-ca --dns 127.0.0.1 --dns localhost --address :443 --deployment-type standalone --provisioner admin --provisioner-password-file /home/step/secrets/provisioner-password --password-file /home/step/secrets/password
Creating volume "step1-connect-ory-hydra-step-ca_hydra-sqlite" with default driver

Generating root certificate... done!
Generating intermediate certificate... done!

✔ Root certificate: /home/step/certs/root_ca.crt
✔ Root private key: /home/step/secrets/root_ca_key
✔ Root fingerprint: e561b20373ddc1d5e902b5630c560f5948039dc38699c43322ed3d3cfab0343c
✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
✔ Database folder: /home/step/db
✔ Default configuration: /home/step/config/defaults.json
✔ Certificate Authority configuration: /home/step/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

FEEDBACK 😍 🍻
  The step utility is not instrumented for usage statistics. It does not phone
  home. But your feedback is extremely valuable. Any information you can provide
  regarding how you’re using `step` helps. Please send us a sentence or two,
  good or bad at [email protected] or join GitHub Discussions
  https://github.com/smallstep/certificates/discussions and our Discord 
  https://u.step.sm/discord.
+ cat step-ca/certs/root_ca.crt
+ docker-compose up -d step-ca
Creating step1-connect-ory-hydra-step-ca_step-ca_1 ... done
+ touch conf/cert.crt conf/cert.key
+ docker-compose run --rm hydra migrate -c /etc/ory/hydra.yaml sql -e --yes
INFO[2021-09-14T06:51:49Z] No tracer configured - skipping tracing setup  audience=application service_name=ORY Hydra service_version=v1.10.6
The following migration is planned:

Version                Name                                              Status    
20190100000001000000   client                                            Pending   
20190100000002000000   client                                            Pending   
20190100000003000000   client                                            Pending   
20190100000004000000   client                                            Pending   
20190100000005000000   client                                            Pending   
20190100000006000000   client                                            Pending   
20190100000007000000   client                                            Pending   
20190100000008000000   client                                            Pending   
20190100000009000000   client                                            Pending   
20190100000010000000   client                                            Pending   
20190100000011000000   client                                            Pending   
20190100000012000000   client                                            Pending   
20190100000013000000   client                                            Pending   
20190100000014000000   client                                            Pending   
20190200000001000000   jwk                                               Pending   
20190200000002000000   jwk                                               Pending   
20190200000003000000   jwk                                               Pending   
20190200000004000000   jwk                                               Pending   
20190300000001000000   consent                                           Pending   
20190300000002000000   consent                                           Pending   
20190300000003000000   consent                                           Pending   
20190300000004000000   consent                                           Pending   
20190300000005000000   consent                                           Pending   
20190300000006000000   consent                                           Pending   
20190300000007000000   consent                                           Pending   
20190300000008000000   consent                                           Pending   
20190300000009000000   consent                                           Pending   
20190300000010000000   consent                                           Pending   
20190300000011000000   consent                                           Pending   
20190300000012000000   consent                                           Pending   
20190300000013000000   consent                                           Pending   
20190300000014000000   consent                                           Pending   
20190400000001000000   oauth2                                            Pending   
20190400000002000000   oauth2                                            Pending   
20190400000003000000   oauth2                                            Pending   
20190400000004000000   oauth2                                            Pending   
20190400000005000000   oauth2                                            Pending   
20190400000006000000   oauth2                                            Pending   
20190400000007000000   oauth2                                            Pending   
20190400000008000000   oauth2                                            Pending   
20190400000009000000   oauth2                                            Pending   
20190400000010000000   oauth2                                            Pending   
20190400000011000000   oauth2                                            Pending   
20200521071434000000   consent                                           Pending   
20200527215731000000   client                                            Pending   
20200527215732000000   client                                            Pending   
20200819163013000000   add_client_id_subject_idx_to_access_and_refresh   Pending   
20200913192340000000   initial_sqlite                                    Pending   
20201110104000000000   drop_uq_oauth2                                    Pending   
20201116133000000000   set_null_time                                     Pending   
Successfully applied migrations!
+ docker-compose up -d hydra
Creating step1-connect-ory-hydra-step-ca_hydra_1 ... done
+ docker-compose exec step-ca step ca certificate --san hydra --san consent --san step-ca --san 127.0.0.1 --san localhost --provisioner-password-file /home/step/secrets/provisioner-password step-ca /tmp/cert.crt /tmp/cert.key
✔ Provisioner: admin (JWK) [kid: yeD4QMdIr6dTDyalf_CFI6jUgnquBZMka0wBQr-bUeo]
✔ CA: https://localhost
✔ Certificate: /tmp/cert.crt
✔ Private Key: /tmp/cert.key
+ docker-compose exec step-ca cat /tmp/cert.crt
+ docker-compose exec step-ca cat /tmp/cert.key
+ docker-compose up -d hydra
step1-connect-ory-hydra-step-ca_hydra_1 is up-to-date
+ docker-compose up -d consent
Creating step1-connect-ory-hydra-step-ca_consent_1 ... done
+ sleep 2
+ docker-compose exec hydra hydra clients create --grant-types client_credentials --grant-types authorization_code --name user --secret password --response-types=code --response-types=token --token-endpoint-auth-method client_secret_post --scope=openid --scope=email --callbacks http://127.0.0.1:10000 --id user
You should not provide secrets using command line flags, the secret might leak to bash history and similar systems
OAuth 2.0 Client ID: user
+ docker-compose exec hydra hydra clients create --grant-types client_credentials --grant-types authorization_code --name step-ca --secret step-ca --response-types=code --token-endpoint-auth-method client_secret_post --scope=openid --scope=email --callbacks http://127.0.0.1:10000 --id step-ca
You should not provide secrets using command line flags, the secret might leak to bash history and similar systems
OAuth 2.0 Client ID: step-ca
+ docker-compose exec step-ca step ca provisioner add hydra --type oidc --ca-config /home/step/config/ca.json --client-id step-ca --client-secret step-ca --configuration-endpoint https://hydra:4444
Success! Your `step-ca` config has been updated. To pick up the new configuration SIGHUP (kill -1 <pid>) or restart the step-ca process.
+ docker-compose restart step-ca
Restarting step1-connect-ory-hydra-step-ca_step-ca_1 ... done
+ printf '%s\n' 'Run ./auth.sh "<url>"'
Run ./auth.sh "<url>"
+ docker run -v /home/user/ory-step-ca/step1-connect-ory-hydra-step-ca/conf/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt -it --rm --net host smallstep/step-ca step oauth --client-id user --client-secret password --provider https://127.0.0.1:4444 --listen=127.0.0.1:10000 --redirect-url=http://127.0.0.1:10000 --oidc
there is no ca.json config file; please run step ca init, or provide config parameters via DOCKER_STEPCA_INIT_ vars
Cannot open a web browser on your platform.

Open a local web browser and visit:

https://127.0.0.1:4444/oauth2/auth?client_id=user&code_challenge=jlynIVUStoiFwm4ATkKWS-rDQbLgN4ddJxznj6RXs9Q&code_challenge_method=S256&nonce=03cb751a041f6d195e9a9fe3658015d804035a06c103a533050f924d0dd55e73&redirect_uri=http%3A%2F%2F127.0.0.1%3A10000&response_type=code&scope=openid+email&state=ih11QEx4rFvIlUA5RBZHWfARGSf7icxi

{
  "access_token": "11WUOTq0w804852v8ABLjCvMBK5zSvoqIEVQ1kfco_I.9gerqFNc-twKlFBjtqEqVvOIb8XSfRy8NhfVyrisyqw",
  "id_token": "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzplNzRlZmEwYy1iMWIwLTQ2YWUtOWQxNy03OTU2NTdkNWRiMmEifQ.eyJhY3IiOiIwIiwiYXRfaGFzaCI6IjRlb09UQnpMdThweDRXcDlHcVZPY0EiLCJhdWQiOlsidXNlciJdLCJhdXRoX3RpbWUiOjE2MzE2MDIzMzMsImV4cCI6MTYzMTYwNTkzNywiaWF0IjoxNjMxNjAyMzM3LCJpc3MiOiJodHRwczovLzEyNy4wLjAuMTo0NDQ0LyIsImp0aSI6Ijc4OGE1ZTFhLWY3OWItNDBjMi05OWRlLWE3MWZmYzAyZDAwMiIsIm5vbmNlIjoiMDNjYjc1MWEwNDFmNmQxOTVlOWE5ZmUzNjU4MDE1ZDgwNDAzNWEwNmMxMDNhNTMzMDUwZjkyNGQwZGQ1NWU3MyIsInJhdCI6MTYzMTYwMjMzMSwic2lkIjoiZDJlYWRhNzQtMjU0OC00OWI0LTg5YjEtNDMyYjQ3NTE3YzMyIiwic3ViIjoiZm9vQGJhci5jb20ifQ.DC4bMB5HJq2UqNXjY3lNzM2Ri3VN1c16Ln_sLzulqpdSaeDgtqabOgLZE7M3QyW6zeLFNTDeESc8nhYvcFfnCi8WueIkg8G9Zi8LLYBkGgVMxV8iOq28vRdJxDl01KCvhdNwbPeM91JLLoKdreixDEccBQSfqdFSa1a-1ZmAQZuHpm7qwOwBL8FhG5AXbR38BEW9cBLdtsO-59eHusfHrKSAJrYVddcn0cF5WxP2-nZDx5dEWTypti4GZoi3gh1SpburDkrXvP8amYiJA3ffnAzvZ8GqHfZ62lxq-On9UAG0XYwCS82cy7BmIUrXkho62zGiaaACnxM_k4lX2XWdpCvRmhg0k5KyQHsdN1r_5iSnsnbqLN8uuWHOCGCU29XTLmNbM-Krgeyn1Mt-D57Jt0zWyFOmZtu-y6IPrmyhbgihV0A9LIFwqkcwbSgnsertzX3GOKRi0DpC0OTMpf3jgUCg-_UYaTNiNlAWzTxL6YY4Eq8pcyFWJgO1k4roGIbdJM8b65_qprG9Cwol1F0_vkWssc0gRSwyxS6HdeGURGPoU35u7ksfoAVV_fPKv5wFXgMcpYZytGDvNR2-FQaIIBZvKCV5xbLDod-kF7dTYLnHLbs-NgUh2RgW53NehxFZpddCZoTcze1UVX4_kpkkVNA0vL5_kyHvTyE8i6nw8vE",
  "refresh_token": "",
  "expires_in": 3600,
  "token_type": "bearer"
}
+ printf '%s\n' 'Run ./token.sh '\''token'\'''
Run ./token.sh 'token'
user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ vim token
user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ ./token.sh "$(<token)"
error parsing token: missing payload in JWS message

[isodude]

You are correct there is no mention of oauth2 in acr

user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ curl --cacert conf/ca-certificates.crt -H "Authorization: Bearer $(cat token | jq -r '.access_token')" https://127.0.0.1:4444/userinfo
{"acr":"0","aud":["user"],"auth_time":1631602333,"iat":1631602336,"iss":"https://127.0.0.1:4444/","rat":1631602331,"sub":"[email protected]"}

[isodude]

I added

- CONFORMITY_FAKE_CLAIMS=1

To the consent app and now there's more info at least. Seems that the consent app is not getting it's ACR value.

user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ curl --cacert conf/ca-certificates.crt -H "Authorization: Bearer $(cat token | jq -r '.access_token')" https://127.0.0.1:4444/userinfo > info
user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ cat info
{"acr":"0","address":{"country":"Localhost","region":"Intranet","street_address":"Local Street 1337"},"aud":["[email protected]"],"auth_time":1631603783,"birthdate":"1.1.2014","email":"[email protected]","email_verified":true,"family_name":"Bar","gender":"robot","given_name":"Foo","iat":1631603787,"iss":"https://127.0.0.1:4444/","locale":"en-US","middle_name":"Baz","name":"Foo Bar","nickname":"foobot","phone_number":"1337133713371337","phone_number_verified":true,"picture":"https://raw.githubusercontent.com/ory/web/master/static/images/favico.png","preferred_username":"robot","profile":"https://www.ory.sh","rat":1631603782,"sub":"56d9d6e4de0cda92d82c825384e774f81112432086e262d029704ee6a62a48f1","updated_at":1604416603,"website":"https://www.ory.sh","zoneinfo":"Europe/Belrin"}

[isodude]

Ok so, I tried to generate a certificate via ca certificate again, now with --net host on dockers etc.

user@internet:~/ory-step-ca/step1-connect-ory-hydra-step-ca$ docker-compose exec step-ca step ca certificate --root /etc/ssl/certs/ca-certificates.crt --ca-url https://localhost --provisioner hydra  [email protected] /tmp/p.crt /tmp/p.key 
✔ Provisioner: hydra (OIDC) [client: step-ca]
Cannot open a web browser on your platform.

Open a local web browser and visit:

https://127.0.0.1:4444/oauth2/auth?client_id=step-ca&code_challenge=4vMZFXl2sOxuGIZpKfubf6T-t3HS22R0IdYHmkG5o_M&code_challenge_method=S256&nonce=2858203479c49e42c7ecbd07f0c8d39d035ee36942379bbd9ac287eb4823f789&redirect_uri=http%3A%2F%2F127.0.0.1%3A41729&response_type=code&scope=openid+email&state=eAdIWqZk69dIBnkjAX3aggYGb6fN7A5R

✔ CA: https://localhost
✔ Certificate: /tmp/p.crt
✔ Private Key: /tmp/p.key

So all good, I've documented in the run.sh-script how the setup works and in the auth.sh-script how the auth-flow works.

[dopey]

@isodude That's great! Is this issue "resolved"? Should we turn it into a discussion? Should we convert this to actual documentation? https://github.com/smallstep/docs? @tashian @devadvocado

[tashian]

If it's resolved, I think we should turn it into a discussion for posterity. @dopey @devadvocado

[dopey]

My only question was -- is there more here than just a discussion? Like a tutorial or something. But I think a discussion is a good start. Let's see if we keep having to point people to it or if people find it organically and are able to resolve their own issue.

[isodude]

I think the most headache is around using localhost in everything.
I want to take this one step further and remove that from the script all together really to make it more like a real world example.

For that to work I need to sign a certificate for a local node such that the redirect uri is a https, also I think I will run into trouble with step-ca ca certificate and the fact that it chooses a port at random I think?

But yeah, as I have a working example (yay!) we can turn this into a discussion and archive it, and start a MR for a proper documentation instead.

The run.sh-script and the auth.sh-script is the actual document here where the whole flow is written down. It is also does not need any user input at all so it can be used as a test :-)

[isodude]

One thing I don't get is that..

When I run step ca certificate I use the provisioner given to me by the CA, which contains clientId and clientSecret.
This is then used for step-ca oauth.

Isn't the provisioner supposed to be a secret between step-ca and hydra?

[mmalone]

Hey @isodude,

The redirect to localhost (er, 127.0.0.1, ideally) is by design, and is suitable in real-world / production environments. That said, it is not your typical OAuth flow. We designed the OAuth flow to work 100% client-side. The CA doesn't participate in the flow. The local step client listens on 127.0.0.1 to handle the OAuth redirect and obtain an OIDC identity token. The CA simply receives the identity token from the client, and doesn't actually care how it was obtained. So these two pieces of the workflow are very loosely coupled.

For tons more details, we follow OAuth Best Current Practices for Native Apps, documented in BCP212 / RFC8252 at https://www.rfc-editor.org/rfc/rfc8252.txt.

Because this is a somewhat unique flow, some OAuth IdPs have issues handling it. Even the big ones like Okta and Azure AD have problems (e.g., only support localhost while 127.0.0.1 is recommended, or require a fixed port instead of accepting a random port on 127.0.0.1 for the redirect). We've built features into step and step-ca to work around these quirks, where necessary. We also try to open issues with the IdP to tell them to fix their stuff and please implement best current practice recommendations.

If there is anything specific in the Ory / Hydra world that needs to be done (e.g., it sounds like we need a fixed port specified?) that's worth documenting somewhere (a Github discussion is probably sufficient for now).

Regarding the client secret... yes, typically it's meant to be kept secret. But this is not possible, and not required, for "public clients" used by "native apps". Some IdPs explicitly support "public clients" that don't even have a secret, just a client id. The BCP / RFC linked above has all the details. If it makes you feel any better, this is the same pattern that the gcloud command-line tool uses for GCP. It's defined in their open source Python code and called the clientNotSoSecret :).

[isodude]

Thanks for the thorough answer.

Yeah, being able to specify a local port would be ideal. However it seems that hydra accepts the local port if http://127.0.0.1 is specified as redirect_uri/callbacks.

It's quite confusing with the clients and how they are meant to be used, even though you read about it over and over.

Just to be clear, issuing step ca certificate, like how I do in run.sh, is the correct way of doing it as a client right?

[mmalone]

No problem!

You can tell the step CLI to listen on a specific port by specifying a listenAddress in your OIDC provisioner configuration, like this:

{
        "type": "OIDC",
        "name": "Hydra",
        "clientID": ...,
        "clientSecret": ...,
        "configurationEndpoint": ...,
        "listenAddress": ":10000",
 ...

Yes, OAuth2 is super complicated and confusing. I've been involved since the early days of OAuth 1.0 almost 15 years ago and I still have to go look stuff up in RFCs on a regular basis!

The way you're using step ca certificate looks fine. One of our design goals is "misuse prevention", so you really have to try to do something unsafe/insecure :)