Unable to config keycloak if use https

[LarryZhangy]

Description

If config https to run che by use nightly image, it can't access keycloak by run /scripts/keycloak_config.sh in keycloak pod， Because there is no https config in keycloak_config.sh

I can access https://keycloak-domaim/auth on chrome.

Reproduction Steps

config keycloak by run job

apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-configure-job
spec:
  template:
    spec:
      initContainers:
      - name: wait-for-keycloak
        image: <my>/alphine:3.5
        command: ["sh", "-c", " adresses_length=0; until [ $adresses_length -gt 0 ]; do echo waiting for keycloak to be ready...; sleep 2; endpoints=`curl -s --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\"     https://kubernetes.default.svc/api/v1/namespaces/$POD_NAMESPACE/endpoints/keycloak`; adresses_length=`echo $endpoints | jq -r \".subsets[]?.addresses // [] | length\"`; done;"]
        env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
      containers:
      - name: keycloak-configure-job
        image: eclipse/che-keycloak:nightly
        command: ["/scripts/keycloak_config.sh"]
        env:
          - name: HTTP_PROTOCOL
            value: "https"
          - name: KC_HOST
            value: <domain>
          - name: CHE_KEYCLOAK_ADMIN_REQUIRE_UPDATE_PASSWORD
            value: "true"
          - name: CHE_HOST
            value: <domain>
      serviceAccountName: che-keycloak
      restartPolicy: Never
  #backoffLimit: 4

OS and version:

all image of che: nightly

Diagnostics:

[LarryZhangy]

@guydaichs

[guydc]

There are a couple of issues with running KeyCloak with TLS:

1. Initial Che realm configuration (keycloak_config.sh) might fail due to SSL enforcement on master realm, if the pod's cluster IP is not in a certain range (10.0.x.x, 192.168.x.x, and 172..16.x.x).
2. Che doesn't configure KeyCloak with a TLS-terminating reverse proxy. As a result, the OIDC provider discovery fails, and you cannot login with KeyCloak.
3. With multi-host configuration, you will also run into CORS issues.

Essentially, running Multi-User Che with TLS is not yet supported (Single-User with TLS should work). I'll update the docs and values files to reflect this.

To support this scenario, some changes need to be made to the KeyCloak configuration script, Ingress definitions and deployment orchestration. This issue should be added to #5908.

There is a nice reference for running KeyCloak on k8s with Ingress TLS termination. This is a very flexible chart, that relies on wildfly CLI configuration.

Currently, Che uses a simple KeyCloak configuration mechanism that is common to docker, OpenShift and Kubernetes infrastructures. Changes to this mechanism require a discussion with all relevant stakeholders.

@skabashnyuk @akorneta @eivantsov @davidfestal @mshaposhnik - WDYT?

[ghost]

@guydaichs we are going to change the way Keycloak is provisioned with master and che realm, as well as admin users for both: https://github.com/eclipse/che/blob/templates/dockerfiles/keycloak/kc_realm_user.sh this will be the entrypoint. You may give it a try with eivantsov/keycloak:nightly image

[ghost]

@LarryZhangy @guydaichs if you change entrypoint in keycloak image and execute kc_realm_user.sh script, no rest calls will be initiated.

I should probably issue a PR since we don't need Keycloak job, as just running a different entrypoint auto configures keycloak https://github.com/eclipse/che/blob/master/deploy/kubernetes/helm/che/charts/che-keycloak/templates/deployment.yaml#L48

[guydc]

Sounds good to me. This will solve the initial configuration issue. Additionally, KeyCloak should be configured to support Ingress as a TLS terminator.

[ghost]

@guydaichs i wonder why it is not required for OpenShift routes that are HAProxy based. We just use Keycloak as is.

[ghost]

@guydaichs @LarryZhangy I have issued a PR - works fine with http on MiniKube - realm and user provisioning happens in entrypoint, and no REST calls are made.

[arielbangiev]

@eivantsov, @guydaichs, we tried to deploy Che to Kubernetes with multiuser and TLS after your merge of PR #9533.

The host of the Keycloak ingress looks something like this:

https://my-server.com/auth/

A valid certificate is served when accessing this url.

The PROTOCOL environment variable of the Keycloak pod is appropriately set to https, but the /auth/admin/master/console/index.html file contains links and scripts that have the http prefix:

    <link href="http://my-server.com/auth/resources/3.3.0.cr2/admin/keycloak/lib/patternfly/css/patternfly.css" rel="stylesheet" />
    <link href="http://my-server.com/auth/resources/3.3.0.cr2/admin/keycloak/node_modules/select2/select2.css" rel="stylesheet" />

Consequently, the Keycloak Administration Console doesn't load due to mixed content.

Can you please take a look?

[ghost]

@arielbangiev PROTOCOL env is used only by the entrypoint script to configure realm and and client reidrectUris and webOrigins.

As far as I understand, automatic redirection to https should happen when tls is on - https://github.com/eclipse/che/blob/master/deploy/kubernetes/helm/che/charts/che-keycloak/templates/ingress.yaml#L16-L17

Does Keycloak fail to load because of this? What does browser dev console yells at?

[perspectivus1]

@eivantsov, the Keycloak ingress is configured correctly -- it accepts https requests and forwards them to the Keycloak server. The issue is the generation of Keycloak Admin Console client resources. All links are generated using full path, including protocol and hostname.

The browser throws a mixed content error:

Mixed Content: The page at 'https://my-server.com/auth/admin/master/console/' was loaded over HTTPS, but requested an insecure favicon 'http://my-server.com/auth/resources/3.3.0.cr2/admin/keycloak/img/favicon.ico'. This request has been blocked; the content must be served over HTTPS.

[ghost]

@perspectivus1 I wonder why it works correctly on OpenShift. Routes are set to Allow or Redirect insecure traffic (both work).

[annatech]

I can confirm that I have experienced the same issue as @perspectivus1. For whatever reason, the resource URLs do appear to be hard-coded to http:// . Is this something that can be fixed in the Keycloak Client configuration? Note however, that I have tried making adjustments there to force HTTPS URL paths, but they have had no effect. I’m pretty sure that if the resource URLs obeyed the page protocol that there should be no issue running multi-user Eclipse Che / Keycloak under HTTPS on OpenShift. Steve Tsiopanos | President/Owner [/Users/stsiopanos/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1391708366] www.annatech.com<http://www.annatech.net/> | [email protected]<mailto:[email protected]> From: Eugene Ivantsov <[email protected]> Reply-To: eclipse/che <[email protected]> Date: Monday, May 7, 2018 at 3:54 AM To: eclipse/che <[email protected]> Cc: Sotirios Tsiopanos <[email protected]>, Manual <[email protected]> Subject: Re: [eclipse/che] Unable to config keycloak if use https (#9429) @perspectivus1<https://github.com/perspectivus1> I wonder why it works correctly on OpenShift. Routes are set to Allow or Redirect insecure traffic (both work). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#9429 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABZREVHampAhbQHwBDLQbMw3uMbMniT1ks5tv_22gaJpZM4TVWzv>.

[ghost]

@annatech

I’m pretty sure that if the resource URLs obeyed the page protocol that there should be no issue running multi-user Eclipse Che / Keycloak under HTTPS on OpenShift.

Eclipse Che multi-user support https mode on OpenShift. It's tried and tested. That's why I ask myself and those you may have any ideas - why does it work with OpenShift routes and doesn't work with ingress?

[dbengtson]

@eivantsov

Setting PROXY_ADDRESS_FORWARDING to true in the keycloak deployment fixed this issue for me in my GKE install.

[mjshashank]

@dbengtson I seem to be facing a similar issue. Where exactly does this change need to be made to fix it?

Note: Mine is a GKE installation as well.

[dbengtson]

@mjshashank

In the -env section for the keycloak deployment
add this:
{{- if .Values.global.tls.enabled }}
- name: PROXY_ADDRESS_FORWARDING
value: "true"
{{- end }}

[mjshashank]

@dbengtson I had just done that and it works. Thanks!

[ghost]

Closed as fixed.

[tony-engineering]

@dbengtson thanks for the hint, that worked

[myasas]

@mjshashank

In the -env section for the keycloak deployment
add this:
{{- if .Values.global.tls.enabled }}

• name: PROXY_ADDRESS_FORWARDING
  value: "true"
  {{- end }}

This is the fix for overcoming the issue !!!! (specially in a docker/kubernetes environment when you are using TLS in your ingress/route)

Thanks a lot @dbengtson !!!