Error "backend not available" when user logs-in for the first time (namespace provisioning returns error 500) #21958
Assignees
Labels
area/che-server
area/dashboard
kind/bug
Outline of a bug - must adhere to the bug report template.
new¬eworthy/che-only
Like 'new¬eworthy' but which do not apply to downstream (eg., plugins or devfiles)
new¬eworthy
For new and/or noteworthy issues that deserve a blog post, new docs, or emphasis in release notes
severity/blocker
Causes system to crash and be non-recoverable or prevents Che developers from working on Che code.
status/analyzing
An issue has been proposed and it is currently being analyzed for effort and implementation approach
Milestone
Comments
alfespa17
added
the
kind/bug
che-bot
added
the
status/need-triage
|
@alfespa17 hi, no need to downgrade the oauth-proxy to 7.2.0. It was temporary workaround. Please feel free to back oauth-proxy v 7.4.0 |
|
Please try to open Dashboard. In case any error, first of all, please have a look into ouath-proxy logs in che-gateway container. |
|
Hello @karatkep I change the oauth-proxy to 7.4.0, now I am getting the following error inside the oauth-proxy container in che-gateway pod Error: This is the api permitions. |
|
@alfespa17, it means your token does not contain |
|
@karatkep where should I put that parameter? inside any config map? Should I update any deployment? |
|
@alfespa17, from my point of view, the easiest way is via environment variables. You can update CheCluster Custom Resource to add proper configuration: |
|
@karatkep I have update the CheCluster Custom Resource and I am not facing the issue the oauth-proxy anymore I used this: spec:
networking:
auth:
identityProviderURL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
identityToken: access_token
oAuthClientName: {AZURE_APP_ID}
oAuthSecret: {AZURE_APP_SECRET}
oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
gateway:
deployment:
containers:
- name: oauth-proxy
env:
- name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
value: "true"
components:
cheServer:
extraProperties:
CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
CHE_OIDC_EMAIL__CLAIM: unique_name
I am able to see the swagger page again using the oauth-proxy 7.4.0
But I see the error "Backend in not available" when I open the dashboard page
Do you have any idea how to fix it?. Thank you for your help |
|
@alfespa17 could you please check oauth-proxy logs when you open Dashboard? |
|
@karatkep from the logs I can see a 500 error with this path "/api/kubernetes/namespace/provision" 10.244.0.13:40326 - 78991fa2b587eb40539d2a0258f81e3e - [email protected] [2023/01/26 20:01:17] che.XXXXXX.me GET / "/dashboard/client.e0f242511ea6888e4b5e.css" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 7273 0.007
10.244.0.13:40310 - f8045c4e99a4959bfddf039c46b4241f - [email protected] [2023/01/26 20:01:17] che.XXXXXX.me GET / "/dashboard/vendor.e0f242511ea6888e4b5e.css" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 819226 0.049
10.244.0.13:40326 - 9a05af9765382ebef4f277346ae3902a - [email protected] [2023/01/26 20:01:17] che.XXXXXX.me GET / "/dashboard/monaco.e0f242511ea6888e4b5e.css" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 140121 0.003
10.244.0.13:40300 - e8e674ef44984fc1285d6d5d210aacd6 - [email protected] [2023/01/26 20:01:17] che.XXXXXX.me GET / "/dashboard/editor.worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 1606 0.003
10.244.0.13:40292 - af0a64f350993899bdeea4e12bc7bc3f - [email protected] [2023/01/26 20:01:17] che.XXXXXX.me GET / "/dashboard/assets/branding/loader.svg" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 456 0.001
10.244.0.13:40310 - 4c5353d0326a5705e9c92276cd71394e - [email protected] [2023/01/26 20:01:18] che.XXXXXX.me GET / "/dashboard/assets/branding/favicon.ico" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 32988 0.006
10.244.0.13:40326 - aefb445b888abc9b49cb98fcf89a610d - [email protected] [2023/01/26 20:01:18] che.XXXXXX.me POST / "/api/kubernetes/namespace/provision" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 500 102 0.086
10.244.0.13:40326 - 52d8581aeaee51abff39d2b086c1d63e - [email protected] [2023/01/26 20:01:19] che.XXXXXX.me POST / "/api/kubernetes/namespace/provision" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 500 103 0.051
10.244.0.13:40326 - 313338fe52416b0561f8b46ec9ad558c - [email protected] [2023/01/26 20:01:19] che.XXXXXX.me GET / "/dashboard/service-worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" 200 63 0.002 |
|
@alfespa17 could you please check che pod logs? |
|
Che pods logs: Using custom assembly from /home/user/eclipse-che
Found a custom cert. Adding it to java trust store /home/user/cacerts based on /usr/lib/jvm/jre/lib/security/cacerts
Trust this certificate? [no]: Certificate was added to keystore
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
26-Jan-2023 19:43:23.439 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/10.0.14
26-Jan-2023 19:43:23.443 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Dec 2 2021 22:01:36 UTC
26-Jan-2023 19:43:23.443 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 10.0.14.0
26-Jan-2023 19:43:23.443 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.0-1100-azure
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-2.el8_7.x86_64
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.18+10-LTS
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Red Hat, Inc.
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /home/user/eclipse-che/tomcat
26-Jan-2023 19:43:23.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /home/user/eclipse-che/tomcat
26-Jan-2023 19:43:23.457 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
26-Jan-2023 19:43:23.457 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
26-Jan-2023 19:43:23.457 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
26-Jan-2023 19:43:23.457 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
26-Jan-2023 19:43:23.458 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
26-Jan-2023 19:43:23.458 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che/tomcat/conf/logging.properties
26-Jan-2023 19:43:23.458 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
26-Jan-2023 19:43:23.458 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=85.0
26-Jan-2023 19:43:23.520 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.redhat.fips=false
26-Jan-2023 19:43:23.520 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/user/cacerts
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/logs/
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
26-Jan-2023 19:43:23.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che/tomcat/conf/tomcat-logger.xml
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/home/user/eclipse-che/tomcat/conf/
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che/tomcat
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che/tomcat
26-Jan-2023 19:43:23.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che/tomcat/temp
26-Jan-2023 19:43:24.151 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
26-Jan-2023 19:43:24.236 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1204] milliseconds
26-Jan-2023 19:43:24.333 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
26-Jan-2023 19:43:24.333 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.0.14]
26-Jan-2023 19:43:24.353 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
26-Jan-2023 19:43:28.234 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [3,880] ms
26-Jan-2023 19:43:28.238 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
26-Jan-2023 19:43:28.472 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [233] ms
26-Jan-2023 19:43:28.473 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2023-01-26 19:43:36,764[main] [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63] - This=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory@487c412f obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.postgresql.PostgreSQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@ca2f6b7 environment={}
2023-01-26 19:43:37,462[main] [INFO ] [.e.c.a.d.WsMasterServletModule 52] - Running in native-user mode ...
2023-01-26 19:43:42,058[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace eclipse-che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-594b788bb8-8ltg8-51772, cluster=RemoteSubscriptionChannel, physical address=10.244.0.34:7800
-------------------------------------------------------------------
2023-01-26 19:43:45,435[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-594b788bb8-8ltg8-51772: no members discovered after 3297 ms: creating cluster as coordinator
2023-01-26 19:43:45,459[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace eclipse-che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-594b788bb8-8ltg8-1125, cluster=WorkspaceLocks, physical address=10.244.0.34:7801
-------------------------------------------------------------------
2023-01-26 19:43:48,541[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-594b788bb8-8ltg8-1125: no members discovered after 3075 ms: creating cluster as coordinator
2023-01-26 19:43:48,549[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace eclipse-che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-594b788bb8-8ltg8-50805, cluster=WorkspaceStateCache, physical address=10.244.0.34:7802
-------------------------------------------------------------------
2023-01-26 19:43:51,622[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-594b788bb8-8ltg8-50805: no members discovered after 3061 ms: creating cluster as coordinator
2023-01-26 19:43:51,625[main] [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 72] - Retrieving OpenId configuration from endpoint: https://sts.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/v2.0/.well-known/openid-configuration
2023-01-26 19:43:52,073[main] [INFO ] [o.e.c.m.oidc.OIDCInfoProvider 81] - openid configuration = {token_endpoint=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/oauth2/v2.0/token, token_endpoint_auth_methods_supported=[client_secret_post, private_key_jwt, client_secret_basic], jwks_uri=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/discovery/v2.0/keys, response_modes_supported=[query, fragment, form_post], subject_types_supported=[pairwise], id_token_signing_alg_values_supported=[RS256], response_types_supported=[code, id_token, code id_token, id_token token], scopes_supported=[openid, profile, email, offline_access], issuer=https://login.microsoftonline.com/59a1b398-5a8d-4060-b3e2-9d7b849527a8/v2.0, request_uri_parameter_supported=false, userinfo_endpoint=https://graph.microsoft.com/oidc/userinfo, authorization_endpoint=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/oauth2/v2.0/authorize, device_authorization_endpoint=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/oauth2/v2.0/devicecode, http_logout_supported=true, frontchannel_logout_supported=true, end_session_endpoint=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/oauth2/v2.0/logout, claims_supported=[sub, iss, cloud_instance_name, cloud_instance_host_name, cloud_graph_host_name, msgraph_host, aud, exp, iat, auth_time, acr, nonce, preferred_username, name, tid, ver, at_hash, c_hash, email], kerberos_endpoint=https://login.windows.net/59a1b398-5a8d-4060-b3e2-9d7b849527a8/kerberos, tenant_region_scope=NA, cloud_instance_name=microsoftonline.com, cloud_graph_host_name=graph.windows.net, msgraph_host=graph.microsoft.com, rbac_url=https://pas.windows.net}
2023-01-26 19:43:52,431[main] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2023-01-26 19:43:52,461[main] [INFO ] [o.f.c.i.util.VersionPrinter 44] - Flyway 4.2.0 by Boxfuse
2023-01-26 19:43:52,467[main] [INFO ] [o.f.c.i.d.DbSupportFactory 44] - Database: jdbc:postgresql://postgres:5432/dbche (PostgreSQL 13.3)
2023-01-26 19:43:52,534[main] [INFO ] [i.f.CustomSqlMigrationResolver 158] - Searching for SQL scripts in locations [classpath:che-schema]
2023-01-26 19:43:52,640[main] [INFO ] [o.f.c.i.command.DbValidate 44] - Successfully validated 63 migrations (execution time 00:00.108s)
2023-01-26 19:43:52,654[main] [INFO ] [o.f.c.i.m.MetaDataTableImpl 44] - Creating Metadata table: "public"."schema_version"
2023-01-26 19:43:52,725[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Current version of schema "public": << Empty Schema >>
2023-01-26 19:43:52,831[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.0.0.8.1 - 1__init.sql
2023-01-26 19:43:53,245[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.0.0.9.1 - 1__add_index_on_workspace_temporary.sql
2023-01-26 19:43:53,267[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.0.0.9.2 - 2__update_local_links_in_environments.sql
2023-01-26 19:43:53,281[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.2.0.1 - 1__increase_project_attributes_values_length.sql
2023-01-26 19:43:53,332[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.4.0.1 - 1__drop_user_to_account_relation.sql
2023-01-26 19:43:53,354[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.4.0.2 - 2__create_missed_account_indexes.sql
2023-01-26 19:43:53,380[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.6.0.1 - 1__add_exec_agent_where_terminal_agent_is_present.sql
2023-01-26 19:43:53,437[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.7.0.1 - 1__add_factory.sql
2023-01-26 19:43:53,571[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.7.0.2 - 2__remove_match_policy.sql
2023-01-26 19:43:53,583[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.8.0.1 - 1__add_foreigh_key_indexes.sql
2023-01-26 19:43:53,817[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.11.0.1 - 1__optimize_user_search.sql
2023-01-26 19:43:53,844[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.19.0.0.1 - 0.1__add_permissions.sql
2023-01-26 19:43:54,032[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.19.0.0.2 - 0.2__add_resources.sql
2023-01-26 19:43:54,071[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 5.19.0.0.3 - 0.3__add_organization.sql
2023-01-26 19:43:54,157[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.1 - 1__add_path_to_serverconf.sql
2023-01-26 19:43:54,164[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.2 - 2__rename_agents_to_installers.sql
2023-01-26 19:43:54,174[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.3 - 3__add_installer.sql
2023-01-26 19:43:54,249[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.3.1 - 3.1__remove_old_recipe_permissions.sql
2023-01-26 19:43:54,258[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.4 - 4__remove_old_recipe.sql
2023-01-26 19:43:54,270[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.5 - 5__add_machine_env.sql
2023-01-26 19:43:54,291[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.6 - 6__remove_snapshots.sql
2023-01-26 19:43:54,299[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.7 - 7__add_machine_volumes.sql
2023-01-26 19:43:54,326[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.8 - 8__add_serverconf_attributes.sql
2023-01-26 19:43:54,361[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.9 - 9__increase_externalmachine_env_value_length.sql
2023-01-26 19:43:54,368[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.10 - 10__move_dockerimage_recipe_location_to_content.sql
2023-01-26 19:43:54,374[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.11 - 11__increase_workspace_attributes_values_length.sql
2023-01-26 19:43:54,381[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.0.0.12 - 12__remove_stack_sources.sql
2023-01-26 19:43:54,388[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.3.0.1 - 1__add_fk_indexes.sql
2023-01-26 19:43:54,415[main] [WARN ] [o.f.c.i.dbsupport.JdbcTemplate 48] - DB: identifier "che_index_factory_on_projects_loaded_action_value_action_entity_id" will be truncated to "che_index_factory_on_projects_loaded_action_value_action_entity" (SQL State: 42622 - Error Code: 0)
2023-01-26 19:43:54,423[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.3.0.1.1 - 1.1__add_fk_indexes.sql
2023-01-26 19:43:54,475[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.4.0.1 - 1__add_workspace_expirations.sql
2023-01-26 19:43:54,499[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.4.0.2 - 2__add_signature_key.sql
2023-01-26 19:43:54,544[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.4.0.3 - 3__add_k8s_runtimes.sql
2023-01-26 19:43:54,630[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.10.0.1 - 1__add_workspace_cfg_attributes.sql
2023-01-26 19:43:54,651[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.10.0.2 - 2__change_signature_key_pair_id.sql
2023-01-26 19:43:54,669[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.11.0.1 - 1__add_signature_key_constraints.sql
2023-01-26 19:43:54,697[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.12.0.1 - 1__rename_project_attributes_values_field.sql
2023-01-26 19:43:54,704[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.15.0.1 - 1__remove_not_null_constraint_from_env_name_fields.sql
2023-01-26 19:43:54,718[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.15.0.2 - 2__add_commands_to_k8s_runtime.sql
2023-01-26 19:43:54,767[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.16.0.1 - 1__increase_workspace_config_attributes_values_length.sql
2023-01-26 19:43:54,777[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.16.0.2 - 2__create_workspace_activity_table.sql
2023-01-26 19:43:54,834[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.16.0.3 - 3__bootstrap_ws_activity_data.sql
2023-01-26 19:43:54,843[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 6.17.0.1 - 1__convert_enums_to_strings.sql
2023-01-26 19:43:54,855[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.4.0.1 - 1__add_devfile.sql
2023-01-26 19:43:55,130[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.5.0.1 - 1__devfile_command_reference.sql
2023-01-26 19:43:55,138[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.6.0.1 - 1__add_devfile_component_prefs.sql
2023-01-26 19:43:55,165[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.7.0.1 - 1__add_registry_url_to_devfile_component.sql
2023-01-26 19:43:55,172[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.8.0.2.0.1 - 1__devfile_metadata.sql
2023-01-26 19:43:55,179[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.0.0.8.0.2.0.2 - 2__devfile_make_some_fields_optional.sql
2023-01-26 19:43:55,187[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.1.0.1 - 1__change_devfile_component_preferences_type.sql
2023-01-26 19:43:55,227[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.1.0.1.1 - 1.1__remove_stack_permissions.sql
2023-01-26 19:43:55,237[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.1.0.2 - 2__remove_stacks.sql
2023-01-26 19:43:55,249[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.2.0.1 - 1__remove_installers.sql
2023-01-26 19:43:55,259[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.4.0.1 - 1__add_devfile_source_sparse_checkout_dir.sql
2023-01-26 19:43:55,266[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.4.0.2 - 2__add_preview_url_to_devfile_command.sql
2023-01-26 19:43:55,281[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.6.0.1 - 1__drop_che_workspace_expiration.sql
2023-01-26 19:43:55,322[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.10.0.1 - 1__add_devfile_plugin_editor_component_cpu_limit_request.sql
2023-01-26 19:43:55,333[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.10.0.2 - 2__add_devfile_plugin_editor_component_ram_request.sql
2023-01-26 19:43:55,340[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.11.0.1 - 1__update_inconsistent_stopped_workspace_activities.sql
2023-01-26 19:43:55,347[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.16.0.1 - 1__add_devfile_component_automount_workspace_secrets.sql
2023-01-26 19:43:55,354[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.20.0.1 - 1__userdevfile.sql
2023-01-26 19:43:55,427[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.20.0.1.1 - 1.1__add_userdevfile_permissions.sql
2023-01-26 19:43:55,483[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.21.0.1 - 1__remove_installers.sql
2023-01-26 19:43:55,491[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Migrating schema "public" to version 7.26.0.1 - 1__remove_factory_button_and_image.sql
2023-01-26 19:43:55,499[main] [INFO ] [o.f.c.i.command.DbMigrate 44] - Successfully applied 63 migrations to schema "public" (execution time 00:02.845s).
2023-01-26 19:43:57,260[main] [INFO ] [o.j.p.kubernetes.KUBE_PING 131] - namespace eclipse-che set; clustering enabled
-------------------------------------------------------------------
GMS: address=che-594b788bb8-8ltg8-22928, cluster=EclipseLinkCommandChannel, physical address=10.244.0.34:7803
-------------------------------------------------------------------
2023-01-26 19:44:00,368[main] [INFO ] [o.jgroups.protocols.pbcast.GMS 125] - che-594b788bb8-8ltg8-22928: no members discovered after 3041 ms: creating cluster as coordinator
2023-01-26 19:44:00,426[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 182] - Configured factories for environments: '[kubernetes, no-environment]'
2023-01-26 19:44:00,426[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 183] - Registered infrastructure 'kubernetes'
2023-01-26 19:44:00,468[main] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 694] - Infrastructure is tracking 0 active runtimes that need to be stopped
2023-01-26 19:44:00,538[main] [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36] - Eclipse Che Api Core: Build info '7.61.0-SNAPSHOT' scmRevision '7323f4776fd0aaf707f076b8baefa3f07837468b' implementationVersion '7.61.0-SNAPSHOT'
2023-01-26 19:44:00,549[main] [WARN ] [p.s.AdminPermissionInitializer 69] - Admin admin not found yet.
2023-01-26 19:44:00,560[main] [INFO ] [o.e.c.c.metrics.MetricsServer 46] - Metrics server started at port 8087 successfully
26-Jan-2023 19:44:01.129 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [32,655] ms
26-Jan-2023 19:44:01.132 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
26-Jan-2023 19:44:01.140 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [36903] milliseconds
|
|
@alfespa17 , don't see any error... So, I can only guessing... Let's try to add one more hack to your config: |
|
Thank you @karatkep, I added the parameter and install che using the following: spec:
networking:
auth:
identityProviderURL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
identityToken: access_token
oAuthClientName: {AZURE_APP_ID}
oAuthSecret: {AZURE_APP_SECRET}
oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
gateway:
deployment:
containers:
- name: oauth-proxy
env:
- name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
value: "true"
components:
cheServer:
extraProperties:
CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
CHE_OIDC_EMAIL__CLAIM: unique_name
CHE_OIDC_USERNAME__CLAIM: unique_nameI checked every pod in the eclipse-che namespace and I dont see any error I just see the same 500 error in the dashboard in the path https://che.XXXXX.me/api/kubernetes/namespace/provision by the way the response is the following: |
|
@alfespa17 , could you please check image version for che pod? |
|
This is what I see from the che pod spec
|
|
@alfespa17 , I got the same issue few days ago... so, I switched to stable channel (7.59) and re-installed che. It helped me |
|
@karatkep thank you for all your help. I was able to make it work.
The CleCluster spec is the following: spec:
networking:
auth:
identityProviderURL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
identityToken: access_token
oAuthClientName: {AZURE_APP_ID}
oAuthSecret: {AZURE_APP_SECRET}
oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
gateway:
deployment:
containers:
- name: oauth-proxy
env:
- name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
value: "true"
components:
cheServer:
extraProperties:
CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{AZURE_TENANT_ID}/v2.0/
CHE_OIDC_EMAIL__CLAIM: unique_name
CHE_OIDC_USERNAME__CLAIM: unique_nameI used the following configuration: chectl --version
chectl/7.60.0 win32-x64 node-v16.13.2Azure Kubernetes Service: 1.24.6 with Azure AD authentication with Azure RBAC After the installation was completed I had to provision a namespace for my user using the following reference Create a file with the following name: "workspace.yaml" kind: Namespace
apiVersion: v1
metadata:
name: cheuser
labels:
app.kubernetes.io/part-of: che.eclipse.org
app.kubernetes.io/component: workspaces-namespace
annotations:
che.eclipse.org/username: [email protected]kubectl apply -f workspace.yamlI am not really sure if creating a namespace for my user is a normal flow but it make it works. |
|
@alfespa17, my congrats! P.S. I remember it took several days (even more - about week) to deep dive into che world before make it workable for me. |
@alfespa17, Che should automatically create the namespace for you. So that's a nasty bug introduced recently. The workaround is indeed to pre-create the namespace. Pre-creating namespaces is also recommended in production clusters where admins typically want to fine tune developers namespaces for security and resource limits. The bug is not related to Azure (I can reproduce it on AWS) but it's a namespace provisioning bug. I am changing the title. |
l0rd
changed the title
l0rd
removed
the
status/need-triage
|
@alfespa17 thank you for reporting this anyway and congrats for finding a workaround |
|
@l0rd, just want to confirm that Che 7.59 creates the namespaces automatically. Not sure about Che 7.60 - I did not try it yet. |
|
Che server logs after switching to DEBUG log level: |
l0rd
added
area/che-server
area/dashboard
severity/blocker
|
Setting the severity as blocker because default Che installation is broken: developers cannot start workspaces. |
|
I observed the same issue for Kubernetes: |
ibuziuk
added
the
status/analyzing
|
I also encountered this issue yesterday evening, deploying Che |
I haven't mentioned but yes, I have experienced the same: creating a "regular" OpenShift user using an htpasswd secret as identity provider fixed the problem without the need to pre-create the namespace. |
10 tasks
|
the culprit is related to the removal of |
|
Hi @ibuziuk , is there any ETA for this fix? |
9 tasks
|
@karatkep PR has been sent - eclipse-che/che-server#431 |
|
merged in main and backported to 7.60.x - eclipse-che/che-server#432 |
|
Closing, the fixes are in main and 7.60.x and we plan to proceed with 7.60.1 bug fix release this week cc: @nickboldt @SDawley |
l0rd
added
new¬eworthy
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Dashboard is not working after installing inside a new AKS cluster
Che version
7.59
Steps to reproduce
1- Install in AKS following https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html
2- Downgrade the oauth-proxy to 7.2.0 as mentioned in #21886
3- Open the dasboard page and get error: "Backend in not available. Try to refresh the page or re-login to the Dashboard."
Expected behavior
Open the eclipse che dashboard successfully
Runtime
other (please specify in additional context)
Screenshots
I follow this tutorial to install eclipse che using AKS
https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html
I also follow the instructions in this issue #21886 to downgrade the oauth-proxy to 7.2.0
I was able to install it without any issue but when I tried to get into the dashboard I get the following error:
I am able to see the swagger page
All pods are running
Installation method
chectl/latest
Environment
Azure
Additional context
Azure Kubernetes Version:
Release Notes Text
Prior to this fix the automatic creation of a developer namespace in Eclipse Che 7.60 was failing when the developer login name contained invalid characters for the RFC 3986 such as
@or:. As a consequence, developers with those login names, would not be able to access to Eclipse Che. This in now fixed and the namespaces are created successfully even if the developer login name had invalid characters for RFC 3986.The text was updated successfully, but these errors were encountered: