Dask-gateway error on deploy #479
Comments
|
That'd be from #477, sorry. Do you have dask-gateway config values in It'd be something like pangeo:
jupyterhub:
hub:
services:
dask-gateway:
apiToken: "<token1>"
dask-gateway:
gateway:
proxyToken: "<token2>"
auth:
type: jupyterhub
jupyterhub:
apiToken: "<token1>"Note that @jhamman may want a way for deployments to opt out of dask-gateway. Presumably helm has a way to do that. |
|
Thanks @TomAugspurger. Definitely moving us forward. Here is a new error: I can delete this Secret, but it would be good if it was able to deal with this. My sense is that these might not have been tested enough before they were deployed. I guess we are testing it now. |
I'm not sure where that original secret would have come from. Perhaps from the failed deployment? I don't think it'd be inheriting from a common config. I'd try deleting it now, adding a dummy commit, and redeploying. |
|
Yes, failed deployment. A helm upgrade on Azure can take a long time, I believe because of the length of time it can take to download a new image. So I sometimes need to run a workflow again after the first run finishes the image download to the cluster machines. I will figure this out. Thanks for your help and happy thanksgiving! |
|
FWIW, I had a similar problem yesterday on the namespace="dev-staging"
kubectl delete serviceaccount ${namespace}-dask-gateway -n ${namespace}
kubectl delete secret ${namespace}-dask-gateway -n ${namespace}
kubectl delete configmap ${namespace}-dask-gateway -n ${namespace}
kubectl delete rolebinding ${namespace}-dask-gateway -n ${namespace}
kubectl delete role ${namespace}-dask-gateway -n ${namespace}
kubectl delete service ${namespace}-dask-gateway -n ${namespace}
kubectl delete service gateway-api-${namespace}-dask-gateway -n ${namespace}
kubectl delete service scheduler-api-${namespace}-dask-gateway -n ${namespace}
kubectl delete service scheduler-public-${namespace}-dask-gateway -n ${namespace}
kubectl delete service web-public-${namespace}-dask-gateway -n ${namespace}
kubectl delete service web-api-${namespace}-dask-gateway -n ${namespace}
kubectl delete deployment gateway-${namespace}-dask-gateway -n ${namespace}
kubectl delete deployment scheduler-proxy-${namespace}-dask-gateway -n ${namespace}
kubectl delete deployment web-proxy-${namespace}-dask-gateway -n ${namespace}Seeing that you ran into this too, I wonder if dask-gateway could do a better job of cleaning up after a failed deployment. |
|
It's worth a lot (IWAL), @jhamman. Thank you! I will try this out after turkey tomorrow. Or, maybe Friday. Thank you very much for your help. Cheers. |
This is a secret created as part of the dask-gateway helm chart. Helm should be able to automatically delete resources from previous deployments provided they have the same name as the current deployment (which the do here). Our helm chart is fairly close to that of JupyterHub's, so I don't think this is a dask-gateway specific issue. I'm not sure what's going on, perhaps @yuvipanda would have an idea?
Yeah, we can do that with a conditional dependency. See https://helm.sh/docs/topics/charts/#tags-and-condition-fields-in-dependencies. |
Let's move this topic to the Pangeo Helm Chart repo. I personally don't think this is something we want to do. Once the daskkubernetes service account is gone, turning off dask-gateway would give you a vanilla zero-to-jupyterhub deployment. |
I just remembered that I fixed a helm chart bug, but hadn't done a new release until recently (0.6.1). The issue you're seeing here may be due to dask/dask-gateway#150, which has since been fixed. After upgrading the dask-gateway chart dependency to >= 0.6.0 things like this should hopefully not happen (the issue was a label that the official helm docs mistakenly recommended). (Note that until you have a version >= 0.6.0 running, version upgrades of dask-gateway won't run smoothly due to this issue - the upgrade failures are due to the currently running chart, not the new chart). |
|
I'm not sure we ever moved past the original problem in this issue but we have a new one: @jcrist - have you seen this error ( |
|
If you scroll to the right you can see that it's indicating that a label selector is immutable. This should have been fixed by dask/dask-gateway#150 (see the issue for details on why this was). I assume you're upgrading from 0.5.0 to 0.6.* here? If you're already on 0.6.* then this is something new. If you're upgrading from 0.5.0 to 0.6.* you'll have to delete the old deployment and reinstall, after that upgrades between versions should run smoothly. Edit: see also my comment above yours, which is about the same issue |
Getting a new error when trying to deploy the OOI deployment:
What is the gateway.proxyToken, and why isn't mine 32 bytes?
The text was updated successfully, but these errors were encountered: