Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open OnDemand Internal Server Error #184

Closed
james-s-willis opened this issue Jul 25, 2024 · 8 comments
Closed

Open OnDemand Internal Server Error #184

james-s-willis opened this issue Jul 25, 2024 · 8 comments

Comments

@james-s-willis
Copy link

When trying to open the OnDemand URL: https://localhost:3443 in a browser I get the following error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

The error appears in the ondemand logs (docker compose logs -f ondemand) as:

ondemand  | Completed successfully!
ondemand  | ---> Starting ondemand-dex...
ondemand  | ---> Starting ondemand httpd24...
ondemand  | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.19.0.12. Set the 'ServerName' directive globally to suppress this message
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="Dex Version: v2.36.0, Go Version: go1.19.2, Go OS/ARCH: linux amd64"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config issuer: https://localhost:5554"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config storage: sqlite3"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config static client: OnDemand"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config connector: ldap"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config skipping approval screen"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="config refresh tokens rotation enabled: true"
ondemand  | time="2024-07-25T14:22:47Z" level=info msg="keys expired, rotating"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="keys rotated, next rotation: 2024-07-25 20:22:47.998753652 +0000 UTC"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (telemetry) on 0.0.0.0:5558"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (http) on 0.0.0.0:5556"
ondemand  | time="2024-07-25T14:22:48Z" level=info msg="listening (https) on 0.0.0.0:5554"
ondemand  | 2024/07/25 14:23:58 http: TLS handshake error from [::1]:52182: local error: tls: bad record MAC
ondemand  | 2024/07/25 14:39:17 http: TLS handshake error from [::1]:51144: local error: tls: bad record MAC
ondemand  | 2024/07/25 14:42:10 http: TLS handshake error from [::1]:40056: local error: tls: bad record MAC

I'm running the containers in Pop OS 20.04. XDMoD and Coldfront interfaces work correctly. The Open OnDemand interface had previously worked as recent as 2 weeks ago. I have tried running ./hpcts destroy and ./hpcts cleanup as well as cloning a new repository and running the setup again. My colleague can also reproduce the error.

The fact that it had previously worked leads me to believe it could be an expired certificate maybe?
@johrstrom
Copy link
Contributor

The fact that it had previously worked leads me to believe it could be an expired certificate maybe?

It is exactly an expired certificate. IDK if they're publishing new images or not, but if you build the images it should work. That is, what's on Dockerhub is bad, but rebuilding locally with docker compose build should work (on x86 chips at least. If you have a Mac M1 I think there's still a patch that this repo needs).

@vanzonr
Copy link

vanzonr commented Jul 25, 2024

I am seeing the same error in Ubuntu 22.04.4.
I noticed that docker exec -it ondemand openssl x509 -enddate -noout -in /etc/pki/tls/certs/localhost.crt gives an expiration date of 'Jul 13 12:27:14 2024 GMT'.

@johrstrom
Copy link
Contributor

Yea the images need to be rebuilt. You can do so locally and it should work. But again, it'll work for x86 machines. If you're on a Mac M1 there's some issue in building that we haven't patched yet.

@james-s-willis
Copy link
Author

Thanks! Rebuilding the images locally worked. However, some functionality fails. When trying to access the Job Composer tab I get:

Exception: OodApp::SetupScriptFailed

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user hpcadmin with output: /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:480:in `materialize': Could not find rdoc-6.3.3 in any of the sources (Bundler::GemNotFound)
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:185:in `specs'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/definition.rb:233:in `specs_for'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/runtime.rb:18:in `setup'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:150:in `setup'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `block in <top (required)>'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:136:in `with_level'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/ui/shell.rb:88:in `silence'
	from /usr/share/gems/gems/bundler-2.2.33/lib/bundler/setup.rb:20:in `<top (required)>'
	from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
<internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- bundler/setup (LoadError)
	from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'

Stack trace:

/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:271:in `block (2 levels) in run_setup_production'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `chdir'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `block in run_setup_production'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `block in with_unbundled_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:692:in `with_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `with_unbundled_env'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:254:in `run_setup_production'
/var/www/ood/apps/sys/dashboard/app/controllers/apps_controller.rb:35:in `show'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/lograge-0.12.0/lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

Any ideas what could be causing this?

@johrstrom
Copy link
Contributor

Sorry about that! I see that appear every now and again on some systems for whatever reason.

I found that I could exec into the container and install the gem and then it would work.

docker exec -it ondemand /bin/bash
[root@ondemand /]# source /opt/ood/ondemand/enable 
[root@ondemand /]# gem install rdoc -v 6.3.3

@james-s-willis
Copy link
Author

Thanks! That fix worked!

@JAM2199562
Copy link

JAM2199562 commented Aug 29, 2024 •
edited
Loading

For those who still don't know how to solve this problem

cd hpc-toolset-tutorial
docker compose build --no-cache base
docker compose build

Additionally, you may encounter websockify compilation failures

add a new line
sed -i 's/numpy/numpy==1.26/' setup.py

before

hpc-toolset-tutorial/slurm/install.sh

Line 53 in 6baa9a3

python3 setup.py install

@mw201608
Copy link

Can we re-open this issue? JAM2199562's approach does not work for me and the certificate does not update after re-build. Am I missing anything here? Thanks.

@mw201608 mw201608 mentioned this issue Oct 11, 2024
Closed
@PaulBreugnot PaulBreugnot mentioned this issue Jan 29, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vanzonr @johrstrom @JAM2199562 @mw201608 @james-s-willis