Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include maintenance node in HTCondor playbook #1057

Merged
merged 4 commits into from
Dec 21, 2023
Merged

Include maintenance node in HTCondor playbook #1057

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a536336
Add maintenance node to the HTCondor playbook
kysrpex Dec 19, 2023
4e37f10
Remove `usegalaxy_eu.htcondor` role from the maintenance playbook
kysrpex Dec 19, 2023
2c57acb
Remove vars specific to `usegalaxy_eu.htcondor` role from maintenance…
kysrpex Dec 19, 2023
3881bf8
Do not run job management roles on the maintenance node
kysrpex Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions group_vars/maintenance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,26 +288,6 @@ dynmotd_custom:
- name: Condor
command: "condor_q -totals | tail -n 2"

# Role: usegalaxy_eu.htcondor
condor_host: "condor-cm.galaxyproject.eu"
condor_fs_domain: bi.uni-freiburg.de
condor_uid_domain: bi.uni-freiburg.de
condor_allow_write: "10.5.68.0/24, 132.230.223.0/24"
condor_daemons:
- MASTER
- SCHEDD
condor_allow_negotiator: "{{ ansible_default_ipv4.address | default(ansible_all_ipv4_addresses[0]) }},$(CONDOR_HOST)"
condor_allow_administrator: "$(ALLOW_NEGOTIATOR)"
condor_system_periodic_hold: "{{ 30 * 24 * 60 * 60 }}"
condor_system_periodic_remove: "{{ 2 * 24 * 60 * 60 }}"
condor_network_interface: "{{ ansible_default_ipv4.interface }}"
condor_extra: |
MASTER_UPDATE_INTERVAL = 150
SCHEDD_INTERVAL = 60
JOB_START_COUNT = 250
JOB_START_DELAY = 0
CLAIM_WORKLIFE = 120

# Role: usegalaxy-eu.galaxy-slurp
galaxy_slurper: galaxy
galaxy_slurp_influx_pass: "{{ influxdb.node.password }}"
Expand Down
1 change: 1 addition & 0 deletions hosts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ ansible_group_priority=4

[htcondor-secondary-submit]
nspawn-htcondor.sn06.galaxyproject.eu ansible_host=127.0.0.1 ansible_port=2222 ansible_ssh_user=root ansible_ssh_common_args='-o HostKeyAlias=nspawn-htcondor.sn06.galaxyproject.eu -o ProxyCommand="ssh -W %h:%p -q [email protected]"'
maintenance.galaxyproject.eu

[htcondor-secondary-submit:vars]
ansible_group_priority=4
Expand Down
6 changes: 3 additions & 3 deletions htcondor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,11 +294,11 @@
when: htcondor_role_submit
- grycap.htcondor
- name: usegalaxy-eu.htcondor_release
when: htcondor_role_submit
when: htcondor_role_submit and inventory_hostname != "maintenance.galaxyproject.eu"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this will be removed again once the nspawn container stops doing that?
(just for my better understanding)

Copy link
Contributor Author

@kysrpex kysrpex Dec 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem here is that usegalaxy-eu.htcondor_release is designed to run on max one submitter node (this is of course a bad solution and needs to be addressed later, but we want to have the monitoring working properly too). If we do not prevent usegalaxy-eu.htcondor_release from running on more than one submitter node, then #896 comes back.

So unfortunately no, this will not be removed again once the systemd-nspawn container is decommissioned. It will not be as simple as that :(

- name: usegalaxy-eu.fix-stop-ITs
when: htcondor_role_submit
when: htcondor_role_submit and inventory_hostname != "maintenance.galaxyproject.eu"
- name: usegalaxy-eu.remove-orphan-condor-jobs
when: htcondor_role_submit
when: htcondor_role_submit and inventory_hostname != "maintenance.galaxyproject.eu"
post_tasks:
- name: Add /usr/local/bin to Galaxy's PATH in bashrc file. (usegalaxy-eu.fix-stop-ITs)
when: htcondor_role_submit
Expand Down
1 change: 0 additions & 1 deletion maintenance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,6 @@
- hxr.monitor-galaxy
- usegalaxy-eu.monitoring
- usegalaxy-eu.bashrc
- usegalaxy_eu.htcondor
- usegalaxy-eu.dynmotd
- ssh-host-sign
- hxr.postgres-connection
Expand Down
Loading