Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Setting Harvester NTP Servers Is Only Additive #5876

Closed
irishgordo opened this issue May 24, 2024 · 5 comments
Closed

[BUG] Setting Harvester NTP Servers Is Only Additive #5876

irishgordo opened this issue May 24, 2024 · 5 comments
Assignees
@brandboat @irishgordo
Labels
area/api Harvester API server backport-needed/1.2.3 backport-needed/1.3.2 kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/always Reproducible 100% of the time require/investigate Identified the issue but require further investigation for resolution (won't be stale) severity/3 Function working but has a major issue w/ workaround
Milestone
v1.4.0

Comments

@irishgordo
Copy link
Contributor

irishgordo commented May 24, 2024 •
edited
Loading

Describe the bug
Harvester Dashboard -> Settings -> NTP Servers do not accurately update '/etc/systemd/timesyncd.conf`.

To Reproduce
Steps to reproduce the behavior:

  1. Set up a single-node Harvester cluster v1.3-head
  2. Ensure that you only implement 1 NTP Server at the Cluster Creation
  3. Once Harvester is installed, check that the single NTP server is reflected on the node via cat /etc/systemd/timesyncd.conf , you should see your single NTP server there and present
  4. In Harvester Dashboard -> Settings -> NTP Servers, modify that, removing your "old" ntp-server and then placing in a "new" ntp-server that is different than the original, for instance if your original NTP Server was "time1.google.com", change that to something like "time2.google.com" or really whatever you desire
  5. Save those changes
  6. Wait 20 minutes or less, (I waited 20), then on the Harvester node, again, audit the timesyncd.conf with cat /etc/systemd/timesyncd.conf you'll notice, that it has only "added" the other server that it is not "truly" reflective of what we set it in the Harvester settings.

Example.

Original NTP Server:

harvester-node-0:/home/rancher # cat /etc/systemd/timesyncd.conf

[Time]
NTP = time1.google.com

After Modification (if let's say you modified the ntp-server in Harvester Dashboard -> Settings -> NTP Servers, from time1.google.com to replacing that with time2.google.com):

harvester-node-0:/home/rancher # cat /etc/systemd/timesyncd.conf

[Time]
NTP = time1.google.com time2.google.com

Workaround

  • Manually modify the /etc/systemd/timesyncd.conf and remove the "legacy"/"old" NTP server from it's configuration, then additionally restart systemd-timesyncd.service

Expected behavior
Expectation would be that we would have parity reflected with /etc/systemd/timesync.conf - but it seems to only be "additive" in nature, and would not remove the original ntp-server from the /etc/systemd/timesyncd.conf even if in our Harvester Settings -> NTP Servers that server is nowhere to be found

Environment

  • Harvester ISO version: v1.3-head
  • Underlying Infrastructure: qemu/kvm
@irishgordo irishgordo added kind/bug Issues that are defects reported by users or that we know have reached a real release severity/3 Function working but has a major issue w/ workaround area/api Harvester API server reproduce/always Reproducible 100% of the time labels May 24, 2024
@irishgordo irishgordo mentioned this issue May 24, 2024
Closed
@bk201 bk201 added this to the v1.4.0 milestone May 25, 2024
@bk201 bk201 added the require/investigate Identified the issue but require further investigation for resolution (won't be stale) label May 25, 2024
@brandboat brandboat mentioned this issue May 27, 2024
Merged
@harvesterhci-io-github-bot
Copy link
Collaborator

harvesterhci-io-github-bot commented May 28, 2024 •
edited by brandboat
Loading

Pre Ready-For-Testing Checklist

  • If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
    The HEP PR is at:

  • Where is the reproduce steps/test steps documented?
    Reproduce steps are described in the issue description.
    The test steps are as follows:

    • Set up a single-node Harvester cluster based on master branch.
    • Dashboard -> Settings -> NTP Servers, remove 0.suse.pool.ntp.org 1.suse.pool.ntp.org and add time1.google.com and time2.google.com
    • Wait a couple seconds, go to each Harvester node and check cat /etc/systemd/timesyncd.conf. The content should be time1.google.com and time2.google.com
    • Run systemctl status systemd-timesyncd.service and check service status is active and time server is sync from 216.239.35.0:123 (time1.google.com)
    ● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-05-27 09:41:14 UTC; 20s ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 4053 (systemd-timesyn)
     Status: "Initial synchronization to time server 216.239.35.0:123 (time1.google.com)."
      Tasks: 2
     CGroup: /system.slice/systemd-timesyncd.service
             └─ 4053 /usr/lib/systemd/systemd-timesyncd

May 27 09:41:14 harvester-node-0 systemd[1]: Starting Network Time Synchronization...
May 27 09:41:14 harvester-node-0 systemd[1]: Started Network Time Synchronization.
May 27 09:41:15 harvester-node-0 systemd-timesyncd[4053]: Initial synchronization to time server 216.239.35.0:123 (time1.google.com).

  • Is there a workaround for the issue? If so, where is it documented?
    The workaround is at:

  • Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
    The PR is at:

    • Does the PR include the explanation for the fix or the feature?

    • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
      The PR for the YAML change is at:
      The PR for the chart change is at:

  • If labeled: area/ui Has the UI issue filed or ready to be merged?
    The UI issue/PR is at:

  • If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
    The documentation/KB PR is at:

  • If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

    • The automation skeleton PR is at:
    • The automation test case PR is at:

  • If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
    The compatibility issue is filed at:

@harvesterhci-io-github-bot
Copy link
Collaborator

Automation e2e test issue: harvester/tests#1295

Open
1 task
Closed
@harvesterhci-io-github-bot
Copy link
Collaborator

added backport-needed/1.3.2 issue: #5892.

Closed
@harvesterhci-io-github-bot
Copy link
Collaborator

added backport-needed/1.2.3 issue: #5893.

This was referenced Jul 2, 2024
Merged
Merged
@irishgordo irishgordo self-assigned this Jul 3, 2024
@irishgordo
Copy link
Contributor Author

Thanks for this fix @brandboat 😄 👍 - this looks great on Harvester Version: v1.4.0-dev-20240705

It is correctly updating the timesyncd.conf and additionally re-initializing the synchronization to the timeserver as updated / needed / etc.

rancher@testnode8:~> cat /etc/systemd/timesyncd.conf

[Time]
NTP = time2.google.com
rancher@testnode8:~> systemctl status ntp.service
Failed to get properties: Access denied
rancher@testnode8:~> cat /etc/systemd/timesyncd.conf

[Time]
NTP = time2.google.com
rancher@testnode8:~> cat /etc/systemd/timesyncd.conf

[Time]
NTP = time2.google.com time3.google.com time1.google.com
rancher@testnode8:~> cat /etc/systemd/timesyncd.conf

[Time]
NTP = time2.google.com time3.google.com time1.google.com
rancher@testnode8:~> sudo su
testnode8:/home/rancher # systemctl status systemd-time
systemd-time-wait-sync.service  systemd-timedated.service       systemd-timesyncd.service       
testnode8:/home/rancher # systemctl status systemd-timedated.service 
○ systemd-timedated.service - Time & Date Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-timedated.service; static)
     Active: inactive (dead)
       Docs: man:systemd-timedated.service(8)
             man:localtime(5)
             man:org.freedesktop.timedate1(5)

Jul 09 01:49:59 testnode8 systemd[1]: systemd-timedated.service: Deactivated successfully.
Jul 09 01:53:44 testnode8 systemd[1]: Starting Time & Date Service...
Jul 09 01:53:45 testnode8 systemd[1]: Started Time & Date Service.
Jul 09 01:54:15 testnode8 systemd[1]: systemd-timedated.service: Deactivated successfully.
Jul 09 01:57:58 testnode8 systemd[1]: Starting Time & Date Service...
Jul 09 01:57:59 testnode8 systemd[1]: Started Time & Date Service.
Jul 09 01:58:29 testnode8 systemd[1]: systemd-timedated.service: Deactivated successfully.
Jul 09 01:59:02 testnode8 systemd[1]: Starting Time & Date Service...
Jul 09 01:59:03 testnode8 systemd[1]: Started Time & Date Service.
Jul 09 01:59:52 testnode8 systemd[1]: systemd-timedated.service: Deactivated successfully.
testnode8:/home/rancher # systemctl status system-timedated.service
Unit system-timedated.service could not be found.
testnode8:/home/rancher # systemctl status systemd-time
systemd-time-wait-sync.service  systemd-timedated.service       systemd-timesyncd.service       
testnode8:/home/rancher # systemctl status systemd-timesyncd.service 
● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-07-09 01:59:22 UTC; 1min 6s ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 14481 (systemd-timesyn)
     Status: "Initial synchronization to time server 216.239.35.4:123 (time2.google.com)."
      Tasks: 2
     CGroup: /system.slice/systemd-timesyncd.service
             └─ 14481 /usr/lib/systemd/systemd-timesyncd

Jul 09 01:59:21 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:59:22 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:59:22 testnode8 systemd-timesyncd[14481]: Initial synchronization to time server 216.239.35.4:123 (time2.google.com).
testnode8:/home/rancher # journalctl -u systemd-timesyncd.service
Jul 09 01:39:35 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:39:35 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:40:06 testnode8 systemd-timesyncd[2106]: Initial synchronization to time server 216.239.35.0:123 (time1.google.com).
Jul 09 01:46:14 testnode8 systemd[1]: Stopping Network Time Synchronization...
Jul 09 01:46:14 testnode8 systemd[1]: systemd-timesyncd.service: Deactivated successfully.
Jul 09 01:46:14 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 01:46:14 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:46:15 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:46:15 testnode8 systemd-timesyncd[4139]: Initial synchronization to time server 216.239.35.0:123 (time1.google.com).
Jul 09 01:57:57 testnode8 systemd[1]: Stopping Network Time Synchronization...
Jul 09 01:57:57 testnode8 systemd[1]: systemd-timesyncd.service: Deactivated successfully.
Jul 09 01:57:57 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 01:57:57 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:57:58 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:57:58 testnode8 systemd-timesyncd[11232]: Initial synchronization to time server 216.239.35.4:123 (time2.google.com).
Jul 09 01:59:21 testnode8 systemd[1]: Stopping Network Time Synchronization...
Jul 09 01:59:21 testnode8 systemd[1]: systemd-timesyncd.service: Deactivated successfully.
Jul 09 01:59:21 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 01:59:21 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:59:22 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:59:22 testnode8 systemd-timesyncd[14481]: Initial synchronization to time server 216.239.35.4:123 (time2.google.com).
testnode8:/home/rancher # journalctl -u systemd-timesyncd.service --follow
Jul 09 01:57:57 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 01:57:57 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:57:58 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:57:58 testnode8 systemd-timesyncd[11232]: Initial synchronization to time server 216.239.35.4:123 (time2.google.com).
Jul 09 01:59:21 testnode8 systemd[1]: Stopping Network Time Synchronization...
Jul 09 01:59:21 testnode8 systemd[1]: systemd-timesyncd.service: Deactivated successfully.
Jul 09 01:59:21 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 01:59:21 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 01:59:22 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 01:59:22 testnode8 systemd-timesyncd[14481]: Initial synchronization to time server 216.239.35.4:123 (time2.google.com).
Jul 09 02:01:32 testnode8 systemd[1]: Stopping Network Time Synchronization...
Jul 09 02:01:32 testnode8 systemd[1]: systemd-timesyncd.service: Deactivated successfully.
Jul 09 02:01:32 testnode8 systemd[1]: Stopped Network Time Synchronization.
Jul 09 02:01:32 testnode8 systemd[1]: Starting Network Time Synchronization...
Jul 09 02:01:33 testnode8 systemd[1]: Started Network Time Synchronization.
Jul 09 02:01:33 testnode8 systemd-timesyncd[19919]: Initial synchronization to time server 216.239.35.0:123 (time1.google.com).
^C
testnode8:/home/rancher # cat /etc/systemd/timesyncd.conf

[Time]
NTP = time1.google.com

This looks great 😄 - I'll go ahead and close this out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandboat brandboat

@irishgordo irishgordo

Labels
area/api Harvester API server backport-needed/1.2.3 backport-needed/1.3.2 kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/always Reproducible 100% of the time require/investigate Identified the issue but require further investigation for resolution (won't be stale) severity/3 Function working but has a major issue w/ workaround
Projects
None yet
Milestone
v1.4.0
Development

No branches or pull requests
4 participants
@bk201 @brandboat @irishgordo @harvesterhci-io-github-bot