Scheduled jobs; Hundreds of duplicate jobs scheduled somehow overnight #16042
Comments
bitcollector1
added
status: needs triage
|
I just deleted all scheduled jobs a few min ago and already these are all back! 
Everything was working fine last night, woke up this morning to this issue and I can't seem to make it stop :( |
|
had to remove the scripts and restart netbox to make the madness stop! The scripts just kept rescheduling themselves and running over and over no matter how many times I deleted them! |
|
Not sure if this is related to the timezone bug that I also submitted but the issue did not seem to present itself until I woke up this morning; before that the scheduled jobs would run once every 15 min as desired. I've been using these scripts for months without any issues. |
|
I just added the script back and rescheduled the job for every 15 min. I will monitor and report back if/when it goes off the rails. 
FYI: I entered this job to run at 8:30am PDT and it scheduled it for UTC time. |
arthanson
added
status: under review
|
Timezone issue is #15984 |
|
Script just completed first iteration successfully but now a second instance has been scheduled. 
|
|
FYI, I do have multiple workers defined in my setup.....not sure if this is coming into play here.....but the multiple workers were also setup under 3.7.x and working well. |
|
Thinking back......I believe I updated this setting from False to True yesterday afternoon while troubleshooting timezone stuff. I'm going to try setting it back to False. LocalizationENABLE_LOCALIZATION = True |
|
Okay, so after changing the localization setting back to false, it looks like the scheduling might be working correctly now The script completed the scheduled job as desired and no additional jobs were scheduled this time. 
|
bitcollector1
changed the title
|
Scheduled jobs are working as expected since I changed the Localization setting to False. |
|
I'm not sure what's going on here. The |
|
hmmmm.....I'm not sure myself :( let me try to enable it again and see if I can recreate again. I can also try going back to just the (1) worker and see if this somehow contributing to this issue. This was all working fine on 3.7.x |
|
@jeremystretch I think this issue was related to the multiple workers possibly. It turns out it looks like I have the default service running.....in ADDITION to the new workers I created :( |
|
Looks like this may have been the issue all along.....sorry for wasting your time on this. Now I only have the (4) workers that I setup running. 
|
|
Huh, that's really strange. Even having multiple workers shouldn't result in an issue like this. Well, glad you got it sorted. And thanks for following up with the resolution! |
|
Hopefully the issue does not pop up again. I will let you folks know if I bump into it again. Thanks again for all you do! |
|
FYI, the backup plugin uses multiple rq workers heavily and this shouldn't normally result in multiple jobs firing off. There must be something else with the way those workers are configured that is causing issues. Perhaps it is related to localization as far as start time and workers aren't properly starting "on-time" (to early or to late) |
|
Happy to say this issue seems to be a one off related to the migration and the multiple workers. I'm really not sure but I do appreciate you folks offering insights. Thanks again for supporting an amazing product and sorry to add to your burden. Cheers! |
Deployment Type
Self-hosted
NetBox Version
v4.0.0
Python Version
3.12
Steps to Reproduce
Schedule a job and wait until the next day and all of a sudden I have hundreds (880) of the same jobs scheduled and tons of script failures!
Expected Behavior
Script runs as scheduled under NetBox 3.7.x
Observed Behavior
Duplicate jobs scheduled resulting in failure of the scripts to complete. This seems to be a loop now, every time I delete the scheduled jobs they come right back :(
The text was updated successfully, but these errors were encountered: