Core is unable to download Spark sources

[zipiju]

This started happening on multiple nodes on Monday 13th, on both containerised and Node.js ones at a different locations, with no configuration changes.
Core is unable to download Spark updates, this module thus won't update and after restart the whole Core is stuck trying to update Spark with Voyager not running, eventually giving up and exiting I think.
Core is able to download Voyager sources with no issues.
Tried rebooting the instances and pulling updates with no effect.

The whole log after Core restart:

[voyager]  ✓ /home/filstation/.cache/filecoin-station-core/sources/voyager
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 2,
  retriesLeft: 9
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 3,
  retriesLeft: 8
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 4,
  retriesLeft: 7
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 5,
  retriesLeft: 6
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 6,
  retriesLeft: 5
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 7,
  retriesLeft: 4
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 8,
  retriesLeft: 3
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 9,
  retriesLeft: 2
}
Failed to download spark source. Retrying...
[spark]  ⇣ checking for updates
[spark]  ⇣ downloading source files via https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
[Error: ENOENT: no such file or directory, lstat '/home/filstation/.cache/filecoin-station-core/sources/spark'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/filstation/.cache/filecoin-station-core/sources/spark',
  attemptNumber: 10,
  retriesLeft: 1
}
Failed to download spark source. Retrying...

# /usr/bin/station --version
@filecoin-station/core: 20.5.1

# node --version
v20.11.0

# ls /home/filstation/.cache/filecoin-station-core/sources/
voyager

# ls /home/filstation/.cache/filecoin-station-core/sources/voyager/
LICENSE.md  README.md  lib  main.js  release.sh  vendor  voyager.js

Can download the file manually, it however is kind of a short I think, Voyager is much bigger (59 bytes vs 780K):

ping bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link
PING bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link (104.18.41.169) 56(84) bytes of data.
64 bytes from 104.18.41.169: icmp_seq=1 ttl=49 time=20.6 ms
64 bytes from 104.18.41.169: icmp_seq=2 ttl=49 time=25.2 ms
^C
--- bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 20.587/22.907/25.228/2.320 ms

# wget https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link?format=car
--2024-05-14 15:57:07--  https://bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link/?format=car
Resolving bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link... 172.64.146.87, 104.18.41.169, 2606:4700:4400::ac40:9257, ...
Connecting to bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link|172.64.146.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Cookie coming from bafkreia6vx56ap5exvmx3xugz7zhznytq5gexubwqb57g2x6ck62w6zj7q.ipfs.w3s.link attempted to set domain to w3s.link
Length: unspecified [application/vnd.ipld.car]
Saving to: ‘index.html?format=car’

index.html?format=car                                         [ <=>                                                                                                                               ]      59  --.-KB/s    in 0s      

2024-05-14 15:57:07 (6.74 MB/s) - ‘index.html?format=car’ saved [59]

# wget https://bafkreiaddduwxba4llnghsw76y53sppn7mt6toidbpvle4j2laplubqlva.ipfs.w3s.link?format=car
--2024-05-14 16:00:34--  https://bafkreiaddduwxba4llnghsw76y53sppn7mt6toidbpvle4j2laplubqlva.ipfs.w3s.link/?format=car
Resolving bafkreiaddduwxba4llnghsw76y53sppn7mt6toidbpvle4j2laplubqlva.ipfs.w3s.link... 172.64.146.87, 104.18.41.169, 2606:4700:4400::ac40:9257, ...
Connecting to bafkreiaddduwxba4llnghsw76y53sppn7mt6toidbpvle4j2laplubqlva.ipfs.w3s.link|172.64.146.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Cookie coming from bafkreiaddduwxba4llnghsw76y53sppn7mt6toidbpvle4j2laplubqlva.ipfs.w3s.link attempted to set domain to w3s.link
Length: unspecified [application/vnd.ipld.car]
Saving to: ‘index.html?format=car.1’

index.html?format=car.1                                       [ <=>                                                                                                                               ] 779.05K  --.-KB/s    in 0.1s    

2024-05-14 16:00:35 (5.90 MB/s) - ‘index.html?format=car.1’ saved [797752]

# cat 'index.html?format=car'
:ots%�U� ކ�6{)rsion�

# hexdump 'index.html?format=car'
0000000 a23a 7265 6f6f 7374 d881 582a 0025 5501
0000010 2012 ad1e e0fb a43f 59bd de7d cf86 7cf2
0000020 13b7 4c87 d04b 8036 f37b fe6a bd12 7bab
0000030 fc29 7667 7265 6973 6e6f 0001          
000003b