Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit wait for slow mirrors #36565

Merged
merged 1 commit into from
Nov 5, 2023
Merged

Limit wait for slow mirrors #36565

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions build/sage_bootstrap/download/mirror_list.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,9 +170,7 @@ def _rank_mirrors(self):
timed_mirrors = []
import time, socket
log.info('Searching fastest mirror')
timeout = socket.getdefaulttimeout()
if timeout is None:
timeout = 1
timeout = 1
for mirror in self.mirrors:
if not mirror.startswith('http'):
log.debug('we currently can only handle http, got %s', mirror)
Expand All @@ -190,14 +188,18 @@ def _rank_mirrors(self):
result_ms = int(1000 * result)
log.info(str(result_ms).rjust(5) + 'ms: ' + mirror)
timed_mirrors.append((result, mirror))
timed_mirrors.sort()
if len(timed_mirrors) >= 5 and timed_mirrors[4][0] < 0.3:
# We don't need more than 5 decent mirrors
break

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where that number 5 comes from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just feeling confident that when 5 mirrors say they don't have the file, it's time to contact the upstream url.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I don't know how things here work, I guessed that it is related with the maximum concurrent number of downloads of packages and wondered how that number is determined to be 5...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Now I learned a little how things work here. A "decent" mirror is a site that allows fast connection and hence the number of them has nothing to do with "concurrency".

Sorry for noise.

if len(timed_mirrors) == 0:
# We cannot reach any mirror directly, most likely firewall issue
if 'http_proxy' not in os.environ:
log.error('Could not reach any mirror directly and no proxy set')
raise MirrorListException('Failed to connect to any mirror, probably no internet connection')
log.info('Cannot time mirrors via proxy, using default order')
else:
timed_mirrors.sort()
self._mirrors = [m[1] for m in timed_mirrors]
log.info('Fastest mirror: ' + self.fastest)

Expand Down