Cloning TensorFlow failed on Bazel CI

[meteorcloudy]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/928#394d5b89-5927-4c2e-8c5b-0526a560c6dc

Fetching tensorflow sources at 187ac8ef39db7856bafe816c7b5d1fe565af2c25 | 11m 6s
-- | --
  |  
  |  
  | git clone --reference /var/lib/bazelbuild https://github.com/tensorflow/tensorflow.git /var/lib/buildkite-agent/builds/bk-docker-58fd/bazel-downstream-projects/tensorflow
  | Cloning into '/var/lib/buildkite-agent/builds/bk-docker-58fd/bazel-downstream-projects/tensorflow'...
  | error: RPC failed; curl 18 transfer closed with outstanding read data remaining
  | fatal: The remote end hung up unexpectedly
  | Traceback (most recent call last):
  | File "bazelci.py", line 2418, in <module>
  | sys.exit(main())
  | File "bazelci.py", line 2398, in main
  | bazel_version=task_config.get("bazel") or configs.get("bazel"),
  | File "bazelci.py", line 668, in execute_commands
  | clone_git_repository(git_repository, platform, git_commit)
  | File "bazelci.py", line 937, in clone_git_repository
  | ["git", "clone", "--reference", "/var/lib/bazelbuild", git_repository, clone_path]
  | File "bazelci.py", line 1397, in execute_command
  | return subprocess.run(args, shell=shell, check=fail_if_nonzero, env=os.environ).returncode
  | File "/usr/lib/python3.6/subprocess.py", line 418, in run
  | output=stdout, stderr=stderr)
  | subprocess.CalledProcessError: Command '['git', 'clone', '--reference', '/var/lib/bazelbuild', 'https://github.com/tensorflow/tensorflow.git', '/var/lib/buildkite-agent/builds/bk-docker-58fd/bazel-downstream-projects/tensorflow']' returned non-zero exit status 128.
  | 🚨 Error: The command exited with status 1

[meteorcloudy]

/cc @philwo Can you take a look?

[meteorcloudy]

Also for the TensorFlow pipeline: https://buildkite.com/bazel/tensorflow/builds/2635#9dbc96f4-3357-42a3-90f5-0449d19a048b

[philwo]

I guess we'll have to switch to cloning over SSH :( For some unknown reason the HTTP connections die during git clone and there seems to be no fix.

[philwo]

I tried cloning via the git:// and ssh:// protocol, but they're also very slow.

It looks a bit like we're being throttled, but we use git clone --reference, so the network traffic isn't that much... :( It's also weird that it's only happening for that repository. The Bazel repository is even larger than Tensorflow's and we don't have the problems there.

@meteorcloudy Do you know if there's a mirror of the Tensorflow repository available on a Google server, similar to http://bazel.googlesource.com?

[meteorcloudy]

@yifeif @gunan Is there a mirror of TensorFlow somewhere? Did you ever encounter similar issue before?

[gunan]

I am not sure if we have a mirror.
@yifeif do we have a googlesource.com mirror of TF head?

[philwo]

This should be fixed now, thanks to the new git-mirrors feature in Buildkite, which uses a repo-specific reference instead of our former "all-in-one" repo, which seems to have caused the problems.