Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 range error is not caught #3608

Closed
ahirner opened this issue May 9, 2020 · 0 comments · Fixed by #3609
Closed

S3 range error is not caught #3608

ahirner opened this issue May 9, 2020 · 0 comments · Fixed by #3609
Assignees
Labels

Comments

@ahirner
Copy link
Contributor

ahirner commented May 9, 2020

Environment information (required)

--- check: autoidentify
INFO: diagnose_tensorboard.py version 724b56cee52e7d8eb89bbeec1f0d5ce3e38c9682

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', release='19.0.0', version='Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64', machine='x86_64')

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.2.1

--- check: full_pip_freeze
boto3==1.13.1
botocore==1.16.1

Issue description

tensorboard/compat/tensorflow_stub/io/gfile.py:292 identifies out of range errors with "416" whereas above mentioned s3 client returns "InvalidRange". This leads to reraising the exception when served from S3.

I set up a public bucket for testing (rate limits might cause probs):
AWS_ACCESS_KEY_ID=002ee7e2d071a250000000003 AWS_SECRET_ACCESS_KEY=K002Obc5W+Oc/nE8sFsCDYnOxMx/Xxk S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com tensorboard --logdir s3://pubtests/tensorboards/tb2

Error:

Traceback (most recent call last):
  File "~/miniconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "~/miniconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/application.py", line 727, in _reload
    multiplexer.Reload()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 259, in Reload
    Worker()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 237, in Worker
    accumulator.Reload()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 183, in Reload
    for event in self._generator.Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 90, in Load
    for event in self._LoadInternal():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 120, in _LoadInternal
    for event in self._loader.Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 164, in Load
    for record in super(EventFileLoader, self).Load():
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 139, in Load
    yield next(self._iterator)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 110, in __next__
    self._reader.GetNext()
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 210, in GetNext
    header_str = self._read(8)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py", line 274, in _read
    new_data = self.file_handle.read(n)
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 498, in read
    self.filename, self.binary_mode, read_size, self.continuation_token
  File "~/miniconda3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 290, in read
    stream = s3.Object(bucket, path).get(**args)["Body"].read()
  File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "~/miniconda3/lib/python3.6/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRange) when calling the GetObject operation: byte range bytes=40-16777256 for a file of length 40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants