httplib patcher: AttributeError: 'HTTPResponse' object has no attribute '_xray_prop'

[IlyaSukhanov]

AWS Xray SDK errors out when executed outside of segment/subsegment

Expected behavior warn but do not fail on this error.

  File "/......../parsers.py", line 40, in .............
  df = pandas.read_csv(s3_url)
  File "/var/task/pandas/io/parsers.py", line 655, in parser_f
  return _read(filepath_or_buffer, kwds)
  File "/var/task/pandas/io/parsers.py", line 405, in _read
  parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/var/task/pandas/io/parsers.py", line 764, in __init__
  self._make_engine(self.engine)
  File "/var/task/pandas/io/parsers.py", line 985, in _make_engine
  self._engine = CParserWrapper(self.f, **self.options)
  File "/var/task/pandas/io/parsers.py", line 1605, in __init__
  self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 562, in pandas._libs.parsers.TextReader.__cinit__ (pandas/_libs/parsers.c:6175)
  File "pandas/_libs/parsers.pyx", line 751, in pandas._libs.parsers.TextReader._get_header (pandas/_libs/parsers.c:9268)
  File "pandas/_libs/parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas/_libs/parsers.c:11755)
  File "pandas/_libs/parsers.pyx", line 2173, in pandas._libs.parsers.raise_parser_error (pandas/_libs/parsers.c:28589)
  File "/var/task/s3fs/core.py", line 1243, in _fetch_range
  return resp['Body'].read()
  File "/var/runtime/botocore/response.py", line 74, in read
  chunk = self._raw_stream.read(amt)
  File "/var/runtime/botocore/vendored/requests/packages/urllib3/response.py", line 239, in read
  data = self._fp.read()
  File "/var/task/aws_xray_sdk/ext/httplib/patch.py", line 106, in _xray_traced_http_client_read
  xray_data = getattr(instance, _XRAY_PROP)
AttributeError: 'HTTPResponse' object has no attribute '_xray_prop'

[haotianw465]

We will look at this issue as soon as possible. It looks like you patched botocore or boto3 and httplib at the same time correct? If you could share the caller code that will help a lot with the debugging.

[IlyaSukhanov]

I'm using patch_all() which patches

[INFO]	2018-04-06T00:25:02.442Z	870efd48-3930-11e8-a425-f7be7f28044b	successfully patched module httplib
[INFO]	2018-04-06T00:25:02.443Z	870efd48-3930-11e8-a425-f7be7f28044b	successfully patched module pynamodb
[INFO]	2018-04-06T00:25:02.443Z	870efd48-3930-11e8-a425-f7be7f28044b	successfully patched module requests
[INFO]	2018-04-06T00:25:02.443Z	870efd48-3930-11e8-a425-f7be7f28044b	successfully patched module sqlite3

The caller code is actually not mine, using pandas.read_csv() (https://github.com/pandas-dev/pandas/blob/v0.22.0/pandas/io/parsers.py#L557-L709)

[haotianw465]

I see. From the stack trace pandas is depending on botocore to call AWS S3 APIs to retrieve files. botocore has some certain part that depends on httplib. The httplib patcher failed because botocore use it in an unexpected way.

We will release a bug fix. With the fix you will not get this error. But you won't see the http roundtrip time since botocore has custom logic to actually send the http request.

[IlyaSukhanov]

Thank you @haotianw465, I can reproduce the error quite easily so let me know if you want a patch tested.

[haotianw465]

Though I don't know how you are using pandas in your application but I'm fairly confident that if you change this line https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/ext/httplib/patch.py#L106 to the following:

    xray_data = getattr(instance, _XRAY_PROP, None)
    if not xray_data:
        return wrapped(*args, **kwargs)

it should work, assuming you have write access to change this file in place.

[IlyaSukhanov]

Confirmed that this patch works.

I see that httplib still gets patched:

successfully patched module httplib

But instead of crashing I get the expected warning:

cannot find the current segment/subsegment, please make sure you have a segment open

[haotianw465]

Cool. We will roll out this fix in the next release as soon as possible, though we cannot provide an exact date.

In the meantime if your application doesn't have direct dependency on httplib and you don't need to trace those http calls, we would recommend to use patch((my_lib_1, my_lib_2)) to explicit capture the libraries you care about.

[haotianw465]

This issue has been fixed in the latest v1.0 release.