Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xmlrpc y2038 problem #1764

Closed
mcepl opened this issue Oct 6, 2022 · 6 comments · Fixed by #1765
Closed

xmlrpc y2038 problem #1764

mcepl opened this issue Oct 6, 2022 · 6 comments · Fixed by #1765
Assignees
Labels

Comments

@mcepl
Copy link

mcepl commented Oct 6, 2022

(originally filed in the openSUSE Bugzilla):

While working on reproducible builds for openSUSE, I found that our python-softlayer package failed tests in 2038 with

 self = <xmlrpc.client.Marshaller object at 0x7fa5770e91f0>, value = 2167459529
 write = <built-in method append of list object at 0x7fa577194740>
 
     def dump_long(self, value, write):
         if value > MAXINT or value < MININT:
 >           raise OverflowError("int exceeds XML-RPC limits")
 E           OverflowError: int exceeds XML-RPC limits
 
 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError
 =========================== short test summary info ============================
 FAILED tests/CLI/modules/cdn_tests.py::CdnTests::test_detail_account - Overfl...
 FAILED tests/managers/cdn_tests.py::CDNTests::test_detail_usage_metric - Over...
 =========== 2 failed, 1831 passed, 6 skipped, 5 deselected in 38.01s ===========

To Reproduce:

osc co openSUSE:Factory/python-softlayer && cd $_
osc build --noservice --trust-all-projects --vm-type=kvm \
  --alternative-project=home:bmwiedemann:reproducible \
  --build-opt=--vm-custom-opt="-rtc base=2038-01-20T00:00:00" \
  openSUSE_Tumbleweed x86_64

(basically run with the machine hardware clock set to post-2038)

Expected behavior
Test suite should pass.

Version
6.1.2 from tarball.

@mcepl mcepl added the Bug label Oct 6, 2022
@allmightyspiff allmightyspiff self-assigned this Oct 6, 2022
allmightyspiff added a commit to allmightyspiff/softlayer-python that referenced this issue Oct 6, 2022
@allmightyspiff
Copy link
Member

I don't know what osc is... so I'm not really able to test this as you mentioned.

However I suspect the problem is some of the fixture data has a large id number, so I'll reduce that below 2167459529 and see if that helps you.

If that fix still doesn't work, the output of the following would be helpful:

pytest -v tests/CLI/modules/cdn_tests.py
pytest -v tests/managers/cdn_tests.py

Really weird this test would only fail in this specific situation though.

@mcepl
Copy link
Author

mcepl commented Oct 6, 2022

@bmwiedemann Could you take over the conversation here, please?

@bmwiedemann
Copy link

The current #1765 does not help as it fixes nothing that would only break in 2038.

Here is a full backtrace:

 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError
 ----------------------------- Captured stdout call -----------------------------
 int exceeds XML-RPC limits
 ______________________ CDNTests.test_detail_usage_metric _______________________

 self = <tests.managers.cdn_tests.CDNTests testMethod=test_detail_usage_metric>

     def test_detail_usage_metric(self):
 >       self.cdn_client.get_usage_metrics(12345, history=30, frequency="aggregate")

 tests/managers/cdn_tests.py:32:
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 SoftLayer/managers/cdn.py:163: in get_usage_metrics
     usage = self.cdn_metrics.getMappingUsageMetrics(unique_id, self._start_date, self._end_date, frequency)
 SoftLayer/API.py:610: in call_handler
     return self(name, *args, **kwargs)
 SoftLayer/API.py:578: in call
     return self.client.call(self.name, name, *args, **kwargs)
 SoftLayer/API.py:302: in call
     return self.transport(request)
 SoftLayer/transports/timing.py:22: in __call__
     result = self.transport(call)
 SoftLayer/transports/xmlrpc.py:81: in __call__
     request.payload = xmlrpc.client.dumps(tuple(largs),
 /usr/lib64/python3.8/xmlrpc/client.py:968: in dumps
     data = m.dumps(params)
 /usr/lib64/python3.8/xmlrpc/client.py:501: in dumps
     dump(v, write)
 /usr/lib64/python3.8/xmlrpc/client.py:523: in __dump
     f(self, value, write)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

 self = <xmlrpc.client.Marshaller object at 0x7f696fc6c520>, value = 2170227406
 write = <built-in method append of list object at 0x7f697658da00>
 
     def dump_long(self, value, write):
         if value > MAXINT or value < MININT:
 >           raise OverflowError("int exceeds XML-RPC limits")
 E           OverflowError: int exceeds XML-RPC limits

 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError

allmightyspiff added a commit to allmightyspiff/softlayer-python that referenced this issue Oct 6, 2022
@allmightyspiff
Copy link
Member

allmightyspiff commented Oct 6, 2022

Sorry, it was not clear to me on reading your initial comment that you were running these tests in the YEAR 2038.

The problem is this section of code uses a timestamp from datetime.now(), which if your running this test in 2038, is going to be > MAXINT for xmlrpc, hence the error.

Since the xmlrpc doesn't support 64 bit ints, I'm not really sure I want to force these into strings just to fix a problem that won't exist for another 16 years. Anyway I will fix the tests so they don't fail.

allmightyspiff added a commit to allmightyspiff/softlayer-python that referenced this issue Oct 6, 2022
@bmwiedemann
Copy link

btw: You can emulate it on any Linux with

date --set=2038-01-20

@bmwiedemann
Copy link

https://stackoverflow.com/questions/107616/xml-rpc-best-way-to-handle-64-bit-values says many xml-rpc implementations support an "i8" data type for 64-bit integers (so if python's xmlrpc.py doesn't it could be time to add it there), but OTOH XML transmits everything as string anyway, so you might just as well convert on the client side.

@allmightyspiff allmightyspiff moved this to 🏗 In progress in Q4 2022 Oct 7, 2022
Repository owner moved this from 🏗 In progress to ✅ Done in Q4 2022 Oct 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants