Using moto 1.1.25 with smart_open raises ValueError: file descriptor cannot be a negative integer (-1)

[matt-pazien]

Smart open uses an older version of moto to test - using a later release causes test issues.

Creating a simple unit test case using smart_open
smart-open==1.5.5
moto==1.1.25

from unittest import TestCase

import boto
import smart_open
from moto import mock_s3_deprecated


class TestSimpleSmartOpen(TestCase):
    @mock_s3_deprecated
    def test_write_encoding(self):
        conn = boto.connect_s3()
        conn.create_bucket('bucket')
        key = "s3://bucket/key.txt"
        text = u'Hello there'

        with smart_open.smart_open(key, 'w', encoding='utf-8') as fout:
            fout.write(text)
        with smart_open.smart_open(key, encoding='utf-8') as fin:
            actual = fin.read()
        self.assertEqual(text, actual)

This raises an error

ValueError: file descriptor cannot be a negative integer (-1)

[menshikh-iv]

@matt-pazien I tried to reproduce, works fine for py27, reproduced only in py36
P/S typically, we used moto==0.4.31 for tests

smart-open==1.5.5
moto==1.1.25

import unittest

import boto
import smart_open
from moto import mock_s3_deprecated


class TestSimpleSmartOpen(unittest.TestCase):
    @mock_s3_deprecated
    def test_write_encoding(self):
        conn = boto.connect_s3()
        conn.create_bucket('bucket')
        key = "s3://bucket/key.txt"
        text = u'Hello there'

        with smart_open.smart_open(key, 'w', encoding='utf-8') as fout:
            fout.write(text)
        with smart_open.smart_open(key, encoding='utf-8') as fin:
            actual = fin.read()
        self.assertEqual(text, actual)

if __name__ == '__main__':
    unittest.main()

@mpenkov wdyt about this problem?

[mpenkov]

We pin the moto version for a reason: newer versions of moto treat boto and boto3 mocks differently. Our tests are older and use boto to mock the data, but our newer implementation uses boto3 to read the mocked data. Therefore, we expect our tests not to work under newer versions of moto.

In order for them to work, we'd have to move to boto3 entirely and rewrite our tests. That is issue #43. I recommend we close the current issue as a duplicate, and add update moto to the scope of #43.

[matt-pazien]

@mpenkov I understand - but it's harder to integrate smart_open into other projects if we want to test it ourselves with moto.

[mpenkov]

@matt-pazien I understand. We will work on this issue as part of #43. Unfortunately, dealing with this change of behavior between old & new moto versions will require some effort.