Skip to content

Commit

Permalink
pythongh-12005: Align FileIO.readall between _pyio and _io
Browse files Browse the repository at this point in the history
This relies on `bytearray.resize()` and `os.readinto()` to reduce copies
and match behavior of `_io.FileIO.readall()`.

There is still an extra copy, and thus twice the memory required,
compared to FileIO because there isn't a zero-copy  path from
`bytearray` -> `bytes` currently.
  • Loading branch information
cmaloney committed Feb 5, 2025
1 parent cdcacec commit a830705
Showing 1 changed file with 16 additions and 9 deletions.
25 changes: 16 additions & 9 deletions Lib/_pyio.py
Original file line number Diff line number Diff line change
Expand Up @@ -1674,22 +1674,29 @@ def readall(self):
except OSError:
pass

result = bytearray()
result = bytearray(bufsize)
bytes_read = 0
while True:
if len(result) >= bufsize:
bufsize = len(result)
bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
n = bufsize - len(result)
# Parallels _io/fileio.c new_buffersize
if bufsize > 65536:
addend = bufsize >> 3
else:
addend = 256 + bufsize
if addend < DEFAULT_BUFFER_SIZE:
addend = DEFAULT_BUFFER_SIZE
bufsize += addend
result.resize(bufsize)
try:
chunk = os.read(self._fd, n)
n = os.readinto(self._fd, memoryview(result)[bytes_read:])
except BlockingIOError:
if result:
if bytes_read:
break
return None
if not chunk: # reached the end of the file
if n == 0: # Reached the end of the file
break
result += chunk

bytes_read += n
result.resize(bytes_read)
return bytes(result)

def readinto(self, buffer):
Expand Down

0 comments on commit a830705

Please sign in to comment.