(server) Qubes: Conversion of medium-sized documents fills up /tmp

[apyrgio]

Problem

The Qubes converter currently handles /tmp in an inefficient way, leading to ENOSPC situations, when a document has more than 160 US letter pages.

Background

We have a known problem regarding the way we store pixel data in tmpfs (see #526), which becomes even worse in Qubes. The reason is that Qubes mounts tmpfs under /tmp this way:

tmpfs on /tmp type tmpfs (rw,size=1048576k,nr_inodes=35349,inode64)

The size of the /tmp dir in app and disposable qubes is therefore max 1GiB. At the same time, pdftoppm converts the pages at 150 DPI, which means each RGB page requires ~6.2MiB. Therefore, we can store at most ~165 pages in /tmp.

This limitation manifests both at the server-side, and at the client side. In the server-side, pdftoppm writes PPM files under /tmp/page*. We have a callback that writes the final RGB buffer to stdout and then removes the original PPM file, but these two processes are not synchronized, especially if the client-side does not catch up:

dangerzone/dangerzone/conversion/doc_to_pixels.py

Lines 336 to 342 in c4fdebc

	# Save pixel data
	await self.write_page_data(data, rgb_filename)
	# Delete the ppm file
	os.remove(ppm_filename)
	page_base = "/tmp/page"

In the client-side, we immediately write the received pixel data under /tmp/dangerzone, since this is where the pixels-to-pdf convert expects to find them:

dangerzone/dangerzone/isolation_provider/qubes.py

Lines 132 to 144 in c4fdebc

	untrusted_pixels = read_bytes(
	self.proc.stdout,
	num_pixels,
	timeout=sw.remaining,
	)
	# Wrapper code
	with open(f"/tmp/dangerzone/page-{page}.width", "w") as f_width:
	f_width.write(str(width))
	with open(f"/tmp/dangerzone/page-{page}.height", "w") as f_height:
	f_height.write(str(height))
	with open(f"/tmp/dangerzone/page-{page}.rgb", "wb") as f_rgb:
	f_rgb.write(untrusted_pixels)

Solution

We don't have a solid solution yet. If we fix the client-side issue, by converting the RGB files to PNG immediately, then the disposable qube can still run out of space, if the client-side does not catch up (e.g., due to the conversion). We need a solution that fixes both sides.

[deeplow]

I have confirmed now that with #627 the server doesn't run out of space. It does however still fail on the 164th page because we haven't yet addressed the client-side limitation #526. This will be easier anyways when we look at #625.

[apyrgio]

This has been resolved by fixing #443.