zstd 1.5.4 crashes when trying to compress a file in a directory without write permissions

[botovq]

Describe the bug

Trying to compress a file in a directory without write permissions results in a segfault of zstd-1.5.4 on at least OpenBSD and Fedora 36.

To Reproduce

[tb@fedora zstd-1.5.4]$ ./zstd /etc/motd
zstd: /etc/motd.zst: Permission denied
Segmentation fault (core dumped)
[tb@fedora zstd-1.5.4]$

Expected behavior
zstd errors out instead of dumping core.

Desktop (please complete the following information):

• OS: OpenBSD and Fedora
• Version [e.g. 22] 7.2 and 36
• Compiler [e.g. gcc] gcc
• Flags [e.g. O2] defaults
• Other relevant hardware specs [e.g. Dual-core] dual core
• Build system [e.g. Makefile] gmake

Additional context

The problem is that setvbuf() is not NULL-safe. A possible fix is this:

--- programs/fileio.c.orig
+++ programs/fileio.c
@@ -644,6 +644,7 @@ FIO_openDstFile(FIO_ctx_t* fCtx, FIO_prefs_t* const pr
 #endif
         if (f == NULL) {
             DISPLAYLEVEL(1, "zstd: %s: %s\n", dstFileName, strerror(errno));
+            return NULL;
         }
         /* An increased buffer size can provide a significant performance boost on some platforms.
          * Note that providing a NULL buf with a size that's not 0 is not defined in ANSI C, but is defined

This avoids running into setvbuf(NULL, ...) a few lines later:

zstd/programs/fileio.c

Lines 645 to 657 in 1be9529

	if (f == NULL) {
	DISPLAYLEVEL(1, "zstd: %s: %s\n", dstFileName, strerror(errno));
	}
	/* An increased buffer size can provide a significant performance boost on some platforms.
	* Note that providing a NULL buf with a size that's not 0 is not defined in ANSI C, but is defined
	* in an extension. There are three possibilities here -
	* 1. Libc supports the extended version and everything is good.
	* 2. Libc ignores the size when buf is NULL, in which case everything will continue as if we didn't
	* call `setvbuf`.
	* 3. We fail the call and execution continues but a warning message might be shown.
	* In all cases due execution continues. For now, I believe that this is a more cost-effective
	* solution than managing the buffers allocations ourselves (will require an API change). */
	if(setvbuf(f, NULL, _IOFBF, 1 MB))