net: Listen fails on Windows when SYSTEMROOT environment variable is not set

[Xjs]

Under Windows, net.Listen fails with the error message socket: The requested service provider could not be loaded or initialized. when the SYSTEMROOT variable is not set. This can e. g. be achieved by running an exec.Command with a non-nil Env that doesn't contain the parent process's environment.

I have written a sample application demonstrating the issue under https://github.com/Xjs/envless-listen.

This issue can be reproduced with go version go1.9 windows/amd64 under Windows 10.

According to https://msdn.microsoft.com/en-us/library/windows/desktop/ms740668(v=vs.85).aspx, this error occurs if some DLL cannot be loaded, so I assume a Windows socket syscall needs to load a DLL which is not found if SYSTEMROOT is unset.

I don't know how this issue is best mitigated. I don't even know if the leverage point is net or rather os/exec. One could imagine something like exec.Cmd never omitting SYSTEMROOT from the environment on Windows, even when passing a non-nil Env. However, if it is possible to infer the system root from some other source, the net package might also consider re-setting it. It might also be fine just leaving it as-is and documenting somewhere that sockets under Windows require access to that library.

[alexbrainman]

I always pass current process environment to the child. If you need to add or delete some environment variables, do that, but leave all other variables as is. If you show your program, we could show you how to do it.

Alex

[Xjs]

@alexbrainman Thank you very much. As I noted in my first paragraph, this issue occurs specially

with a non-nil Env that doesn't contain the parent process's environment.

In certain situations it may be desired not to inherit the parent process's environment. In these cases, it seems undesirable to especially have to check for the operating system, and if Windows, cherry-pick and pass the SYSTEMROOT variable as I did in my example program (that I linked to, by the way).

[alexbrainman]

In certain situations it may be desired not to inherit the parent process's environment.

That is something I have never seen myself. What are those situations?

Alex

[Xjs]

That is something I have never seen myself. What are those situations?

I imagine systems that need to be relied upon producing reproducible outputs. For example (not written in Go, though), I have seen stripping of environments in build contexts in the Bazel project.

[slrz]

Adding such magic to the standard library is a bad idea. If I remove something from the child's environment, I expect it to be gone and not for the stdlib to silently add it back behind my back.

It's not really a Go problem though, is it? Just normal system behaviour. What parts (if any) to scrub from the child's environment is inherently application-specific and should be solved there.

[bradfitz]

Bugs like this one are why I filed #25210

[alexbrainman]

I have seen stripping of environments in build contexts in the Bazel project.

Thanks for explaining. I never used Bazel myself, so I would not know why they want all environment variables removed.

I would not be surprised if you find other broken code (not just networking) once you start removing environment variables. This would include Windows itself, but also other Windows software.

Alex

[bradfitz]

@Xjs, if Bazel is stripping Windows environment variables, that seems like a bug in Bazel and please report it to them.

[Xjs]

@bradfitz will do if I encounter a case where a necessary variable is removed. (In fact, bazel-contrib/rules_go#736 contains a discussion about a missing TMPDIR variable, but I digress.)

I appreciate #25210 very much. It would at probably make the present issue more transparent. Also, There, you suggested force-inheriting certain special variables in os/exec, too. I would consider this a possible resolution for the present issue as well.

[Xjs]

@slrz

It's not really a Go problem though, is it? Just normal system behaviour. What parts (if any) to scrub from the child's environment is inherently application-specific and should be solved there.

Maybe so. On the other hand, I didn't expect the standard library to import a DLL at all, when everybody keeps telling me that one of the strengths of Go is that it produces statically-linked, self-contained binaries.

I'm still not sure what exactly needs to be done here, but I think there should be done something to avoid the original issue (net not functioning when exec'd with an empty environment on Windows) without me having to add Windows-specific code to my exec.Command invocation.

I have found that MinGW's env command-line tool doesn't remove certain variables (among those SYSTEMROOT) when using env -i. Similar magic might be justified in os/exec, wouldn't it?

[alexbrainman]

I didn't expect the standard library to import a DLL at all, when everybody keeps telling me that one of the strengths of Go is that it produces statically-linked, self-contained binaries.

DLLs is how you access system services on Windows. So, if you want to open a file or read from network connection, you have to use DLL. But you don't need to know about these, because Go standard library calls into DLLs. So in that regard Go programs is very muc "self-contained binaries".

I'm still not sure what exactly needs to be done here, but I think there should be done something to avoid the original issue (net not functioning when exec'd with an empty environment on Windows) without me having to add Windows-specific code to my exec.Command invocation.

exec.Command invocation returns exec.Cmd struct

https://golang.org/pkg/os/exec/#Cmd

if you leave Env field nil (it is set to nil by default), then your child process will inherited all your environment variables. That child process will not have problem using Windows networking.

I suspect you are modifying Env filed. Well some of those variables are important for other code that runs on your system. So you cannot just do as you please.

Alex

[Xjs]

My point exactly. It took me some time to figure out that the net.Listen call failed due to the missing environment variable SYSTEMROOT. The example repository demonstrates the difference between a nil and a non-nil Env field.

I don't want to do as I please, but I want to know what I'm doing. ;) It would be great if it were documented that code is relying on specific environment conditions.

I honestly didn't know that Go needs to access DLLs on Windows. (A colleague just told me that they reckon go binaries also dynamically-linked against glibc under Linux. I didn't know that, either.) Is there an overview about dependencies of Go binaries within the standard library? Where can I see that a DLL is required for some stdlib function?

I like @bradfitz' suggestion to platform-specifically preserve some environment variables in os/exec. This would resolve @25210 and also the present issue. However, I don't know which variables could be vital on Windows, you seem to have much more insight. If you point me in the right direction, I am happy to provide a pull request for os/exec or for the documentation of net that addresses these Windows specifics.

[alexbrainman]

I don't want to do as I please, but I want to know what I'm doing. ;)

From my experience, when you creating child process on Windows, you should not remove or change existing variables, unless you have good reason for changing them. Because software installed on your computer might be using these variables.

That is what you need to know. And you must follow that rule.

It would be great if it were documented that code is relying on specific environment conditions.

Environment variables change from one computer to another. There are no rules about which variables are safe to delete and which are not. These variables depend on your OS version and software that you install on your computer.

Is there an overview about dependencies of Go binaries within the standard library?

What kind of "overview" you are after? Why do you need to know Go dependencies? Go binaries will run on your Windows OS without you installing anything else. Go binaries will find whatever DLLs your WIndows provides automatically.

Where can I see that a DLL is required for some stdlib function?

You can read stdlib function source code to find what DLLs they use.

Alex

[Xjs]

What kind of "overview" you are after? Why do you need to know Go dependencies? Go binaries will run on your Windows OS without you installing anything else. Go binaries will find whatever DLLs your WIndows provides automatically.

Wrong. If you remove SYSTEMROOT, Go binaries won't be able to net.Listen anymore. This is exactly what I provided in the example in my first post. That's the entire point of this issue. (It's even the issue's title.)

You can read stdlib function source code to find what DLLs they use.

Right. Why isn't this documented? This is the kind of stdlib dependency I'm after.

[alexbrainman]

Why isn't this documented?

I do not know.

This is the kind of stdlib dependency I'm after.

I understand.

Alex

[iwdgo]

I found some MS reference about sub-process creation, cf. under section lpEnvironment, https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-createprocessa

It appears that Go implements correctly the expected behavior because if cmd.Env is left to nil, the environment variables are inherited from the parent process. Assigning an empty array as in the example, imposes explicitly to start the sub-process with an empty set of environment variables.

The relevant part becomes:

        env := []string{}
	cmd := exec.Command("listener")
	switch {
	case working:
		env = append(env, "SYSTEMROOT="+os.Getenv("SYSTEMROOT"))
		cmd.Env = env
	case failing:
		// nothing assigned and Env remains nil
		// failing means only that SystemRoot was not explicitly assigned.
	}

	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	err := cmd.Run()

[iwdgo]

As stated in go1.13 release notes, %SYSTEMROOT% is now inherited.
The failure submitted cannot be reproduced:

Xjs\envless-listen>go version
go version go1.13 windows/amd64

Xjs\envless-listen>envless-listen.exe
# envless-listen

This is to demonstrate that on Windows, the SYSTEMROOT variable needs to be set in order for a `net.Listener` to work. This is especially not the case when executed from an os.Cmd with a custom environment.

## Setup

`cd listener && go build`
`go build`

## Working case

`./envless-listen -working`

## Failing case

`./envless-listen -failing`

Xjs\envless-listen>envless-listen.exe -working

Xjs\envless-listen>envless-listen.exe -failing

It seems issue can be closed.

[bradfitz]

Thanks, closing.