Support for SSL man-in-the-middle proxies

[Vermillionaire]

What is the issue?

For security reasons many companies man-in-the-middle all SSL traffic to the internet with their own certificate. This causes a x509: certificate signed by unknown authority issue in two spots for dagger:

1. Dagger engine. This is easy to solve by mounting the certificates in the container.
2. Dagger modules. The modules appear to be loading a tar container /usr/local/share/dagger/go-module-sdk-image.tar and building a go program when running any module command. The commands all fail when the go build does a go get for dependencies and hits the same x509 issue.

Steps I have tried to work around this issue:

1. Building a ModuleRuntime as recommended in the discord thread here. I quickly found that all the existing module runtimes point back to go as a dependency, which was the issue.
2. Putting my certificates in the engine and the tar container. This resolves the certificate issues but breaks modules for some reason. On a fresh project (or any project) made with dagger mod init it would always fail with "main object not found".

Posting this as recommended in discord since there appears to be no workarounds and modules are completely broken in this environment.

Dagger version

v0.9.6

Steps to reproduce

No response

Log output

No response

Current Solutions

(this section and below is from edits by @sipsma)

Proxy Support

As of v0.11.4, the engine has support for "standard" proxy envs: HTTP_PROXY, HTTPS_PROXY, NO_PROXY, ALL_PROXY, and FTP_PROXY.

• If any of these env vars are set on the engine container (e.g. registry.dagger.io/engine:v0.11.4) they will be automatically passed to all containers started by the engine.
• They will be passed in both UPPER and lower case to support applications that have a preference for one or the other.
• These values also will be honored for internal operations (such as pulling containers).
• Changing these values has no impact on the caching of operations.
• If a container has any of these values explicitly set, the engine will not override them with the values from the system.

Additionally, we also support custom GOPROXY values being set on the Go SDK (only the Go SDK for now, not all containers).

• In this case, the engine needs to be provisioned with an environment variable _DAGGER_ENGINE_SYSTEMENV_GOPROXY=<value>

Custom CA Support

The engine also has support for use of custom CAs. When configured, the engine will use them for any of its internal requests (e.g. pulling/pushing container images) and makes a best-effort attempt at installing them in every container started by the engine (explained more below).

There was a bug in v0.11.4, so currently you will need to use the engine image registry.dagger.io/engine:2ae9f4d815c1fcf48626ce9437c788d0efb91a79 which is a build off of our main branch.

• The next release will have this fix in place.

To configure the engine with custom CA certs, you will need to provision a custom engine with the certs placed in the directory /usr/local/share/ca-certificates.

• This is all that's needed, the engine will see those on startup and run update-ca-certificates automatically so they are configured for use from there.

The support for automatic installation in all containers is best-effort because CA certificates are very far from standardized and their configuration depends on the distro of the base image of the container being installed into.

• We currently have support for:
  • Alpine
  • Debian based distros (debian, ubuntu, etc.)
  • RHEL based distros (rhel/ubi, fedora, centos, etc.)

If a container is supported, CAs will be installed before the container runs and uninstalled after it finishes.

• This is to ensure that no CA certs leak into any images that are published from containers created by Dagger
• This unfortunately adds some performance overhead, though it remains to be seen how much in practice and whether it's actually a noticeable bottleneck. Feedback on this from real-world users will be appreciated!

If a container is not supported or a non-fatal error is hit during installation, then Dagger will not fail the exec and instead just continue on without the CAs installed.

• If you hit this situation, let us know and we can look into adding support.
• A workaround in the mean time is to explicitly install them into containers (i.e. withMountedFile).

Further support planned

Support for client-side settings

We should add support for configuring CA certs and proxy settings from the client side. This is intended for corporate use-cases where Dagger users are on a machine that also has all these settings available+configured to the same values that the engine should use (whether or not it's on the same machine).

The values may be set either from env vars, a config file, or both. Leaving that open as TBD for now, but will note that if env vars are supported we should require them to have a specific prefix like DAGGER_ENGINE_ to ensure that users more explicitly opt-in to these values being propagated to their pipelines.

This helps solve the problem of needing to custom provision an engine to set these values, though only for users that have these settings available client-side.

Generalized custom proxy support

Right now GOPROXY is supported only for our Go SDK. We also don't have support for other language/application specific proxy settings other than the "standard" ones mentioned above (HTTP_PROXY, HTTPS_PROXY, etc.).

This is fairly limiting, so we need to design a way for more arbitrary support for passing system-wide application-specific proxy settings to containers. The current workaround is to explicitly set them on containers, but that has the problem of being tedious/boilerplate-y and also includes those values in the cache key of the operations, which invalidates them if they change.

[nipuna-perera]

Thank you for posting this issue I am in a similar env and have the same issues

Some context/related work:

#6077
#6112 (comment)

They are more aligned to corp proxies but same deal with things like certificates.

CC @sipsma

[nipuna-perera]

Additional context based on recent findings - any version mismatch (go toolchain) in go.mod or go.work files within existing go modules may cause the engine pull additional dependencies which may break within corporate environments which are unable to pull and validate modules from the internet without the right proxies or certificates.

https://discord.com/channels/707636530424053791/1209532081697325147

A temporary fix is the make sure any go.mod or go.work files reference the same go version used by dagger.

[sipsma]

Fully tackling every variation on this general issue and making everything totally seamless will be a significant effort, but as a first step to improve the current situation I think we could perhaps mount the engine's /etc/ssl/certs into SDK containers. I think that would remove the currently impassable hard blocker.

It would still require users setup a custom engine (as described here). But it should at least stop the worst of the bleeding and set us up for more iterative improvements.

(adding to v0.10.x since I think this would be a relatively small scope of effort)

[nipuna-perera]

Thank you @sipsma I think that will unblock us partially. The other requirement that I think goes hand in hand are system proxies. I think having a way to "passthrough" the system proxies to the SDK containers is equally important as the certificates. My company requires the proxies to pull any go dependencies from the internet.

[sipsma]

EDIT: put most up-to-date proposal in the PR description

Previous proposal

Want to contextualize the problem and possible solutions short to long term so we can get a consensus on how to approach it before going through any one-way doors.

Unfortunately this issue goes very wide and deep, so I apologize in advance for the wall of text here 😅

The problem is obviously that users need to use dagger with custom CA certs and proxy settings. I'd break this down into 4 problem areas:

1. The core API - Container.from, Git, HTTP, etc.
   • Workaround today is to provision a custom engine with certs at /etc/ssl/certs and any standard proxy env vars set (HTTP_PROXY, etc.)
   • This is not ideal though as provisioning a custom engine is a lot of overhead for users
2. In withExec/Service containers - needed if whatever is running in there makes external network calls
   • Workaround today is for users to explicitly mount CA certs/set proxy env vars into the Container/Service from their Dagger SDK code
   • Pre-Zenith, this workaround was okay but not ideal in that it leaks what should be operator concerns into developer code
   • But in the new world of reusable Functions, this really breaks down since you can't update external module dependency code with these settings unless you forked it
3. In module containers (i.e. the containers that functions execute in) - needed if function code directly makes calls to the external network
   • Workaround today is to create CA certs and set env vars from function code itself at the beginning of every function that needs it
   • That workaround is bad enough, but it also means that external module dependecies would have to be forked to add these settings
4. In SDK implementation containers, e.g. the go sdk may need CA certs/GOPROXY/etc. settings in order to build user modules
   • There's no workaround currently, due to the fact that the go sdk is the special "root" sdk of all others and it can't be configured by users with these settings yet.

---

My previous comment was a quick fix to just 4. above (and, as @nipuna-perera pointed out, only to the CA problem, not proxy env vars), but it would still leave users with a long trail of awful workarounds.

That doesn't rule it out as a first step, but here's an outline of one way we could start there and lift the one insurmountable blocker relatively quickly, but in such a way that we can follow-up with more comprehensive solutions iteratively.

Step 1 - Read-only API for Engine-wide configuration and SDK support

CA certs and proxy settings are operator-level and system-wide concerns, so they should still be configured on the engine as a whole rather than hardcoded by developers in their Dagger code.

However, these settings should be exposed via a core API (read-only to start). Putting an API in front of these allows us more safely iterate on how they are configured in subsequent steps.

The API could look something like this:

extend type Query {
  daggerEngine: DaggerEngine!
}

type DaggerEngine {
  caCerts: Directory
  httpProxy: String
  httpsProxy: String
  # etc.

  # arbitrary key/value engine-wide settings
  setting(key: String!): String
}

The idea is that CA certs and "standard" proxy settings are wide-spread enough that that can get their own dedicated fields, but we also need to support things like GOPROXY and similar that are much more specific and not possible to generically support, which is where the setting field comes in.

This alone would help with (referring to the list of problems above)

• Problem 4. - We can update the Go SDK implementation to mount in CA certs, set GOPROXY, etc. when it's invoked by the engine, which removes the hard blocker there. Same idea for other builtin SDKs.
• Problem 3. - We can update existing SDKs to setup module runtime containers with CA certs and proxy settings
• Problem 2. - Users will still need to explicitly configure withExec/Service containers with these settings for now, but at least can read the values from an API rather than hardcoding values

To start, these settings will be read-only in the API and instead still set by custom provisioning an engine:

• CA certs should still be mounted into the engine container at /etc/ssl/certs
• "Standard" proxy env vars should be set on the engine container (e.g. HTTP_PROXY, HTTPS_PROXY, etc.)
  • It seems that there is not any actual standard for these, but the convention is wide-spread enough that it's a pseudo-standard (link)
• Any other system-wide setting strings can be set as env vars on the engine container with a prefix like DAGGER_ENGINE_. This would be used for more specific settings like DAGGER_ENGINE_GOPROXY.

Thankfully, this would also be fairly straightforward implementation-wise, so if we go with it we can still unblock users relatively quickly.

Open Questions

The arbitrary key/value store exposed by DaggerEngine.setting feels like the Windows registry in that user code could become relient on ambient system properties that may or may not be set. Maybe that tradeoff is worth it in the end, but it's far from ideal and worth more consideration.

(Optional) Step 2 - Default all containers to use CA and standard proxy settings

This next step would result in CA certs and standard proxy settings to be applied to all containers automatically, including withExec/Service.

This solves the problems in 2. where a user who requires specific CA/proxy settings can't consume an external module that uses withExec unless that module explicitly does e.g.

dag.Container.
   From("foo").
   WithMountedDirectory("/etc/ssl/certs", dag.DaggerEngine().CACerts()).
   WithEnv("HTTP_PROXY", dag.DaggerEngine().HTTPProxy())
   // etc.

This would be a breaking-change, though only to any users that have custom provisioned an engine with custom CA certs and proxy settings, and the extent to which it would actually break them is not totally clear since in many (if not most) cases it would actually be fixing something that previously didn't work.

If we default all containers with this, there would still be corner cases to deal with:

• /etc/ssl/certs is not 100% standardized across base images (depending on libc and other distro differences), so defaulting to mounting them into all containers there would not work in all cases
  • The only generic way I can think to solve this would be an API like Container.withCACertsPath that allows specifying where they should be mounted by default, though that still requires an explicit call in user code.
  • Encouraging users to consume base images via modules rather than directly using Container.from may help here. E.g. dag.Alpine("3.18") could be a module that returns a base image with the correct default CA certs path for alpine, etc.
• Any existing user settings on the Container definition should override the defaults in order to support use cases that call for specific CA certs or proxy settings on just that container (using containers to run integration tests that need specific settings comes to mind for example)
  • Similarly, we may need an API like Container.withoutSystemDefaults or similar to allow opting out when there's no value to override with.

Technically, this step is optional. We could just accept the problems in 2., but they seem glaring enough that it's worth thinking through all this.

Step 3 - Write API for Engine-wide configuration

The last step would allow the engine-wide settings to be set via an API rather than set by custom provisioning an engine.

The hard part here is that this should ideally be a "privileged" API that's only intended for use by operators setting up an engine as opposed to something any module or any user could call.

• Leaving the details of how to support that as TBD for now, it would be its own wall of text.

Doing this would save users from having to custom provision engines in order to configure any of this (and reprovision them if the settings change, etc.)

---

Those steps are just one possibility of course. There are also options to read settings from the client (i.e. a session attachable that reads CA certs, HTTP_PROXY, GOPROXY, etc. from the caller) to implement the DaggerEngine API, but the above currently feels like the right place to start the discussion.

I really want us to be able to unblock users here quickly, so even if we leave open a few questions on the long-term plan, provided we agree on the immediate steps and general outline we can get going on the actual fixes.

[jedevc]

I like this API idea a lot 🎉

One potential point of contention though - how does this new config API interact with the old config.toml file? Are settings loaded from both? Which takes precedence? Could we even deprecate the old one? - and if we do that, what impact does that have on pre-provisioned dagger instance, e.g. in a k8s cluster like our CI (note, a little bit of overlap with #5583).

[nipuna-perera]

Fantastic analysis and potential solution!

One issue I have with the "setting everything in a custom engine image" is that proxies sometimes tend to need authentication. So we'd pass in http://user:[email protected]. I don't think the above solutions handle that well. Maybe except Step 3? We obviously don't want to hardcode credentials into the Engine container. I will do some more analysis next week but I think there is a way to configure the authentication in the local git config. If that's the case, would any of the above solutions work to pass that through to the sdk implementation containers?

The other issue is we have different egress proxies for local (on-prem) vs cloud (remote, aws). For that scenario we'd have to produce 2 different engine images and ask users to set different _EXPERIMENTAL... flags depending on where they are executing the pipeline. Come to think of this, it's strange how this works today on my local laptop. I am assuming none of the proxies are set inside the sdk impl containers, yet, this works in my local machine. I'm pretty sure we still need the proxies to reach GH as it's hosted outside of our internal network.

I really like the dagger SDK (non function) way of being able to "pass-through" these settings from the local environment. Maybe some kind of config we can set where we tell the dagger engine to use the local env vars (just for proxies?). I believe that's what Step 3 is working towards.

[vito]

@sipsma Nice job unpacking all that. Having gone through almost exactly this situation with Concourse (concourse/concourse#1027) and arrived at a similar conclusion (engine-side TSL/proxy config auto-injected into containers), here's what I can provide:

• Agree with your assessment that there is no "standard" for any of this 😭 - proxy env vars are "meh, ok, fine" but CA certs are pretty painful to have to deal with across all kinds of container images.
• Agree with auto-installing certs into containers as we run them, despite the difficulty. /etc/ssl/certs is also the path that we landed on in Concourse, but I think we can be a bit smarter in the shim and detect common paths and even try to install it "properly" using the appropriate CA cert installation binary. I already had to implement this for Bass so we can just copy-(review-)pasta. :) Maybe we can even build on this to support c2c TLS.
• Folks will get big mad if you clobber their SSL certs: Certificate Propagation Breaks Alpine OpenJDK Images concourse/concourse#2042 - probably a waste of time to read in full, but the amount of comments shows how much people care about it. 😅 I think we can avoid clobbering with the shim logic above.
• The setting API does give me uneasy Windows registry vibes. :P Do we admit conceptual defeat (no global env) but win a terminology battle by just putting that + proxy config into one big 'env' field?

How do we want proxy + TLS config to affect caching? If someone re-configures their engine with a new proxy env var or TLS cert, does that bust every cache? Combined with @nipuna-perera's point about creds sometimes being in the proxy env var, I'm thinking we don't want these to end up in cache keys. But does that also mean we need the proxy envs to be Secret?

[sipsma]

Thanks for the feedback everyone! All super helpful.

One potential point of contention though - how does this new config API interact with the old config.toml file? Are settings loaded from both? Which takes precedence? Could we even deprecate the old one?

Yeah good question, I think it would be reasonable to do something like this order of precedence:

1. (If/when write-API is supported) values written via the API
2. If relevant env vars are set in engine contanier, use those
3. Otherwise, use values in config.toml, if set
4. Finally, default to no value

If we do end up supporting client-side settings (i.e. picked up from their env, a config file, etc. via a session attachable), then I think that would come first in the precedence order, even above writes to the engine API (though this could be debated separately).

One issue I have with the "setting everything in a custom engine image" is that proxies sometimes tend to need authentication. So we'd pass in http://user:[email protected]. I don't think the above solutions handle that well. Maybe except Step 3? We obviously don't want to hardcode credentials into the Engine container.

Combined with @nipuna-perera's point about creds sometimes being in the proxy env var, I'm thinking we don't want these to end up in cache keys. But does that also mean we need the proxy envs to be Secret?

Great callouts, didn't think about that at all.

For the security aspects, I agree that this means you'd want to avoid hardcoding any credentials into the engine container. I think the options are:

1. If the container runtime you are using to run the engine supports some sort of "secrets" feature, you can use that. E.g. k8s supports mounting secrets as files/env vars, etc.
2. If/when a write API is supported, that would solve this problem
3. If/when client-side settings are supported, that would also solve this problem

This is all making me a bit more biased towards implementing support for client-side configuration of all this early on (with precedence as described above). If the user's container runtime supports secrets, that seems best all around; but the write-API is a pretty huge rabbit hole so client-side settings may make sense to prioritize support for first as the next best thing.

• Will think a bit more on what this would look like specifically

Another thing that came to mind is that transparently setting proxy env vars that contain credentials on all containers inherently means that any 3rd party modules will have access to those creds.

• This would be a problem in a scenario of a "poisoned" dependency that was maliciously updated to exfiltrate those creds.
• I'm not sure there's all that much we can do here though; if the container needs the proxy creds to work, then it has to have access to them. Maybe some really fleshed out "trust" system could allow users to specify which modules should have access, but that's an enormous separate can of worms, unclear how much it matters yet.

For the caching aspect, yeah I think it would make sense to just always use a SecretEnv if/when we start transparently setting them on all containers. That would result in the the cache not busting due to a value change and prevent the values from being stored in the on-disk cache. Same idea for CAs/tls certs, using secret mounts, etc.

The other issue is we have different egress proxies for local (on-prem) vs cloud (remote, aws). For that scenario we'd have to produce 2 different engine images and ask users to set different _EXPERIMENTAL... flags depending on where they are executing the pipeline.

Yeah makes sense, that is the sort of pain that would be alleviated with a write-API. But this is another pro for implementing client-side settings early on.

Agree with auto-installing certs into containers as we run them, despite the difficulty. /etc/ssl/certs is also the path that we landed on in Concourse, but I think we can be a bit smarter in the shim and detect common paths and even try to install it "properly" using the appropriate CA cert installation binary. I already had to implement this for Bass so we can just copy-(review-)pasta. :)

Nice! Thanks for all the links to the prior art, super helpful. Yeah I was avoiding "autodetect" idea because I wasn't sure it was really viable, but if we can get it to work in like 99% of cases that changes everything quite a bit. It may mean we could avoid the API entirely (at least, it would be internal only and not exposed to end users). @vito in your experience was this sort of approach pretty robust or was there a small-but-significant tail of cases where it didn't work?

The setting API does give me uneasy Windows registry vibes. :P Do we admit conceptual defeat (no global env) but win a terminology battle by just putting that + proxy config into one big 'env' field?

Yeah I really don't like settings at all, but couldn't think of anything else at the time.

I could go either way on whether proxy config should just be combined into the k/v map, but the CA certs specifically should probably stay a separate field since they should be a Directory.

• Tangentially, maybe the proxy settings and k/v map should be string->Secret instead, in light of the other discussion? At which point, it becomes more like our existing in-memory secret store (just "global" rather than per-session) 🤔... Maybe there's something there. Just thinking out loud.

But either way, thinking again today, I'm wondering if there's some other approach to support the "arbitrary global settings" like GOPROXY that could reduce a lot of the pain (though not entirely), very vaguely outlined:

• For modules (which includes SDK implementations) they need to accept these settings as arguments (their constructor making the most sense). But we could support a variation on our existing default pragma/annotation that lets you say "default to the global engine value", which would be set as previously described. E.g. //+systemdefault GOPROXY/@system_default("GOPROXY"), etc.
  • In combination with transparently setting all of this on all containers, I think it would allow us to remove the need to put settings in the API entirely. For the case of users still running Dagger SDKs on their host rather than in modules, once we implement support for module codegen, they would be able to call out to a module to retrieve these values instead (slight overhead relative to direct API call, but not that bad and could be worth it for other benefits below).
• The nice aspect of this is it makes which arbitrary values are requested by which module explicit+discoverable. And while they still are "ambient" to some degree, it's only in terms of default values now, which feels a bit less bad.
• e.g. if the Python SDK needs something like PYPI_PROXY (not sure if that's a thing, but you get the idea), the Python SDK module's constructor would accept it as an argument with a //+systemdefault PYPI_PROXY pragma
  • The Go SDK of course is not actually a module, but we can implement the same behavior for it either way as a special case.

---

I'm gonna write an updated proposal based on all this and post to PR description, I think there's enough new considerations/ideas to make some changes. Will re-ping once done.

[vito]

Nice! Thanks for all the links to the prior art, super helpful. Yeah I was avoiding "autodetect" idea because I wasn't sure it was really viable, but if we can get it to work in like 99% of cases that changes everything quite a bit. It may mean we could avoid the API entirely (at least, it would be internal only and not exposed to end users). @vito in your experience was this sort of approach pretty robust or was there a small-but-significant tail of cases where it didn't work?

Well, with a user base of 1, I can say it worked great for everyone.

In all seriousness, it didn't get a whole lot of mileage, but it never gave me any trouble either once I figured out a versatile enough pattern (which was done in the first commit). It'll probably take some fiddling to deal with exotic distros and such as they come up, but the shim is in a pretty luxurious position since it runs in the container context and can auto-detect that all pretty easily, so I'm somewhat confident in that approach. I'm at least more confident in it than mounting over a not-quite-even-standard directory and crossing our fingers. 😄

---

Don't have much to add re: the rest, agree they sound worth exploring. I think it's a great idea to think up a few approaches here without getting too married to the first one.

[sipsma]

@nipuna-perera @vito @jedevc updated the PR description with a new iteration on the proposed fixes. It's cobbled together from the previous one and other comments, so apologies for anything you've already read; just want it to be clear for anyone else reading it the first time. Have a few remaining open questions at the end that I'd appreciate any feedback on.

[nipuna-perera]

I like the updated plan! My favorite is Step 4 where we get to configure this client side as I would love to not have to maintain our own dagger engine. One note: I am assuming you are including NO_PROXY also as a special proxy env. Just want to confirm as I don't see a mention of it in your description.

For No. 2 in open questions.

I am currently using the _EXPERIMENTAL_DAGGER_ENGINE var. We are already doing docker-in-docker on our Jenkins CI agents so I could technically spin up the dagger engine image before the dagger call and point to that. I can pass in the -e args as credentials for sure. Once step 1 is complete, I can test that :)

Finally, not totally related, but do you have plans to remove the _EXPERIMENTAL prefix from key env vars? That may scare some users.

[cndoit18]

This is important to us, we can't wait to use dagger, but are blocked by this issue

[haoqixu]

I have submitted a PR (#7186) to propagate DAGGER_ENGINE_* environ variables to the pipelines.

We can't access proxy.golang.org and need to set GOPROXY for go. Otherwise, dagger will time out every time during the codegen process of gosdk. There doesn't seem to be a way to do it yet.

[sipsma]

Just FYI for everyone following this issue, I updated the description of the issue with the current state of support and what's planned for the future.

These features are still very much experimental. They have e2e tests and some limited real-world validation, but given the complexity of proxies and especially CA certs, we will need some more real-world testing to help validate them.

If you give them a try, please let us know how it goes (either here or on our discord)!

[nipuna-perera]

Hey @sipsma , thank you so much for the awesome work on this! I tested the proxies and they definitely work.

I don't have a use case currently to test cacerts but I will try to find a way to test it. I had a use case sometime back where it failed. We don't use GOPROXY yet but will start in Q3 or Q4 and I'll be able to test it.

I can finally run dagger functions on our corporate Jenkins instance. yay! Final piece of this puzzle for me is #6113 and I can start sharing my modules with folks :)

[Vermillionaire]

I have been testing the certificates side and no issues so far! #6113 is also something I'm waiting on before fully diving into modules.

[OptikMist]

Good day. I've been waiting for the issues with CA certs to be resolved so I could start building out some dagger pipelines in our corporate environment. Really appreciate the summarized status on the initial post. Wondering if I should venture into using "registry.dagger.io/engine:2ae9f4d815c1fcf48626ce9437c788d0efb91a79 " or just wait for the next Dagger build. Been using stock dagger installs up to this point so it is not clear what needs to be done to use a interim build.

Any thoughts or suggestions would be welcome. Also, just to chime in, getting #6113 is of great interest and will open up wider corp adoption.

[jedevc]

@OptikMist fyi we've fixed the CA certs issues with modules in #7356, which has been released (just now!) in https://github.com/dagger/dagger/releases/tag/v0.11.5 🎉

[OptikMist]

Thanks @jedevc, I pulled the latest dagger version and reran my pipeline. Unfortunately I'm still seeing the issue.

(dagger-env) clarge@Ubuntu-22:~/Projects/Automation/Misc-Automation-Tools/github-automation/org-management/dagger$ dagger run python run-org-mgmnt.py
✔ Host.directory(exclude: [], include: [], path: "/home/clarge/Projects/Automation/Misc-Automation-Tools/github-automation/org-management"): Directory! 2.3s
✔ Host.directory(exclude: ["node_modules/", "tmp/"], include: [], path: "."): Directory! 2.1s
✘ Container.from(address: "internal docker repo reference"): Container! 0.0s
! failed to resolve source metadata for internal docker repo reference: failed to do request: Head "https: internal docker repo reference": tls: failed to verify certificate: x509: certificate signed by unknown authority
✘ remotes.docker.resolver.HTTPRequest 0.0s
! tls: failed to verify certificate: x509: certificate signed by unknown authority
✘ HTTP HEAD 0.0s
! tls: failed to verify certificate: x509: certificate signed by unknown authority

So this is pulling from an internal docker repo. I can do a docker login and docker pull on that url from the host machines shell.

[sipsma]

@OptikMist

✘ Container.from(address: ""): Container! 0.0s

Just double checking, did you leave out the name of the image for privacy reasons? Or did it actually show up as address: ""?

Either way, the CA support currently requires that you custom provision an engine using registry.dagger.io/engine:v0.11.5 as the base image and custom CA certs installed in /usr/local/share/ca-certificates, so just want to also double-check you did that too.

[OptikMist]

@sipsma , yes, I actually had added text with brackets into the output but it looks like the angle brackets is consumed by the formatter. So yes, there is a valid docker image URI specified, I removed it as it is an internal service.

[OptikMist]

Here is a nicer version of the error with the sensitive info redacted.

[akkuman]

set env

export _DAGGER_ENGINE_SYSTEMENV_GOPROXY=https://goproxy.io

I run dagger init --sdk=go --source=./dagger -vvv

it still download package from proxy.golang.org

...

Stdout:
generating go module: z_deploy
creating directory . [skipped]
creating directory z_deploy [skipped]
creating directory z_deploy/dagger [skipped]
writing z_deploy/dagger/dagger.gen.go
writing z_deploy/dagger/go.mod
writing z_deploy/dagger/go.sum
creating directory z_deploy/dagger/internal
creating directory z_deploy/dagger/internal/dagger
writing z_deploy/dagger/internal/dagger/dagger.gen.go
creating directory z_deploy/dagger/internal/querybuilder
writing z_deploy/dagger/internal/querybuilder/marshal.go
writing z_deploy/dagger/internal/querybuilder/querybuilder.go
creating directory z_deploy/dagger/internal/telemetry
writing z_deploy/dagger/internal/telemetry/attrs.go
writing z_deploy/dagger/internal/telemetry/env.go
writing z_deploy/dagger/internal/telemetry/exporters.go
writing z_deploy/dagger/internal/telemetry/init.go
writing z_deploy/dagger/internal/telemetry/live.go
writing z_deploy/dagger/internal/telemetry/logging.go
writing z_deploy/dagger/internal/telemetry/metrics.go
writing z_deploy/dagger/internal/telemetry/proxy.go
writing z_deploy/dagger/internal/telemetry/span.go
writing z_deploy/dagger/internal/telemetry/transform.go
writing z_deploy/dagger/main.go
creating directory z_deploy/dagger/internal [skipped]
creating directory z_deploy/dagger/internal/dagger [skipped]
writing z_deploy/dagger/internal/dagger/dagger.gen.go [skipped]
creating directory z_deploy/dagger/internal/querybuilder [skipped]
writing z_deploy/dagger/internal/querybuilder/marshal.go [skipped]
writing z_deploy/dagger/internal/querybuilder/querybuilder.go [skipped]
creating directory z_deploy/dagger/internal/telemetry [skipped]
writing z_deploy/dagger/internal/telemetry/attrs.go [skipped]
writing z_deploy/dagger/internal/telemetry/env.go [skipped]
writing z_deploy/dagger/internal/telemetry/exporters.go [skipped]
writing z_deploy/dagger/internal/telemetry/init.go [skipped]
writing z_deploy/dagger/internal/telemetry/live.go [skipped]
writing z_deploy/dagger/internal/telemetry/logging.go [skipped]
writing z_deploy/dagger/internal/telemetry/metrics.go [skipped]
writing z_deploy/dagger/internal/telemetry/proxy.go [skipped]
writing z_deploy/dagger/internal/telemetry/span.go [skipped]
writing z_deploy/dagger/internal/telemetry/transform.go [skipped]
running post-command: go mod tidy
post-command failed: exit status 1
Stderr:
go: downloading github.com/stretchr/testify v1.9.0
go: github.com/go-logr/[email protected] requires
        github.com/go-logr/[email protected]: Get "https://proxy.golang.org/github.com/go-logr/logr/@v/v1.2.2.mod": dial tcp 142.251.211.241:443: i/o timeout
Error: exit status 1

[cchanley2003]

You have to set the env variable on the engine. Not the client side.

 docker run  -d --name customized-dagger-engine --privileged -e _DAGGER_ENGINE_SYSTEMENV_GOPROXY

[akkuman]

_DAGGER_ENGINE_SYSTEMENV_GOPROXY

thanks, it works!!!

[shykes]

Renamed the title to be more specific (this is a request for supporting corporate man-in-the-middle SSL proxies found in some corporate environments), and clarify that it's a feature request, not a bug report.