Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow variable expansion #9

Merged
merged 1 commit into from
Jul 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,52 @@ if err := env.Unmarshal(&cfg); err != nil {
// { "Username": "test", "Password": "password123" }
```

### Expand variables

The `expand` tag option can be used to indicate that the value of the variable
should be expanded (in either `${var}` or `$var` format) before being set.

```go
type Config struct {
Username string `env:"USERNAME,expand"`
Password string `env:"PASSWORD,expand"`
}
```

This works great with the `default` tag option:

```go
type Config struct {
Address string `env:"ADDRESS,expand,default=${HOST}:${PORT}"`
}
```

Which results in:

```bash
HOST=localhost PORT=8080 go run main.go
{Address:localhost:8080}
```

Additionally, default values can be referenced from other struct fields.
Allowing you to chain default values rather than falling back to an empty value
when an environment variable is not set:

```go
type Config struct {
Host string `env:"HOST,default=localhost"`
Port string `env:"PORT,default=8080"`
Address string `env:"ADDRESS,expand,default=${Host}:${Port}"`
}
```

Which results in:

```bash
go run main.go
{Host:localhost Port:8080 Address:localhost:8080}
```

## Contributing

Feel free to open issues or contribute to the project. Contributions are always
Expand Down
2 changes: 2 additions & 0 deletions example/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ type DatabaseConfig struct {

type Config struct {
Debug bool `env:"DEBUG"`
Host string `env:"HOST,default=localhost"`
Port string `env:"PORT,default=8080"`
Address string `env:"ADDRESS,default=${HOST}:$PORT,expand"`

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should use different name here than host and port, because the example might let people think HOST and PORT already need to be provided as fields

So maybe, something like

Suggested change
Address string `env:"ADDRESS,default=${HOST}:$PORT,expand"`
Address string `env:"ADDRESS,default=${ADDR}:$BIND,expand"`

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty standard variable expansion behavior, take Bash for example:

HOST="localhost"
PORT="8080"
ADDR="${HOST}:${PORT}"
echo $ADDR

In bash, you do define them in order. Since variables are expected to be defined before you start your code, IE:

HOST=localhost PORT=8080 go run main.go

The behavior of expand would work with that.

The reverse lookup of default values also allows for the code to be able to leverage those, so if values aren't set in the environment, you don't have to run into errors.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point about the default value for variable used in another variable using expand.

It should be covered by a dedicated test maybe

Setting just

PORT="8081"

and expect to have

type Config struct {
	Debug    bool           `env:"DEBUG"`
	Host     string         `env:"HOST,default=localhost"`
	Port     string         `env:"PORT,default=8080"`
	Address  string         `env:"ADDRESS,default=${HOST}:$PORT,expand"`
}
cfg := Config{
	Host: "localhost"
	Port: "8081"
	Address: "localhost:8081"
}

Roles []string `env:"ROLES,default=[admin,editor]"`
Database DatabaseConfig `env:"DATABASE"`
Redis RedisConfig
Expand Down
73 changes: 67 additions & 6 deletions unmarshal.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func unmarshalWithPrefix(data interface{}, prefix string) error {
continue
}

if err := unmarshalField(field, tag, prefix); err != nil {
if err := unmarshalField(field, tag, prefix, data); err != nil {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's getting too complicated, it was OK when you had 2 fields

you should define a struct with all the fields you need, then create a variable from this struct, then pass a reference to the variable and, that's it

No more code rewrites when adding an new feature, and also less pointers

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There wasn't really a way to make the default value lookup work without this. I'm open to suggestions, but variable expansion is a strong feature of environment variables.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could make a suggestion, but I have to get back from holidays, as I only have a phone with me right now.

So, I'm opening a refactoring issue

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return err
}
}
Expand All @@ -54,7 +54,7 @@ func unmarshalStruct(data interface{}, prefix, tag string) error {
}

// unmarshalField handles unmarshaling individual fields based on tags
func unmarshalField(field reflect.Value, tag string, prefix string) error {
func unmarshalField(field reflect.Value, tag string, prefix string, structPtr interface{}) error {
tagOpts := parseTag(tag)
value, found := findFieldValue(tagOpts.keys, prefix)

Expand All @@ -67,10 +67,14 @@ func unmarshalField(field reflect.Value, tag string, prefix string) error {
found = true
}

if !found {
if !found && tagOpts.fallback != "" {
value = tagOpts.fallback
}

if tagOpts.expand {
value = expandVariables(value, structPtr)
}

if tagOpts.required && value == "" {
return fmt.Errorf("required environment variable %s is not set", tagOpts.keys[0])
}
Expand All @@ -82,6 +86,58 @@ func unmarshalField(field reflect.Value, tag string, prefix string) error {
return nil
}

// expandVariables replaces placeholders with actual environment variable values or defaults if not set.
func expandVariables(value string, structPtr interface{}) string {
// Handle both ${var} and $var syntax
re := regexp.MustCompile(`\$\{([^}]+)\}|\$([A-Za-z_][A-Za-z0-9_]*)`)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regexp will be computed each time you call the method, it uses a lot of resources

Move it out the method, especially because you are using must compile. It would panic quickly if you update the regexp to something invalid, not when this feature is used

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find, I've moved the MustCompiles to variables in main but will wait to do a release until we've got through any other feedback.

matches := re.FindAllStringSubmatch(value, -1)

for _, match := range matches {
var envVar string
if match[1] != "" {
envVar = match[1] // ${var} syntax
} else {
envVar = match[2] // $var syntax
}

envValue, ok := Lookup(envVar) // Lookup the environment variable; use default if not set
if !ok {
envValue = getDefaultFromStruct(envVar, structPtr)
}
value = strings.Replace(value, match[0], envValue, -1)
}

return value
}

// getDefaultFromStruct retrieves the default value from the struct if available
func getDefaultFromStruct(fieldName string, structPtr interface{}) string {
v := reflect.ValueOf(structPtr).Elem()
t := v.Type()

for i := 0; i < v.NumField(); i++ {
fieldType := t.Field(i)
tag := fieldType.Tag.Get("env")
tagOpts := parseTag(tag)

if tagOpts.keys[0] == fieldName {
if tagOpts.fallback != "" {
return tagOpts.fallback
}
}
// Handle nested structs
if fieldType.Type.Kind() == reflect.Struct {
nestedStructPtr := v.Field(i).Addr().Interface()
nestedValue := getDefaultFromStruct(fieldName, nestedStructPtr)
if nestedValue != "" {
return nestedValue
}
}
}

return ""
}

// Helper function to read file content
func readFileContent(filePath string) (string, error) {
content, err := os.ReadFile(filePath)
Expand All @@ -108,6 +164,7 @@ type tagOptions struct {
fallback string
required bool
file bool
expand bool
}

// parseTag parses the struct tag into tagOptions
Expand All @@ -117,6 +174,7 @@ func parseTag(tag string) tagOptions {
var fallbackValue string
required := false
file := false
expand := false

if len(parts) > 1 {
extraParts := parts[1]
Expand All @@ -132,23 +190,24 @@ func parseTag(tag string) tagOptions {
if !inBrackets {
part := extraParts[start:i]
start = i + 1
parsePart(part, &fallbackValue, &required, &file)
parsePart(part, &fallbackValue, &required, &file, &expand)
}
}
}
part := extraParts[start:]
parsePart(part, &fallbackValue, &required, &file)
parsePart(part, &fallbackValue, &required, &file, &expand)
}

return tagOptions{
keys: keys,
fallback: fallbackValue,
required: required,
file: file,
expand: expand,
}
}

func parsePart(part string, fallbackValue *string, required *bool, file *bool) {
func parsePart(part string, fallbackValue *string, required *bool, file *bool, expand *bool) {
if strings.Contains(part, "default=[") || strings.Contains(part, "fallback=[") {
re := regexp.MustCompile(`(?:default|fallback)=\[(.*?)]`)
matches := re.FindStringSubmatch(part)
Expand All @@ -165,6 +224,8 @@ func parsePart(part string, fallbackValue *string, required *bool, file *bool) {
*required = true
} else if strings.TrimSpace(part) == "file" {
*file = true
} else if strings.TrimSpace(part) == "expand" {
*expand = true
}
}

Expand Down
114 changes: 114 additions & 0 deletions unmarshal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -663,3 +663,117 @@ func TestReadFileContentError(t *testing.T) {
t.Errorf("expected error to start with %s, got %s", expectedErrPrefix, err.Error())
}
}

func TestUnmarshalExpand(t *testing.T) {
setEnvForTest(t, "HOST", "localhost")
setEnvForTest(t, "PORT", "8080")
setEnvForTest(t, "BASE_URL", "http://${HOST}:${PORT}")

type Config struct {
BaseURL string `env:"BASE_URL,expand"`
}

var cfg Config
err := Unmarshal(&cfg)
assertNoError(t, err, "Unmarshal with expand")

expected := Config{
BaseURL: "http://localhost:8080",
}

assertEqual(t, expected, cfg, "UnmarshalExpand")
}

func TestUnmarshalExpandWithDefault(t *testing.T) {
setEnvForTest(t, "HOST", "localhost")
setEnvForTest(t, "PORT", "8080")

type Config struct {
BaseURL string `env:"BASE_URL,default=http://${HOST}:${PORT}/api,expand"`
}

var cfg Config
err := Unmarshal(&cfg)
assertNoError(t, err, "Unmarshal with expand and default")

expected := Config{
BaseURL: "http://localhost:8080/api",
}

assertEqual(t, expected, cfg, "UnmarshalExpandWithDefault")
}

func TestUnmarshalExpandWithMissingEnv(t *testing.T) {
type Config struct {
BaseURL string `env:"BASE_URL,default=http://${HOST}:${PORT}/api,expand"`
}

var cfg Config
err := Unmarshal(&cfg)
assertNoError(t, err, "Unmarshal with expand and missing env variables")

expected := Config{
BaseURL: "http://:/api", // Expanded with empty strings for missing HOST and PORT
}

assertEqual(t, expected, cfg, "UnmarshalExpandWithMissingEnv")
}

func TestGetDefaultFromStructWithFallback(t *testing.T) {
type Config struct {
Host string `env:"HOST,default=localhost"`
Port string `env:"PORT,default=8080"`
Address string `env:"ADDRESS,default=${HOST}:${PORT},expand"`
}

var cfg Config
defaultHost := getDefaultFromStruct("HOST", &cfg)
defaultPort := getDefaultFromStruct("PORT", &cfg)

if defaultHost != "localhost" {
t.Errorf("expected default host to be 'localhost', got '%s'", defaultHost)
}

if defaultPort != "8080" {
t.Errorf("expected default port to be '8080', got '%s'", defaultPort)
}
}

func TestGetDefaultFromStructWithNestedStruct(t *testing.T) {
type NestedConfig struct {
NestedField string `env:"NESTED_FIELD,default=nested_default"`
}

type Config struct {
Host string `env:"HOST,default=localhost"`
Nested NestedConfig `env:"NESTED"`
}

var cfg Config
defaultNestedField := getDefaultFromStruct("NESTED_FIELD", &cfg)

if defaultNestedField != "nested_default" {
t.Errorf("expected default nested field to be 'nested_default', got '%s'", defaultNestedField)
}
}

func TestExpandVariables(t *testing.T) {
setEnvForTest(t, "HOST", "localhost")
setEnvForTest(t, "PORT", "8080")

type Config struct {
BaseURL1 string `env:"BASE_URL1,default=http://${HOST}:${PORT}/api,expand"`
BaseURL2 string `env:"BASE_URL2,default=http://$HOST:$PORT/api,expand"`
}

var cfg Config
err := Unmarshal(&cfg)
assertNoError(t, err, "Unmarshal with expand and default")

expected := Config{
BaseURL1: "http://localhost:8080/api",
BaseURL2: "http://localhost:8080/api",
}

assertEqual(t, expected, cfg, "ExpandVariables")
}