proposal: spec: allow assignment-compatible values in append and copy

[rogpeppe]

As discussed here, it might be useful to make the append and copy built-ins somewhat more accepting of different types.

I have made a proposal document here:

https://docs.google.com/document/d/1HilKzERLb521XaG0Lgi6zkkj25VlXqc1oHEUmQ1v3aI

[robpike]

This will require allocating a slice under the covers. []int and []interface{} do not have the same memory and unless something magic happens, the only way to achieve this is to allocate the []interface slice and run the loop. In other words, except for microoptimizations this must do what the current user-written code does, and involves an allocation and an O(len(slice)) copy.

I am not making an argument for or against, just saying something that should appear in the proposal.

[rogpeppe]

Perhaps confusingly, I've actually made two proposals here. In the first one, only the copy and append primitives are affected, and no slice will need to be allocated under the covers. In the second ("Alternative proposal") one, yes, a new slice would need to be allocated, which I hope I've made a little clearer now.

[cznic]

The more magic builtins copy/append will learn, the less readable the code using those features gets.

The explicit loop communicates the intent, well, explicitly. And there's room for the compiler to recognize and optimize the hell out of those loops.

[rogpeppe]

cznic: At this point I think I'm mostly trending towards the second proposal, because it's actually reasonably intuitive that the "..." modifier could (when necessary) assign each element in turn,
and use of that modifier is a good potential signifier of a "allocation may occur here" place.

As for readability, I don't think it would make for less readable code, but perhaps you're seeing
something I'm not. Can you give an example of where it would make code less readable?

[cznic]

Code requiring awareness of more features packed to an "overloaded" function is less readable in my books. I should have perhaps said comprehensible instead, to be more accurate.

[bits01]

I believe the name copy() communicates the intention of what is happening very well, it does not make it look like magic. Plus it already does some magic, at least for string/[]byte copies. But also the type conversions string([]byte]) and []byte(string) do hidden copies.

[champioj]

I really want to like the alternative proposal, but I think it is important that you must explicitly indicate that your are making an allocation. Like fmt.Println(range xs…) but that syntax feels out of place...any ideas?

Another possibility (is it the right place?) that would fulfill the use case in the document would be to only have make assignment-compatible. That is, allow the seconde parameter (replacing the size) to be a slice* of a type assignable to the slice type of the first parameter.
Keeping the use case in the document, that would give:

func printInts(xs []int) {
    args := make([]interface{}, xs)
    fmt.Println(xs...)
}

func allReader(fs []*os.File) io.Reader {
    rs := make([]io.Reader, fs)
    return io.MultiReader(rs...)
}`

which feels less verbose than the first proposal while still completely explicit.
It is less overloaded that the alternative proposal too.

While it cover the usecase in the proposal, its scope is narrower as you cannot append with it. So I think it's important to define what is really important and what is less.

What I like about using make is that, unlike append, it is quite never used in convoluted code so it doesn't render like the code is less comprehensible.

*and perhaps map/channel, but that's another story.

[earthboundkid]

From the proposal:

With these

Is an incomplete sentence.

And

    args := append([]string, string(ts…))
    strings.Join(args, “ “)

should be

    args := append([]string{}, string(ts...))
    strings.Join(args, " ")

FWIW, I prefer the first proposal to the alternative proposal. This "looks like Go" to me:

    args := make([]interface{}, len(xs))
    copy(args, xs)

In my view, Go generally tries not to allocate a slice without explicit instruction to do so.

[extemporalgenome]

update: I just saw the part about the alternative proposal -- indeed that would involve a slice allocation in the worst case.

@robpike (not taking a stance on this proposal), but I don't believe that a slice "allocation" would be necessary as you described. There are two cases:

1. append(dst, src...)
2. append(dst, elem1, elem2, elem3)

In the first case, the source is already a slice (thus no allocation is needed). append and copy would simply need to be generalized to accept a src whose elements are assignment-compatible to dst, and perform an assignment loop; the current memcpy case would just be an optimization.

In the second case, the compiler can place an array on the stack (in the above example, a [3]T). It could derive a slice header from this stack-resident array, and place that on the stack as well -- in any case, this is a throwaway slice that clearly does not escape, and thus should never end up on the heap. If it's considered an allocation, it's certainly not an expensive one.

[bradfitz]

/cc @griesemer

[adg]

ping @griesemer

[griesemer]

Sorry, just haven't had time to look into this yet.

[rsc]

There doesn't seem to be a pressing need for this, so leaving for Go 2.

[earthboundkid]

Normally I interpret "Go 2" to mean language changes that are impossible without breaking backwards compatibility. Has the definition been changed to "non-urgent language changes"? Or did I misunderstand "Go 2"?

[griesemer]

@carlmjohnson What comprises "Go 2" or it's timeframe have not been defined yet. That said, while many agree that this might be a nice expansion on append and copy, there's no urgency. In the interest of keeping unnecessary language churn at a minimum, the decision was to postpone so we can look at this within the larger context of Go 2.

(There has been at least one other proposal in related direction, and we should probably consider all our options before venturing out to a relatively involved change to append.)

[dolmen]

It seems from the examples given that the need that triggered this proposal is array types conversion. But the syntax proposed is awkward. Go already has a type conversion syntax and it already has magic (that does allocation, copy and conversion) to convert []byte to string and the reverse.
So instead of adding magic to append and copy, a simpler syntax change would just be to just extend the type conversion syntax to have magic for any array type.
Here are the examples of the proposal adapted for this syntax:

func printInts(xs []int) {
    fmt.Println([]interface{}(xs)...)
}

func allReader(fs []*os.File) io.Reader {
    return io.MultiReader([]io.Reader(rs)...)
}

However I'm not endorsing this proposal. Instead, I am against it as it hides a costly conversion (that requires allocation, loop, conversion of each array element) under a very short syntax that people will abuse. Currently code for array conversions is verbose because this is costly at runtime, and I'm fine with that.

[akavel]

One thing implicit in the proposal, but not explicitly mentioned in the examples (not sure why), is that with the proposed change to append, the following snippet (copied from the proposal):

    func printInts(xs []int) {
        args := make([]interface{}, len(xs))
        for i, x := range xs {
            args[i] = x
        }
        fmt.Println(args...)
    }

could actually be shortened even more, to an append-based snippet (instead of the copy-based one shown in the proposal):

    func printInts(xs []int) {
        fmt.Println(append([]interface{}{}, xs...))
    }

Also, regarding append doing "extra allocations" and "O(n) complexity", I don't really undrestand the concern: AFAIK, both copy & append are already expected to do an extra allocation (of the new array; possibly amortized in case of append), and have O(n) complexity (both of them are by definition a shorthand for a loop).

Also, for me personally, the above proposed pattern seems short enough, and at the same time implicit enough (the loop and the allocation are all clearly visible by virtue of using append, and the target type is also clearly listed), that I don't see a need for the "alternative proposal".

edit: Also, I see it as further advantage of the proposal that it could become the single recommended solution the 3+ "commonly used" variants of the "covariance conversion code pattern", i.e.:

    // variant a
    args := make([]interface{}, len(xs))
    for i, x := range xs {
        args[i] = x
    }

    // variant b
    args := []interface{}{}
    for _, x := range xs {
        args = append(args, x)
    }

    // variant c
    args := make([]interface{}, 0, len(xs))
    for _, x := range xs {
        args = append(args, x)
    }

removing the need for choice here and attempts at microoptimization by using len(xs) here and there. And as such, could possibly become a recommended response to #7512, maybe even landing in the FAQ.

edit 2: Regarding "implicitness" of the conversion, note that in the above loops (variant b and c), the append already does an implicit conversion of the x from int to interface{}. In my book, keeping the []interface{} phrase closer to the append could make it actually more explicit.

[cznic]

AFAIK, both copy & append are already expected to do an extra allocation ...

copy never allocates.

[akavel]

copy never allocates.

Ouch, right; but the O(n) still stands I believe.

[DeedleFake]

Is it possible to implement a converting copy via generics, as the draft proposal stands now? I can't think of a way to specify that one generic type is convertible to another:

func Convert[T someConstraint, F otherConstraint](f F) T {
  return T(f) // Is it possible to write a constraint to allow this?
}

If this function was possible to write, a converting copy could be implemented somewhere in the standard library, such as a theoretical slices package, and the existing built-in copy() could just handle the simpler variant where no allocation is necessary. Same thing for append(), minus the allocation concerns.

[zigo101]

According to what I know, there is not a simple way to define a constraint to represent "values of F may be converted to T" by the current generic draft.

[ianlancetaylor]

In the current design draft, that is correct. See https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md#no-way-to-express-convertibility .

[bcmills]

@DeedleFake, I examined append and copy under the generics draft in significant depth a while back. See append*.go2 in https://github.com/bcmills/go2go.

[jimmyfrasche]

Now that we have generics there's a new way to break this down.

The one thing generics can't do is express convertibility so add a builtin convert(S, T) which returns a func(T) S that converts values of type T to type S (given t of type T this computes S(t)) and it fails to compile for invalid conversions.

Next add two functions to package slices: AppendMap (or MapAppend?) and CopyMap (MapCopy?). These perform append/copy but with an element-wise mapping.

Using the two together provides the necessary functionality but they are each useful on their own.

[jimmyfrasche]

Filed #71277 for adding the new funcs to the slices package. The convert method seems too special and with #21498 it could be as short as t -> S(t) so I'm undecided on that for now.