pkg/**: stdlib and builtin expansion

[verdverm]

Generally, we would like to see the standard library expanded.

At this point I am starting a conversation about the topic. We can change this header as things become better understood.

Thoughts:

• More functions from go-stdlib, these seem like low hanging fruit
• both pure / tools portions, how to separate (document this, where things belong), want to expand both most likely
• builtins for implementing structural func's
• builtins for things discussed around the evaluator rewrite and recursion
• external libs or packages, how do we decide what / when to bring in deps to cue?
• should we continue to generate the builtins as is today? or do we need to consider something that scales with the increased size of the stdlib?

Another thought is...

Can we provided a way for users to register their own builtins?

iirc, the way builtins are happened, this might not be overly difficult. There is the issue (still to be created) that fields named the same as builtin funcs are sorted before other fields in a struct.

---

Functions and Packages:

From Go, for pkg/tool

• archive/*
• compress/*
• database (I can see people wanting to do something like an http call here)
• something like a file lock or concurrent access for commands
• more of / from os/, net/, file / filesystem / walktree
• something for logging
• support for readers / writers to chain other funcs and reduce overhead, something like Node/JS pipes
• better filesystem support, globs, in memory FS

From Go, for pure cue pkg expansion

• strings (not sure if there is a gap for this one and the next)
• strconv
• path.Join
• text/template (expose more of the interface, want to register partials for example)

External, for pure cue pkg expansion

• strings, some additions like Kace or a custom written version of the one or two missing funcs
• semver, as configurations are often dealing with these
• structural, object semantics based [diff,patch,merge,pick,filter]

Moonshot:

• access to cue runtime internals, like if I want to walk cue instances / values as defined in cue.Value rather than the exported / data forms.

I can see writing a cue tool which has a schema with it, and then processing local cue packages with the tool. So there may be a need to load and process other cue files from within the tool portion of a cue "program"

[mikelnrd]

Can we provided a way for users to register their own builtins?

I came here to request this. 

I've had some recent success implementing some existing kubernetes tooling using cue. The existing tooling involves hashing, and I am unable to choose the hash function as it needs to match the existing go implementation for compatibility reasons. 

I've done this twice recently (currently in love with cue, thanks a lot for the hard work!):

In the first case the hash function used was sha256 which is already in the cue stdlib, so I was able to complete the implementation in cue successfully.

But in the second case the hash function is https://golang.org/pkg/hash/fnv/ which sadly isn't in cue's stdlib. It'd be great if you'd accept a pull request to add it (?), but more generally I'd love to able to define my own functions like this (understanding, a bit like with Bazel, that they need to be hermetic & deterministic).

Thought I'd post this here as an end-user experience report in case it's helpful.

..................................................

PS: Here's the kubernetes tooling I'm working with by the way:

hash/fnv is used in kubernetes-sigs/cli-utils (which is used by google's new kpt tool):

https://github.com/kubernetes-sigs/cli-utils/blob/f18d7c260fb3f5d3a963088159332a3081cdd5f5/pkg/apply/prune/inventory.go#L263

and here's my cue code (albeit using the wrong hash function):

_hash: hex.Encode(strings.ByteSlice(sha256.Sum256(strings.Join(list.SortStrings(_list), "")), 0, 4))

[verdverm]

Same comment about user defined builtins: #423 (comment)

[apstndb]

I have also noticed that to generate JWT in CUE is hard because there are no crypto/rsa package.
It requires exec.Run with sign helper command for workaround.
https://github.com/apstndb/cue-sandbox/blob/7d252af7b9f3665243fa319b54d5cbb3ebc68ebc/jwt/jwt.cue#L21-L33

[verdverm]

CUE is hermetic and I'm not sure secure crypto generation aligns with that goal

[apstndb]

Yeah, generally crypto/rsa is not hermetic but SignPKCS1v15(without RSA blinding) and VerifyPKCS1v15 doesn't rely on RNG so they can be hermetic.
If they are supported, CUE can process JWTs.

[verdverm]

SignPKCS1v15 (without RSA blinding)

• The Go function takes a random reader, how do you propose getting around this?
• Does allowing this create weakened security for the sake of doing something with CUE?

[verdverm]

Of note, I just noticed tou are working in the non-hermetic area of the cue cli which has the tooling layer on top of CUE. So it could be feasible to add the crypto functions there, though I wonder how one might select a different source of randomness from outside of the go stdlib (i.e. something like Cloudflare using lava lamps)

[cueckoo]

This issue has been migrated to cue-lang/cue#348.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.