introduce LinkSystem #143
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| package multihash | ||
|
|
||
| import ( | ||
| "bytes" | ||
| "crypto/sha256" | ||
| "hash" | ||
| ) | ||
|
|
||
| type identityMultihash struct { | ||
| bytes.Buffer | ||
| } | ||
|
|
||
| func (identityMultihash) BlockSize() int { | ||
| return 32 // A prefered block size is nonsense for the "identity" "hash". An arbitrary but unsurprising and positive nonzero number has been chosen to minimize the odds of fascinating bugs. | ||
| } | ||
|
|
||
| func (x identityMultihash) Size() int { | ||
| return x.Len() | ||
| } | ||
|
|
||
| func (x identityMultihash) Sum(digest []byte) []byte { | ||
| return x.Bytes() | ||
| } | ||
|
|
||
| type doubleSha256 struct { | ||
| main hash.Hash | ||
| } | ||
|
|
||
| func (x doubleSha256) Write(body []byte) (int, error) { | ||
| return x.main.Write(body) | ||
| } | ||
|
|
||
| func (doubleSha256) BlockSize() int { | ||
| return sha256.BlockSize | ||
| } | ||
|
|
||
| func (doubleSha256) Size() int { | ||
| return sha256.Size | ||
| } | ||
|
|
||
| func (x doubleSha256) Reset() { | ||
| x.main.Reset() | ||
| } | ||
|
|
||
| func (x doubleSha256) Sum(digest []byte) []byte { | ||
| intermediate := [sha256.Size]byte{} | ||
| x.main.Sum(intermediate[:]) | ||
| h2 := sha256.New() | ||
| h2.Write(intermediate[:]) | ||
| return h2.Sum(digest) | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| package multihash | ||
|
|
||
| import ( | ||
| "crypto/md5" | ||
| "crypto/sha1" | ||
| "crypto/sha256" | ||
| "crypto/sha512" | ||
| "hash" | ||
| ) | ||
|
|
||
| // Registry is a simple map which maps a multihash indicator number | ||
| // to a standard golang Hash interface. | ||
| // | ||
| // Multihash indicator numbers are reserved and described in | ||
| // https://github.com/multiformats/multicodec/blob/master/table.csv . | ||
| // The keys used in this map must match those reservations. | ||
| // | ||
| // Hashers which are available in the golang stdlib are registered here automatically. | ||
| // | ||
| // Packages which want to register more hashing functions (and have a multihash number reserved!) | ||
| // are encouraged to do so at package init time. | ||
| // (Doing this at package init time ensures this map can be accessed without race conditions.) | ||
| // | ||
| // The linking/cid.DefaultLinkSystem will use this map to find hashers | ||
| // to use when serializing data and computing links, | ||
| // and when loading data from storage and verifying its integrity. | ||
| // | ||
| // This registry map is only used for default behaviors. | ||
| // If you don't want to rely on it, you can always construct your own LinkSystem. | ||
| // (For this reason, there's no special effort made to detect conflicting registrations in this map. | ||
| // If more than one package registers for the same multicodec indicator, and | ||
| // you somehow end up with both in your import tree, and yet care about which wins: | ||
| // then just don't use this registry anymore: make a LinkSystem that does what you need.) | ||
| // This should never be done to make behavior alterations | ||
| // (hash functions are well standardized and so is the multihash indicator table), | ||
| // but may be relevant if one is really itching to try out different hash implementations for performance reasons. | ||
| var Registry = make(map[uint64]func() hash.Hash) | ||
|
There was a problem hiding this comment. The reality is there are a bunch of 3rd party hashes that end up in the current multihash registry in ipfs world and won't conform to hash.Hash How do we expect to support that backwards compatibility? There was a problem hiding this comment. With all the terrible glue code I just wrote. ┐_(ツ)_┌ See the new gadgets in (Like actually, review appreciated. I just utterly invented a BlockSize which is nonsensical. My theory is that it's harmless.) I'll admit, I had kind of a discovery process today. I didn't realize how many things in golang don't conform to At the same time: I think still more things conform to There was a problem hiding this comment. again - isn't the reality that we need to take existing 3rd party codecs currently being registered with mulithash, and be able to keep them working after this transition? not "write a custom shim for each case" but be backwards compatible for them? If that grows to a week+ of work, it's less likely to make it into this refactor. There was a problem hiding this comment. Oh, "3rd party" in the sense of beyond-the-third-party-things-already-directly-depended-on-by-go-mulitcodec. Mmm. Okay, so, I think I had stated earlier that this was impossible, and I think I walk that back. On further look, it seems possible; I just didn't see how at first, because it requires multiple steps, not just the so-called register function in that library. But I still suspect, looking over the flavor of some of the issues in that repo, and how issues and PRs seem to focus on getting new things added to the library, that the number of third-party registrations made in the wild is asymptotically close to zero. So I think at the moment: we can functionally disregard the possibility. And if there are any applications out there which hit a speedbump here, they hit it when converting to go-ipld-prime... which is probably already in their roadmap as a speedbump, so there shouldn't be a category-change in speedbumpism, and so as a result I think everyone involved in such a hypothetical will probably survive. Longer run: yeah, conversation is needed about how to re-reconcile this with go-multihash. I think a bigger venue than this PR is going to be needed for that, though. |
||
|
|
||
| func init() { | ||
| Registry[0x00] = func() hash.Hash { return &identityMultihash{} } | ||
| Registry[0xd5] = md5.New | ||
| Registry[0x11] = sha1.New | ||
| Registry[0x12] = sha256.New | ||
| Registry[0x13] = sha512.New | ||
| // Registry[0x1f] = sha256.New224 // SOON | ||
| // Registry[0x20] = sha512.New384 // SOON | ||
| Registry[0x56] = func() hash.Hash { return &doubleSha256{} } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| /* | ||
| This package has no purpose except to perform registration of mulithashes. | ||
|
|
||
| It is meant to be used as a side-effecting import, e.g. | ||
|
|
||
| import ( | ||
| _ "github.com/ipld/go-ipld-prime/mulithash/register/all" | ||
| ) | ||
|
|
||
| This package registers many multihashes at once. | ||
| Importing it will increase the size of your dependency tree significantly. | ||
| It's recommended that you import this package if you're building some | ||
| kind of data broker application, which may need to handle many different kinds of hashes; | ||
| if you're building an application which you know only handles a specific hash, | ||
| importing this package may bloat your builds unnecessarily. | ||
| */ | ||
| package all | ||
|
|
||
| import ( | ||
| _ "github.com/ipld/go-ipld-prime/multihash/register/blake2" | ||
| _ "github.com/ipld/go-ipld-prime/multihash/register/murmur3" | ||
| _ "github.com/ipld/go-ipld-prime/multihash/register/sha3" | ||
| ) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| /* | ||
| This package has no purpose except to perform registration of multihashes. | ||
|
|
||
| It is meant to be used as a side-effecting import, e.g. | ||
|
|
||
| import ( | ||
| _ "github.com/ipld/go-ipld-prime/mulithash/register/blake2" | ||
| ) | ||
|
|
||
| This package registers several multihashes for the blake2 family | ||
| (both the 's' and the 'b' variants, and in a variety of sizes). | ||
| */ | ||
| package blake2 | ||
|
|
||
| import ( | ||
| "hash" | ||
|
|
||
| "github.com/minio/blake2b-simd" | ||
| "golang.org/x/crypto/blake2s" | ||
|
|
||
| "github.com/ipld/go-ipld-prime/multihash" | ||
| ) | ||
|
|
||
| const ( | ||
| BLAKE2B_MIN = 0xb201 | ||
| BLAKE2B_MAX = 0xb240 | ||
| BLAKE2S_MIN = 0xb241 | ||
| BLAKE2S_MAX = 0xb260 | ||
| ) | ||
|
|
||
| func init() { | ||
| // BLAKE2S | ||
| // This package only enables support for 32byte (256 bit) blake2s. | ||
| multihash.Registry[BLAKE2S_MIN+31] = func() hash.Hash { h, _ := blake2s.New256(nil); return h } | ||
|
|
||
| // BLAKE2B | ||
| // There's a whole range of these. | ||
| for c := uint64(BLAKE2B_MIN); c <= BLAKE2B_MAX; c++ { | ||
| size := int(c - BLAKE2B_MIN + 1) | ||
| multihash.Registry[c] = func() hash.Hash { | ||
| hasher, err := blake2b.New(&blake2b.Config{Size: uint8(size)}) | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
| return hasher | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can this not be inferred from the type?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{not(:) it's an adapter type, not an interface cast.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think i was wanting the apparently deprecated '-1' to indicate default length because a lot of these hashes have a single length that makes sense, and i don't want to be remembering / fumble that sha1 should be 20 bytes while sha224 should be 28 bytes every time i make one of these.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, sorry, I thought the comment was on the diff line, since that's what github highlighted most brightly.
Yeah, agree. I wish the go-cid and/or go-multihash libraries were more friendly about this very common user story.
I think a -1 should flow through and do whatever go-multihash does, still. And I have no idea why that's deprecated, fwiw. (A lot of things in go-multihash seem deprecated without much comment on why or what to do instead. I think some review and renovation of that is overdue.)
It's slightly on the other side of where I'm cordoning my renovation today, though.