Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/storage/mkvs: Use cbor.UnmarshalTrusted for internal metadata #2800

Merged
merged 2 commits into from
Apr 1, 2020
Merged

go/storage/mkvs: Use cbor.UnmarshalTrusted for internal metadata #2800

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
05e3030
go/common/cbor: Add UnmarshalTrusted for trusted inputs
kostko Apr 1, 2020
ffa4115
go/storage/mkvs: Use cbor.UnmarshalTrusted for internal metadata
kostko Apr 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changelog/2800.internal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
go/common/cbor: Add UnmarshalTrusted for trusted inputs

The new method relaxes some decoding restrictions for cases where the inputs
are trusted (e.g., because they are known to be generated by the local node
itself).
27 changes: 25 additions & 2 deletions go/common/cbor/cbor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,23 @@ var (
TagsMd: cbor.TagsForbidden,
}

// decOptions are decoding options for UNTRUSTED inputs (used by default).
decOptions = cbor.DecOptions{
DupMapKey: cbor.DupMapKeyEnforcedAPF,
IndefLength: cbor.IndefLengthForbidden,
TagsMd: cbor.TagsForbidden,
}

encMode cbor.EncMode
decMode cbor.DecMode
// decOptionsTrusted are decoding options for TRUSTED inputs. They are only used when explicitly
// requested by using the UnmarshalTrusted method.
decOptionsTrusted = cbor.DecOptions{
MaxArrayElements: 134217728, // Maximum allowed.
MaxMapPairs: 134217728, // Maximum allowed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the defaults for these on the untrusted decOptions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default is 131072. Maybe we should consider making this explicit and/or even changing it (e.g., lowering it) for untrusted settings?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eh, maybe a comment if anything

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kostko hello. I have an issue with calling StateToGenesis beacon api: rpc error: code = Internal desc = grpc: failed to unmarshal the received message cbor: exceeded max number of key-value pairs 131072 for CBOR map at block 7188323. How I can explicitly call UnmarshalTrusted method at this point?

}

encMode cbor.EncMode
decMode cbor.DecMode
decModeTrusted cbor.DecMode
)

func init() {
Expand All @@ -45,6 +54,9 @@ func init() {
if decMode, err = decOptions.DecMode(); err != nil {
panic(err)
}
if decModeTrusted, err = decOptionsTrusted.DecMode(); err != nil {
panic(err)
}
}

// FixSliceForSerde will convert `nil` to `[]byte` to work around serde
Expand Down Expand Up @@ -74,6 +86,17 @@ func Unmarshal(data []byte, dst interface{}) error {
return decMode.Unmarshal(data, dst)
}

// UnmarshalTrusted deserializes a CBOR byte vector into a given type.
//
// This method MUST ONLY BE USED FOR TRUSTED INPUTS as it relaxes some decoding restrictions.
func UnmarshalTrusted(data []byte, dst interface{}) error {
if data == nil {
return nil
}

return decModeTrusted.Unmarshal(data, dst)
}

// MustUnmarshal deserializes a CBOR byte vector into a given type.
// Panics if unmarshal fails.
func MustUnmarshal(data []byte, dst interface{}) {
Expand Down