Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String conversion from decimal (#108) #122

Merged
merged 51 commits into from
Feb 8, 2023
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
bb5f382
relaxed marshaling, slow base 10 parsing
Sep 25, 2022
fb12295
update version
Sep 25, 2022
6866965
add control to benchmark
Sep 25, 2022
55f7ffc
make the ci thing less angry
Sep 25, 2022
5689315
make the ci even less angy
Sep 25, 2022
3c7131f
add fuzz tests
elee1766 Nov 28, 2022
5a106b0
change map
elee1766 Nov 28, 2022
b296df7
magic
elee1766 Nov 28, 2022
cb2c910
clear z on SetFrombig, even if b is nil
elee1766 Nov 28, 2022
df6b926
Update conversion.go
elee1766 Dec 6, 2022
bd117d9
Update conversion.go
Tjudice Dec 14, 2022
048efc0
Merge pull request #1 from Tjudice/patch-1
elee1766 Dec 14, 2022
9a76ab7
base10: minor api change, added docs
holiman Dec 15, 2022
42313ad
ci, go.mod: require go 1.18+
holiman Dec 15, 2022
7e0467a
squashme: circle fixes
holiman Dec 15, 2022
7f61ee9
properly cgheck for overflow in fuzz
elee1766 Dec 15, 2022
1d8f726
number overflow test
elee1766 Dec 15, 2022
f4e6c19
added comments to explain fromBase10Long
elee1766 Dec 15, 2022
84510f7
Update base10.go
elee1766 Dec 15, 2022
6e4f914
Update base10.go
elee1766 Dec 15, 2022
e25ce45
lift up
elee1766 Dec 15, 2022
8420f46
change cutLength to local constant
elee1766 Dec 15, 2022
3fc5f08
better doc
elee1766 Dec 15, 2022
d6f32f6
base10: modify algorithm + fixup tests a bit
holiman Dec 15, 2022
8c858a1
base10: fuzzing found some bugs, which were fixed
holiman Dec 15, 2022
078a8e6
fuzzing: some improvements to the fuzzer
holiman Dec 15, 2022
2413444
add oss-fuzz fuzzer, rename base10 to decimal
holiman Dec 15, 2022
2e6c49f
decimal: fix conversion on base 0, fallback to big.Int
holiman Dec 16, 2022
765b02b
decimal: move some unused constants
holiman Dec 16, 2022
3430ee5
more failing tests
holiman Dec 16, 2022
cbdcdcb
decimal: remove SetString method
holiman Dec 16, 2022
38d53f3
decimal: simplify tests
holiman Dec 16, 2022
ca9a5ea
rm unused constants, fix benchmarks
holiman Dec 16, 2022
9518831
Update decimal.go
elee1766 Dec 17, 2022
ad5ba70
Update conversion.go
elee1766 Dec 17, 2022
301af32
change sql valuer
elee1766 Dec 20, 2022
b5a4906
remove extra struct def
elee1766 Dec 29, 2022
fb7ceef
testing: try to get 100% coverage again
holiman Dec 29, 2022
4ef03d7
lintfix
holiman Dec 29, 2022
5587dd6
fuzzing: fix up fuzzer for string conversion
holiman Dec 29, 2022
7ff0d8f
circle: try to get circleci fuzzing going
holiman Dec 29, 2022
5019647
conversion: more coverage + fix in Scan
holiman Dec 29, 2022
2f0b0b7
decimal_test: more coverage
holiman Dec 29, 2022
dcd75fd
go.mod: fuzzing dep
holiman Dec 29, 2022
0472e9b
conversion: consistent use of ptr receiver
holiman Dec 29, 2022
d55274d
circle: make use of restored corpus
holiman Dec 29, 2022
82fd325
properly parse scientific notation
elee1766 Jan 12, 2023
35d62f2
add some overflow cases for scan, and error on overflow
elee1766 Jan 27, 2023
88226a1
conversion minor nitpicks
holiman Feb 8, 2023
5ff06b0
conversion: fix test
holiman Feb 8, 2023
091c3f9
conversion_test: bring coverage back to 100
holiman Feb 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions base10.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
package uint256

import (
"strconv"
"strings"
)

const twoPow256Sub1 = "115792089237316195423570985008687907853269984665640564039457584007913129639935"
const twoPow128 = "340282366920938463463374607431768211456"
const twoPow64 = "18446744073709551616"

func (z *Int) Base10() string {
return z.ToBig().String()
}

// SetString implements a subset of (*big.Int).SetString
// ok will be true iff i == nil
func (z *Int) SetString(s string, base int) (i *Int, ok bool) {
switch base {
case 0:
if strings.HasPrefix(s, "0x") {
err := z.fromHex(s)
if err != nil {
return nil, false
}
return z, true
}
err := z.FromBase10(s)
if err != nil {
return nil, false
}
return z, true
case 10:
err := z.FromBase10(s)
if err != nil {
return nil, false
}
return z, true
case 16:
err := z.fromHex(s)
if err != nil {
return nil, false
}
return z, true
}
return nil, false
}
func (z *Int) FromBase10(s string) (err error) {
if len(s) < len(twoPow256Sub1) {
return z.fromBase10Long(s)
}
if len(s) == len(twoPow256Sub1) {
if s > twoPow256Sub1 {
return ErrBig256Range
}
return z.fromBase10Long(s)
}
return ErrBig256Range
}

var scaleTable10 [78]Int
elee1766 marked this conversation as resolved.
Show resolved Hide resolved

func init() {
for k := range scaleTable10 {
scaleTable10[k].Exp(NewInt(10), NewInt(uint64(k)))
}
}

func (z *Int) fromBase10Long(bs string) error {
z[0] = 0
z[1] = 0
z[2] = 0
z[3] = 0
if bs == "" {
return nil
}
iv := 19
c := 0
if len(bs) >= (iv * 4) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a bit of description what happens inside this method -- what algorithm you're following?
Doesn't have to be a lot, but a brief description.
Also, what does iv mean? I associate the term with cryptographic input vector, but this is probably something else?
Also, is the unrolled loop really worth it? If there's no tangible gain, I'd prefer to have it in a shorter form

Copy link

@gfxlabs gfxlabs Dec 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol. i think iv originally meant "index value" or something stupid like that.

I've cleaned up the function and added some comments - let me know if it's still unclear

nm, err := strconv.ParseUint(bs[c:(c+iv)], 10, 64)
if err != nil {
return ErrSyntaxBase10
}
z.Add(z, new(Int).Mul(&scaleTable10[len(bs)-iv], NewInt(uint64(nm))))
c = c + iv
}
if len(bs) >= (iv * 3) {
nm, err := strconv.ParseUint(bs[c:(c+iv)], 10, 64)
if err != nil {
return ErrSyntaxBase10
}
z.Add(z, new(Int).Mul(&scaleTable10[len(bs)-c-iv], NewInt(uint64(nm))))
c = c + iv
}
if len(bs) >= (iv * 2) {
nm, err := strconv.ParseUint(bs[c:(c+iv)], 10, 64)
if err != nil {
return ErrSyntaxBase10
}
z.Add(z, new(Int).Mul(&scaleTable10[len(bs)-c-iv], NewInt(uint64(nm))))
c = c + iv
}
if len(bs) >= (iv * 1) {
nm, err := strconv.ParseUint(bs[c:(c+iv)], 10, 64)
if err != nil {
return ErrSyntaxBase10
}
z.Add(z, new(Int).Mul(&scaleTable10[len(bs)-c-iv], NewInt(uint64(nm))))
c = c + iv
}
if len(bs) == c {
return nil
}
nm, err := strconv.ParseUint(bs[c:], 10, 64)
if err != nil {
return ErrSyntaxBase10
}
z.AddUint64(z, uint64(nm))
return nil
}
266 changes: 266 additions & 0 deletions base10_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,266 @@
package uint256

import (
"errors"
"fmt"
"math/big"
"testing"
)

func TestStringScanBase10(t *testing.T) {
z := new(Int)

type testCase struct {
i string
err error
val string
}

cases := []testCase{
{i: twoPow256Sub1 + "1", err: ErrBig256Range},
{i: "2" + twoPow256Sub1[1:], err: ErrBig256Range},
{i: twoPow256Sub1[1:]},
{i: twoPow128},
{i: twoPow128 + "1"},
{i: twoPow128[1:]},
{i: twoPow64 + "1"},
{i: twoPow64[1:]},
elee1766 marked this conversation as resolved.
Show resolved Hide resolved
{i: "banana", err: ErrSyntaxBase10},
{i: "0xab", err: ErrSyntaxBase10},
{i: "ab", err: ErrSyntaxBase10},
{i: "0"},
{i: "000", val: "0"},
{i: "010", val: "10"},
{i: "01", val: "1"},
{i: "-0", err: ErrSyntaxBase10},
{i: "-10", err: ErrSyntaxBase10},
{i: "115792089237316195423570985008687907853269984665640564039457584007913129639936", err: ErrBig256Range},
{i: "115792089237316195423570985008687907853269984665640564039457584007913129639935"},
}

for _, v := range cases {
err := z.FromBase10(v.i)
if !errors.Is(err, v.err) {
t.Errorf("expect err %s, got %s", v.err, err)
}
if err == nil {
got := z.ToBig().String()
want := v.i
if v.val != "" {
want = v.val
}
if got != want {
t.Errorf("expect val %s, got %s", v.i, got)
}
}
}
}

func FuzzBase10StringCompare(f *testing.F) {
bi := new(big.Int)
z := new(Int)
max256 := new(Int)
max256.FromBase10(twoPow256Sub1)
testcase := []string{
twoPow256Sub1 + "1",
"2" + twoPow256Sub1[1:],
twoPow256Sub1,
twoPow128,
twoPow128,
twoPow128,
twoPow64,
twoPow64,
"banana",
"0xab",
"ab",
"0",
"000",
"010",
"01",
"-0",
"-10",
"115792089237316195423570985008687907853269984665640564039457584007913129639936",
"115792089237316195423570985008687907853269984665640564039457584007913129639935",
"apple",
"04112401274120741204712xxxxxz00",
"0x10101011010",
"熊熊熊熊熊熊熊熊",
}
for _, tc := range testcase {
f.Add(tc)
}
f.Fuzz(func(t *testing.T, orig string) {
err := z.FromBase10(orig)
val, ok := bi.SetString(orig, 10)
// if fail, make sure that we failed too
if !ok {
if err == nil {
t.Errorf("expected base 10 parse to fail: %s", orig)
}
return
}
// if its negative number, we should err
if len(orig) > 0 && (orig[0] == '-') {
if !errors.Is(err, ErrSyntaxBase10) {
t.Errorf("should have errored at negative number: %s", orig)
}
return
}
// if its too large, ignore it also
if val.Cmp(max256.ToBig()) > 0 {
return
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If >256, we should have a non-nil err. Should check that

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this

}
// so here, if it errors, it means that we failed
if err != nil {
t.Errorf("should have parsed %s to %s, but err'd instead", orig, val.String())
return
}
// otherwise, make sure that the values are equal
if z.ToBig().Cmp(bi) != 0 {
t.Errorf("should have parsed %s to %s, but got %s", orig, bi.String(), z.Base10())
return
}
// make sure that bigint base 10 string is equal to base10 string
if z.Base10() != bi.String() {
t.Errorf("should have parsed %s to %s, but got %s", orig, bi.String(), z.Base10())
return
}
value, err := z.Value()
if err != nil {
t.Errorf("fail to Value() %s, got err %s", val, err)
return
}
if z.Base10()+"e0" != fmt.Sprint(value) {
t.Errorf("value of %s did not match base 10 encoding %s", value, z.Base10())
return
}
})
}

func BenchmarkStringBase10BigInt(b *testing.B) {
val := new(big.Int)
bytearr := twoPow256Sub1
b.Run("generic", func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
val.SetString(bytearr[:2], 10)
val.SetString(bytearr[:4], 10)
val.SetString(bytearr[:6], 10)
val.SetString(bytearr[:8], 10)
val.SetString(bytearr[:16], 10)
val.SetString(bytearr[:12], 10)
val.SetString(bytearr[:14], 10)
val.SetString(bytearr[:16], 10)
val.SetString(bytearr[:18], 10)
val.SetString(bytearr[:20], 10)
val.SetString(bytearr[:22], 10)
val.SetString(bytearr[:24], 10)
val.SetString(bytearr[:26], 10)
val.SetString(bytearr[:28], 10)
val.SetString(bytearr[:30], 10)
val.SetString(bytearr[:32], 10)
val.SetString(bytearr[:34], 10)
val.SetString(bytearr[:36], 10)
val.SetString(bytearr[:38], 10)
val.SetString(bytearr[:40], 10)
val.SetString(bytearr[:42], 10)
val.SetString(bytearr[:44], 10)
val.SetString(bytearr[:46], 10)
val.SetString(bytearr[:48], 10)
val.SetString(bytearr[:50], 10)
val.SetString(bytearr[:52], 10)
val.SetString(bytearr[:54], 10)
val.SetString(bytearr[:56], 10)
val.SetString(bytearr[:58], 10)
val.SetString(bytearr[:60], 10)
val.SetString(bytearr[:62], 10)
val.SetString(bytearr[:64], 10)
}
})
}

func BenchmarkStringBase10(b *testing.B) {
val := new(Int)
bytearr := twoPow256Sub1
b.Run("generic", func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
val.SetString(bytearr[:2], 10)
val.SetString(bytearr[:4], 10)
val.SetString(bytearr[:6], 10)
val.SetString(bytearr[:8], 10)
val.SetString(bytearr[:16], 10)
val.SetString(bytearr[:12], 10)
val.SetString(bytearr[:14], 10)
val.SetString(bytearr[:16], 10)
val.SetString(bytearr[:18], 10)
val.SetString(bytearr[:20], 10)
val.SetString(bytearr[:22], 10)
val.SetString(bytearr[:24], 10)
val.SetString(bytearr[:26], 10)
val.SetString(bytearr[:28], 10)
val.SetString(bytearr[:30], 10)
val.SetString(bytearr[:32], 10)
val.SetString(bytearr[:34], 10)
val.SetString(bytearr[:36], 10)
val.SetString(bytearr[:38], 10)
val.SetString(bytearr[:40], 10)
val.SetString(bytearr[:42], 10)
val.SetString(bytearr[:44], 10)
val.SetString(bytearr[:46], 10)
val.SetString(bytearr[:48], 10)
val.SetString(bytearr[:50], 10)
val.SetString(bytearr[:52], 10)
val.SetString(bytearr[:54], 10)
val.SetString(bytearr[:56], 10)
val.SetString(bytearr[:58], 10)
val.SetString(bytearr[:60], 10)
val.SetString(bytearr[:62], 10)
val.SetString(bytearr[:64], 10)
}
})
}
func BenchmarkStringBase16(b *testing.B) {
val := new(Int)
bytearr := "aaaa12131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f3031bbbb"
b.Run("generic", func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
val.SetString(bytearr[:1], 16)
val.SetString(bytearr[:2], 16)
val.SetString(bytearr[:3], 16)
val.SetString(bytearr[:4], 16)
val.SetString(bytearr[:5], 16)
val.SetString(bytearr[:6], 16)
val.SetString(bytearr[:7], 16)
val.SetString(bytearr[:8], 16)
val.SetString(bytearr[:9], 16)
val.SetString(bytearr[:10], 16)
val.SetString(bytearr[:11], 16)
val.SetString(bytearr[:12], 16)
val.SetString(bytearr[:13], 16)
val.SetString(bytearr[:14], 16)
val.SetString(bytearr[:15], 16)
val.SetString(bytearr[:16], 16)
val.SetString(bytearr[:17], 16)
val.SetString(bytearr[:18], 16)
val.SetString(bytearr[:19], 16)
val.SetString(bytearr[:20], 16)
val.SetString(bytearr[:21], 16)
val.SetString(bytearr[:22], 16)
val.SetString(bytearr[:23], 16)
val.SetString(bytearr[:24], 16)
val.SetString(bytearr[:25], 16)
val.SetString(bytearr[:26], 16)
val.SetString(bytearr[:27], 16)
val.SetString(bytearr[:28], 16)
val.SetString(bytearr[:29], 16)
val.SetString(bytearr[:20], 16)
val.SetString(bytearr[:31], 16)
val.SetString(bytearr[:32], 16)
}
})
}
Loading