Blocks and snapshot compression #1687

keorn · 2016-07-21T16:54:35Z

reduce Compressible interface
check migrations
benchmark effect of blocks DB compression on syncing

…ompressdb

debris · 2016-07-23T12:45:17Z

ethcore/src/account.rs

+    let again_raw = UntrustedRlp::new(&compact_vec).decompress(RlpType::Snapshot);
+    assert_eq!(raw, again_raw.to_vec());
+  }
+


replace spaces with tabs

debris · 2016-07-23T13:11:07Z

util/src/rlp/rlpcompression.rs

+			// Shortest decompressed account is 70, so simply try to swap the value.
+			Ok(ref p) if p.value_len < 70 => simple_swap(),
+			_ => {
+				if let Ok(d) = rlp.data() {


line 109 already proves that rlp is data. You can use expect(...) here or replace line 109 with:

if let Ok(d) = rlp.data() {

*edit, my previous comment may not be valid. is_data does not check validity of data, whereas rlp.data() does. Is rlp expected to be invalid here?

It can be invalid, since data field is not always an RLP.

coveralls · 2016-07-23T13:27:23Z

Coverage decreased (-0.03%) to 75.429% when pulling 5b39dab on blockscompression into aafb014 on master.

debris · 2016-07-23T13:35:31Z

overall LGTM

coveralls · 2016-07-23T14:20:00Z

Coverage decreased (-0.04%) to 75.41% when pulling cede71e on blockscompression into aafb014 on master.

gavofyork · 2016-07-23T15:25:14Z

i guess it's not yet optional?

keorn · 2016-07-23T15:36:37Z

Ah ok, will make it optional for blocks.

gavofyork · 2016-07-27T10:01:12Z

ok looks good. actually results in a ~ 8% speedup for the first 200k blocks.

keorn · 2016-07-27T10:12:50Z

I got some speedup on full sync too.
Master:

real    405m49.204s
user    587m33.366s
sys     27m45.241s

With blocks compression:

real    395m6.620s
user    557m51.039s
sys     25m58.350s

Conflicts: ethcore/src/blockchain/blockchain.rs

coveralls · 2016-07-27T13:05:48Z

Coverage decreased (-0.3%) to 86.904% when pulling c220631 on blockscompression into c65ee93 on master.

keorn added 30 commits July 1, 2016 11:34

new Compressible rlp trait

6f6bfb6

make compressed rlp iterable

a53bc83

invalid rlp slice swapper

7abe09a

switch compress to swapper, add reverse swapper test case

31cde1f

add basic account compression test

bac3060

add new rlp trait

30cceaf

new Compressible rlp trait

c0c3e8d

make compressed rlp iterable

a7b44da

invalid rlp slice swapper

d31a154

switch compress to swapper, add reverse swapper test case

72b8679

new Compressible rlp trait

6478c20

make compressed rlp iterable

e88f925

invalid rlp slice swapper

ab17d8a

switch compress to swapper, add reverse swapper test case

8499190

add account compress/ decompress test

8fc8412

make compressor cleaner, use hashmaps for swapper

25fb657

Merge branch 'master' of https://github.com/ethcore/parity into nullc…

ea8ac9f

…ompressdb

improve compression tests

6fcb49d

add a DecompressingDecoder, change Decoder to take refernce

ae616ef

separate rlp compression related stuff

05a4626

new Compressible rlp trait

95429c7

make compressed rlp iterable

2d1c3fd

invalid rlp slice swapper

0e12f73

switch compress to swapper, add reverse swapper test case

67d50ba

add basic account compression test

4a43590

add new rlp trait

fe16017

new Compressible rlp trait

54cc631

make compressed rlp iterable

e833d12

invalid rlp slice swapper

298c13f

switch compress to swapper, add reverse swapper test case

ce0dcc9

keorn removed the A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. label Jul 22, 2016

Merge branch 'master' into blockscompression

d4ddfbb

debris reviewed Jul 23, 2016
View reviewed changes

move tests to tests modules

5b39dab

debris reviewed Jul 23, 2016
View reviewed changes

remove redundant &

cede71e

keorn added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jul 23, 2016

gavofyork added A8-looksgood 🦄 Pull request is reviewed well. and removed A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. labels Jul 27, 2016

keorn added A7-looksgoodcantmerge 🙄 Pull request is reviewed well, but cannot be merged due to conflicts. and removed A8-looksgood 🦄 Pull request is reviewed well. labels Jul 27, 2016

Merge branch 'master' into blockscompression

c220631

Conflicts: ethcore/src/blockchain/blockchain.rs

keorn added A8-looksgood 🦄 Pull request is reviewed well. and removed A7-looksgoodcantmerge 🙄 Pull request is reviewed well, but cannot be merged due to conflicts. labels Jul 27, 2016

keorn closed this Jul 27, 2016

keorn reopened this Jul 27, 2016

gavofyork merged commit 02cf486 into master Jul 27, 2016

gavofyork deleted the blockscompression branch July 27, 2016 15:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blocks and snapshot compression #1687

Blocks and snapshot compression #1687

keorn commented Jul 21, 2016 •

edited

Loading

debris Jul 23, 2016

debris Jul 23, 2016 •

edited

Loading

debris Jul 23, 2016

keorn Jul 23, 2016

coveralls commented Jul 23, 2016 •

edited

Loading

debris commented Jul 23, 2016

coveralls commented Jul 23, 2016 •

edited

Loading

gavofyork commented Jul 23, 2016

keorn commented Jul 23, 2016

gavofyork commented Jul 27, 2016

keorn commented Jul 27, 2016

coveralls commented Jul 27, 2016 •

edited

Loading

Blocks and snapshot compression #1687

Blocks and snapshot compression #1687

Conversation

keorn commented Jul 21, 2016 • edited Loading

debris Jul 23, 2016

Choose a reason for hiding this comment

debris Jul 23, 2016 • edited Loading

Choose a reason for hiding this comment

debris Jul 23, 2016

Choose a reason for hiding this comment

keorn Jul 23, 2016

Choose a reason for hiding this comment

coveralls commented Jul 23, 2016 • edited Loading

debris commented Jul 23, 2016

coveralls commented Jul 23, 2016 • edited Loading

gavofyork commented Jul 23, 2016

keorn commented Jul 23, 2016

gavofyork commented Jul 27, 2016

keorn commented Jul 27, 2016

coveralls commented Jul 27, 2016 • edited Loading

keorn commented Jul 21, 2016 •

edited

Loading

debris Jul 23, 2016 •

edited

Loading

coveralls commented Jul 23, 2016 •

edited

Loading

coveralls commented Jul 23, 2016 •

edited

Loading

coveralls commented Jul 27, 2016 •

edited

Loading