Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor hashBoinc for binary claim contexts #1558

Merged

Conversation

cyrossignol
Copy link
Member

@cyrossignol cyrossignol commented Oct 8, 2019

The hashBoinc field of a coinbase transaction stores delimited strings of data that contain the context that augments each generated block. These changes refactor the storage of this data to allow for binary serialization of the claim contexts.

Based on discussions in Slack, this PR moves the storage of claim contexts from the coinbase transaction to the block itself to facilitate submission of much larger superblock data than a transaction allows. This accommodation prepares the protocol for the removal of the team requirement and a greater number of users. The application begins to use the binary serialization format after the next mandatory threshold. This format significantly reduces the size overhead of the claim context and eliminates issues with string-based serialization.

To further reduce the size of claim data, I removed the following legacy fields:

  • last payment time
  • last block hash
  • last PoR block hash
  • research age
  • average magnitude
  • public key

Although it provides an easy-to-access historical log, this data can be reproduced from blockchain data, so it's not necessary to store it in every block. We can also remove the following informational fields from the claim context, but we agreed to keep them in the short-term for explorers:

  • block subsidy (interest/CBR)
  • research subsidy
  • magnitude
  • magnitude unit

The new NN::Claim class replaces MiningCPID and abstracts the related functionality. The changes here continue clean-up work from #1480 by removing the pervasive use of the GlobalCPUMiningCPID state, and RPC methods now produce structured JSON data for claims and superblocks. Claims now limit the size of version and organization fields to 30 and 50 characters for version 11 blocks. This includes a fix for #525 that enables researchers to stake investor-only blocks with no beacon or one that expired.

A subsequent pull request will hook-up the new NN::Superblock class to the main block acceptance and research reward code.

@cyrossignol cyrossignol force-pushed the superblock-claim branch 3 times, most recently from 10f2ac8 to 4c84b62 Compare October 8, 2019 18:01
@jamescowens
Copy link
Member

I am requesting all three other active core devs to look at this due to the importance of the changes.

@cyrossignol
Copy link
Member Author

Thanks, I have a couple more tweaks to squash and push.

@cyrossignol cyrossignol force-pushed the superblock-claim branch 2 times, most recently from 17c8559 to 6d26605 Compare October 9, 2019 06:19
@cyrossignol cyrossignol marked this pull request as ready for review October 9, 2019 06:30
@jamescowens jamescowens added this to the Fern milestone Oct 13, 2019
Copy link
Member

@jamescowens jamescowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow. This looks good. How much testing have you done on this?

@cyrossignol
Copy link
Member Author

@jamescowens Sync smoke tests on mainnet and testnet completed without issue, and I staked blocks as a researcher and investor. Real testing starts after v11 switch-over.

@cyrossignol
Copy link
Member Author

Rebased and added check for superblock size.

@cyrossignol
Copy link
Member Author

Added a check for zero research reward when staking a block--in the same vein as 6d3cbf8, this switches the block to an investor stake so we don't add unnecessary CPID data to the chain. The research age system needs to scan every research reward block at startup, including blocks with zero research reward. By omitting the CPID from the block, the index scan can skip the block at startup without needing to double-check the research reward amount.

Copy link
Member

@iFoggz iFoggz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After an hour reviewing this code + 2 sandwiches and a tea. I can say there is a lot going on here and this is a big change and a lot of refactoring to accommodate the change. However I did not see anything that pops out that concerns me. I approve to push ahead for more wide scale testing on testnet and see how well this performs for many users. a set bv11 start should be setup and testnet should be upgraded before hand. Good work @cyrossignol, These are changes that need to be done to improve the overall quality of life and expansion of Gridcoin into the future.

@jamescowens
Copy link
Member

As a general comment, tremendous work on this!

@jamescowens
Copy link
Member

jamescowens commented Oct 20, 2019

Ok. Only one unresolved to address and we can merge this.

Fixes gridcoin-community#525

If a researcher's beacon expired, generate the block as an investor. We
cannot sign a research claim without the beacon key, so this avoids the
issue that prevents a researcher from staking blocks if the beacon does
not exist (it expired or it has yet to be advertised).
This allows generated blocks greater than MAX_BLOCK_SIZE only when those
contain a superblock. The superblock must serialize on the network to at
most Superblock::MAX_SIZE bytes. The rest of the block must hold at most
MAX_BLOCK_SIZE bytes still.
If no pending research subsidy value exists, build an investor claim.
This avoids polluting the block index with non-research reward blocks
that contain CPIDs which increases the effort needed to load research
age context at start-up.
@cyrossignol
Copy link
Member Author

Rebased to fix an unused argument in ComputeResearchAccrual() added by #1569.

@jamescowens
Copy link
Member

Merging

@jamescowens jamescowens merged commit e88643b into gridcoin-community:development Oct 20, 2019
@cyrossignol cyrossignol deleted the superblock-claim branch February 25, 2020 05:46
jamescowens added a commit that referenced this pull request Sep 4, 2020
Added
 - Backport newer uint256 types from Bitcoin #1570 (@cyrossignol)
 - Implement project level rain for rainbymagnitude #1580 (@jamescowens)
 - Upgrade utilities (Update checker and snapshot downloader/application) #1576 (@iFoggz)
 - Provide fees collected in the block by the miner #1601 (@iFoggz)
 - Add support for generating legacy superblocks from scraper stats #1603 (@cyrossignol)
 - Port of the Bitcoin Logger to Gridcoin #1600 (@jamescowens)
 - Implement zapwallettxes #1605 (@jamescowens)
 - Implements a global event filter to suppress help question mark #1609 (@jamescowens)
 - Add next target difficulty to RPC output #1615 (@cyrossignol)
 - Add caching for block hashes to CBlock #1624 (@cyrossignol)
 - Make toolbars and tray icon red for testnet #1637 (@jamescowens)
 - Add an rpc call convergencereport #1643 (@jamescowens)
 - Implement newline filter on config file read in #1645 (@jamescowens)
 - Implement beacon status icon/button #1646 (@jamescowens)
 - Add gridcointestnet.png #1649 (@caraka)
 - Add precision to support magnitudes less than 1 #1651 (@cyrossignol)
 - Replace research accrual calculations with superblock snapshots #1657 (@cyrossignol)
 - Publish example gridcoinresearch.conf as a md document to the doc directory #1662 (@jamescowens)
 - Add options checkbox to disable transaction notifications #1666 (@jamescowens)
 - Add support for self-service beacon deletion #1695 (@cyrossignol)
 - Add support for type-specific contract fee amounts #1698 (@cyrossignol)
 - Add verifiedbeaconreport and pendingbeaconreport #1696 (@jamescowens)
 - Add preliminary testing option for block v11 height on testnet #1706 (@cyrossignol)
 - Add verified beacons manifest part to superblock validator #1711 (@cyrossignol)
 - Implement beacon, vote, and superblock display categories/icons in UI transaction model #1717 (@jamescowens)
 - neuralnet: Add integrity checking to researcher accrual snapshot registry #1727 (@jamescowens)
 - Add workaround for scrypt assembly on macOS #1740 (@cyrossignol)
 - gui: Build onboarding/beacon wizard #1739 (@cyrossignol)
 - doc: Add CONTRIBUTING.md from bitcoin #1723 (@div72)
 - rpc: Implement inspectaccrualsnapshot and parseaccrualsnapshotfile #1744 (@jamescowens)
 - scraper: Add disk based state backing for verified beacon list in scraper #1751 (@jamescowens)
 - Add ability to recover beacon in block version 11+ #1768 (@cyrossignol)
 - refactor: Add transaction context to contract handlers #1777 (@cyrossignol)
 - gui: Add context for when BOINC is attached to a pool #1775 (@cyrossignol)
 - doc: Clarify what to do if PR in multiple categories (for CONTRIBUTING.md) #1798 (@RoboticMind)
 - qt: Add option to choose not to start the wallet minimized #1804 (@jamescowens)
 - superblock: Add check for OutOfSyncByAge to SuperblockValidator::Validate #1806 (@jamescowens)
 - contract: Standardize contract validation and add block context #1808 (@cyrossignol)
 - add seed.gridcoin.pl to default config #1812 (@wilkart)
 - gui: Implement sidestake send display #1813 (@jamescowens)
 - gui: Add pool/investor pages to researcher wizard #1819 (@cyrossignol)
 - ci: Port lint scripts from Bitcoin #1823 (@div72)
 - doc: Create basic readme in contrib #1826 (@RoboticMind)
 - gui: Implement TransactionRecord::Message #1829 (@jamescowens)
 - rpc: Add private_key_available to beaconstatus #1833 (@a123b)
 - gui: Validate email address in researcher wizard #1840 (@a123b)
 - rpc: Add "getrawwallettransaction" RPC function #1842 (@cyrossignol)
 - consensus: Set block version 11 threshold height for mainnet #1862 (@cyrossignol)

Changed
 - Upgrade LevelDB from v1.17 to v1.20 #1562 (@cyrossignol)
 - Re-enable scrypt optimizations #1450 (@denravonska)
 - Derive CScript from prevector type (optimization) #1554 (@cyrossignol)
 - Disable quorum for grandfathered blocks to speed up sync #1568 (@cyrossignol)
 - Refactor hashBoinc for binary claim contexts #1558 (@cyrossignol)
 - integrated_scraper_2 branch tracking PR #1559 (@jamescowens)
 - Upgrade depends  - OpenSSL to 1.1.1d #1581 (@jamescowens)
 - Ubuntu 19.10 fixes #1590 (@denravonska)
 - Force a re-parse of legacy claims in generated blocks #1592 (@cyrossignol)
 - Improve the "versionreport" RPC output #1595 (@cyrossignol)
 - Overhaul the core tally and accrual system #1583 (@cyrossignol)
 - Overhaul the superblock quorum system #1597 (@cyrossignol)
 - Add more data to the "superblocks" RPC output #1599 (@cyrossignol)
 - Update Windows Build doc #1606 (@barton2526)
 - Change the order of calls in gridcoinresearchd.cpp to optimize rpc shunt path #1610 (@jamescowens)
 - Change staking tooltip to display frequency #1611 (@jamescowens)
 - Enhancements to ETTS #1442 (@jamescowens)
 - Standardize money values as integers #1614 (@cyrossignol)
 - Clean up and optimize legacy coin age code #1616 (@cyrossignol)
 - Some scraper cleanups #1620 (@jamescowens)
 - Reorganize accrual code and fix 6-month cutoff #1630 (@cyrossignol)
 - Update Copyright years #1633 (@barton2526)
 - Change team whitelist delimiter to <> for CPID detection #1634 (@cyrossignol)
 - Change team whitelist separator to <> to accomodate more team names #1632 (@jamescowens)
 - Change Curl download speed type to support older environments #1640 (@cyrossignol)
 - Optimize logo SVGs used for tray icons #1638 (@cyrossignol)
 - Tweak consolidateunspent rpc function #1644 (@jamescowens)
 - ETTS and staking icon enhancements #1650 (@jamescowens)
 - Implement new transaction fees for block version 11 #1652 (@jamescowens)
 - Optimize in-memory storage of superblock data #1653 (@cyrossignol)
 - Miscellaneous superblock API improvements and housekeeping #1654 (@cyrossignol)
 - Update openssl to 1.1.1f compatibility #1660 (@jamescowens)
 - Optimize bdb to avoid synchronous flush of database #1659 (@jamescowens)
 - Add support for CPID input to "lifetime" RPC function #1668 (@cyrossignol)
 - Overhaul the contract handling system #1669 (@cyrossignol)
 - Make the autostart mainnet/testnet aware #1671 (@jamescowens)
 - Remove slashes from User Agent in peers tab #1674 (@div72)
 - Refactor contracts for polymorphic binary payloads #1676 (@cyrossignol)
 - Overhaul the beacon system #1678 (@cyrossignol)
 - Replace boost::optional<T&> with non-owning pointers #1680 (@cyrossignol)
 - Optimize proof-of-stake validation #1681 (@cyrossignol)
 - Updated Slack link #1683 (@NeuralMiner)
 - Update build-unix.md #1686 (@Quezacoatl1)
 - Replace deprecated QT methods #1693 (@Pythonix)
 - Made protocol.h more similar to bitcoin #1688 (@Pythonix)
 - Touch up some details for block version 11 #1697 (@cyrossignol)
 - More tweaks for block version 11 #1700 (@cyrossignol)
 - Finish the conversion to the BCLog class based logger #1699 (@jamescowens)
 - Move claim version transitional code in miner for proper signature #1712 (@cyrossignol)
 - doc: Update threads in coding.txt #1730 (@div72)
 - qt: Include QPainterPath in trafficgraphwidget.cpp #1733 (@div72)
 - doc: Update doc/build-unix.md #1731 (@div72)
 - gui: Show peers tab on connections icon click #1734 (@div72)
 - refactor: Change return type of IsMine to isminetype && move wallet files to wallet directory #1722 (@div72)
 - build: Updates boost to 1.73.0 for depends #1673 (@jamescowens)
 - doc: Update Unit Test Readme #1743 (@RoboticMind)
 - wallet: Change Assert To Error Message In kernel.cpp #1748 (@RoboticMind)
 - scraper: Shorten display representation of verification codes #1754 (@cyrossignol)
 - log: Change ".B." to Clear Message #1758 (@RoboticMind)
 - util: Fix braindamage in GetDefaultDataDir() #1737 (@jamescowens)
 - scraper: Improve scraper processing of beacon verifications #1760 (@jamescowens)
 - scraper: Add instrumentation to convergencereport #1763 (@jamescowens)
 - rpc: Improve rpc stress test script #1767 (@tunisiano187)
 - Generalize enum serialization #1770 (@cyrossignol)
 - scraper: Improve handling of ETags in http class and tweak verified beacon logic #1776 (@jamescowens)
 - scraper: Improve ProcessNetworkWideFromProjectStats and other tweaks #1778 (@jamescowens)
 - researcher: Automate beacon advertisement for renewals only #1781 (@cyrossignol)
 - gui: Tweak behavior of beacon page in researcher wizard #1784 (@cyrossignol)
 - Prepare for block version 11 hard-fork on testnet #1787 (@cyrossignol)
 - scraper: Modify UpdateVerifiedBeaconsFromConsensus #1791 (@jamescowens)
 - gui: Optimize OverviewPage::updateTransactions() #1794 (@jamescowens)
 - ci: Adopt ci changes from Bitcoin #1795 (@div72)
 - consensus: switch snapshot accrual calculation to integer arithmetic #1799 (@cyrossignol)
 - voting: Overhaul the voting system #1809 (@cyrossignol)
 - contract: Optimize contract replay after chain reorganization #1815 (@cyrossignol)
 - contract: Reimplement transaction messages as contracts #1816 (@cyrossignol)
 - staking: Sign claim contracts with coinstake transaction #1817 (@cyrossignol)
 - gui: Change research wizard text #1820 (@div72)
 - net: Update protocol version and clean up net messaging #1824 (@cyrossignol)
 - rpc, wallet: Corrections to GetAmounts #1825 (@jamescowens)
 - gui: Tweak some minor researcher wizard details #1830 (@cyrossignol)
 - gui: Change GetEstimatedStakingFrequency text #1836 (@jamescowens)
 - scraper: Scraper global statistics cache optimization #1837 (@jamescowens)
 - doc: Update Vulnerability Response Process #1843 (@RoboticMind)
 - scraper: Optimization of manifest and parts sharing between ConvergedScraperStatsCache, mapManifest, and mapParts #1851 (@jamescowens)
 - consensus: Update Checkpoints #1855 (@barton2526)
 - docs: Update docs to build off master #1856 (@barton2526)
 - gui: Fix and improve GUI combo box styles #1858 (@cyrossignol)
 - build: Tweak Gridcoin installer for Fern release #1863 (@jamescowens)

 Removed
 - Remove old research age checks (rebase #1365) #1572 (@cyrossignol)
 - Remove PrimaryCPID check from diagnostics dialog #1586 (@cyrossignol)
 - Remove missed label for PrimaryCPID from diagnostics #1588 (@cyrossignol)
 - Remove legacy quorum messaging system (@neural network) #1589 (@cyrossignol)
 - Remove old remnants of legacy smart contract experiments #1594 (@cyrossignol)
 - Remove block nonce for version 11 #1622 (@cyrossignol)
 - Delete obsolete contrib/Installer and Upgrader directories #1623 (@jamescowens)
 - Remove redundant LoadAdminMessages() calls #1625 (@cyrossignol)
 - Remove some legacy informational RPC commands #1658 (@cyrossignol)
 - Remove informational magnitude field from binary claims #1661 (@cyrossignol)
 - Remove fDebug3,4, and net and convert to BCLog::LogFlags #1663 (@jamescowens)
 - Remove qt5.7.1 depends support build System #1665 (@iFoggz)
 - Remove unused jQuery library #1679 (@cyrossignol)
 - Remove unused NetworkTimer() function and global state #1701 (@cyrossignol)
 - Refactor claim context objects into contracts #1704 (@cyrossignol)
 - Clean old assets up #1718 (@div72)
 - Remove legacy "rain" RPC (not by-project rain) #1742 (@cyrossignol)
 - Temporarily disable voting system on testnet #1769 (@cyrossignol)
 - gui: Remove legacy GUI transaction description for contracts #1772 (@cyrossignol)
 - gui: Remove transaction fee setting #1780 (@cyrossignol)
 - trivial: Cleanup unused legacy functions #1793 (@cyrossignol)
 - mining, rpc: Remove kernel-diff-best and kernel-diff-sum #1796 (@jamescowens)
 - refactor: Remove libs subdirectory #1802 (@div72)
 - scraper: cleanup unused/unnecessary functions #1803 (@jamescowens)
 - gui: Remove useless "Detach databases at shutdown" #1810 (@jamescowens)
 - test: Remove testnet condition for standard transactions #1814 (@cyrossignol)
 - consensus: Remove transitional testnet code #1854 (@cyrossignol)

 Fixed
 - Fix "Owed" amount in output of "magnitude" RPC method #1569 (@cyrossignol)
 - Add support for paths with special characters on Windows #1571 (@cyrossignol)
 - Fix lingering peers.dat temp files and clean up remaining paths #1582 (@cyrossignol)
 - Fix incorrect beacon length warning in GUI transaction list #1585 (@cyrossignol)
 - Fix default config file line endings on Windows #1587 (@cyrossignol)
 - Reenable Travis builds for MacOS #1591 (@jamescowens)
 - Correct peer detail info background color #1593 (@jamescowens)
 - Fix exception in debug3 mode #1598 (@cyrossignol)
 - Fix deadlock in "getmininginfo" RPC function #1596 (@cyrossignol)
 - Fix accuracy of statistics in "network" RPC output #1602 (@cyrossignol)
 - Fix heights for quorum vote weight calculations #1604 (@cyrossignol)
 - Fix deadlock in log archiver when rename fails #1607 (@cyrossignol)
 - Fix a spurious segmentation fault during client load on Windows with fast CPUs #1608 (@jamescowens)
 - Fix lock order debugging and potential deadlocks #1612 (@jamescowens)
 - Add dependencies #1613 (@Scalextrix)
 - Fix std namespace pollution #1617 (@denravonska)
 - Add missing condition for newbie accrual computer #1618 (@cyrossignol)
 - Track first reward blocks in research accounts #1619 (@cyrossignol)
 - Fix lingering beacon warning after advertisement #1627 (@cyrossignol)
 - Fix accrual calculation for new, zero-magnitude CPIDs #1636 (@cyrossignol)
 - Fix diagnostics, add ETTS test, fix tooltipcolor, add missing lock, and add email=investor check #1647 (@jamescowens)
 - Fix help message of two RPC methods #1656 (@div72)
 - Fix legacy accrual for newbie with non-zero past reward #1667 (@cyrossignol)
 - Fix GUI autostart on Windows for paths with wide characters #1670 (@cyrossignol)
 - Qualify boost bind placeholders with their full namespace #1672 (@Ponce)
 - Fix suffix when copying txids #1677 (@div72)
 - Unnecessary if-statement removed #1685 (@Pythonix)
 - Fix consolidatemsunspent Help Message #1687 (@Pythonix)
 - Fix gettransaction help message #1691 (@Pythonix)
 - Fix GetNewMint To Look for Stakes #1692 (@RoboticMind)
 - Suppress deprecated copy warnings for Qt with GCC 9+ #1702 (@cyrossignol)
 - Fix exclusion error on stats processing and misplaced ENDLOCK logging entry #1710 (@jamescowens)
 - Removed unnecessary comparison #1708 (@Pythonix)
 - Fixed typo #1707 (@Pythonix)
 - Fix out-of-bounds exception for peers tab version slashes #1713 (@cyrossignol)
 - Fix transition for v1 superblocks when reorganizing #1714 (@cyrossignol)
 - Touch up transition to version 2 transactions #1715 (@cyrossignol)
 - Avoid mutating transactions in ConnectBlock() #1716 (@cyrossignol)
 - Skip beacon advertisement when already pending #1726 (@cyrossignol)
 - Fix Windows cross-compilation in newer environments #1728 (@cyrossignol)
 - Fix out-of-bounds access in IsMineInner() #1736 (@cyrossignol)
 - Fix a couple of block version 11 issues #1738 (@cyrossignol)
 - Fix null pointer dereference in GUI researcher model #1741 (@cyrossignol)
 - accrual: Reset research accounts when rebuilding accrual snapshots #1745 (@cyrossignol)
 - scraper: Correct update for verified beacons #1747 (@jamescowens)
 - accrual: Refactor tally initialization for snapshot rebuild #1749 (@cyrossignol)
 - rpc: Fix "cpid" field in "beaconconvergence" RPC output #1750 (@cyrossignol)
 - accrual: Fix snapshot accrual superblock state transitions #1752 (@cyrossignol)
 - scraper: Correct stale verified beacon logic #1753 (@jamescowens)
 - rpc: Correct possible divide by zero in getblockstats #1755 (@jamescowens)
 - gui: Fix issues with researcher wizard flow #1756 (@cyrossignol)
 - wallet: Stop Error When Starting From Zero #1759 (@RoboticMind)
 - Don't count empty email as explicit investor #1761 (@cyrossignol)
 - accrual: Fix snapshot accrual superblock state transitions #1764 (@cyrossignol)
 - rpc: Cleanup Help Message and Fix Typo #1771 (@RoboticMind)
 - scraper: Fix scraper etag header case sensitivity #1773 (@cyrossignol)
 - consensus: Use explicit time to check if superblock needed #1774 (@cyrossignol)
 - gui: Fix scroll area dark theme styles #1785 (@cyrossignol)
 - rpc, gui: Fix three divide by zero possibilities #1789 (@jamescowens)
 - rpc: Fix balance pre-check in "rainbymagnitude" RPC #1792 (@cyrossignol)
 - accrual: Fix outdated comment and correct grammar #1800 (@RoboticMind)
 - gui: Fix stuck cursor on labels #1801 (@div72)
 - beacon: Fix research wizard beacon renewal status #1805 (@cyrossignol)
 - gui: Fix translations for port numbers #1818 (@cyrossignol)
 - util: Create parent directory #1821 (@div72)
 - mining: Fix coinstake/claim signature order #1828 (@cyrossignol)
 - voting: Remove double increment in loop #1831 (@cyrossignol)
 - neuralnet, scraper: Fix compilation with gcc5 and older libcurl #1832 (@a123b)
 - wallet: Fix smallest coin selection for contracts #1841 (@cyrossignol)
 - gui: Fix display of polls with no votes yet #1844 (@cyrossignol)
 - gui: add indentation to diagnostic status bar labels #1849 (@jamescowens)
 - voting, gui: Fix formatting and alignment of vote shares and percent #1850 (@jamescowens)
 - wallet, rpc: Fix for self-transactions in listtransactions #1852 (@jamescowens)
 - accrual: Clear any accrual snapshots when syncing from pre-v11 #1853 (@cyrossignol)
 - accrual: Fix reset of accrual directory if starting sync below research age height #1857 (@jamescowens)
 - gui: Fix researcher wizard layout on macOS with native theme #1860 (@cyrossignol)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants