Releases: hiero-ledger/hiero-mirror-node
v0.20.0
This is a big release that contains support for a new HAPI service and whole new runtime component to dramatically improve performance. Due to the magnitude of the changes, it did take us a little longer to mark it as generally available as we wanted to ensure it was tested as much as possible beforehand.
First up is support for the Hedera Token Service (HTS) that was recently announced and rolled out to previewnet. A lot of work was put into supporting the new transaction types on the parser side including enhancing the schema with new tables and ingesting them via the record stream. HTS also required a new balance file version that adds token information to the CSV. Token information is now returned for our existing REST APIs while the next release will contain token specific REST APIs for further granularity. Check it out in previewnet and let us know if you have any feedback!
We made a lot of strides in improving the ingestion performance in previous releases, but since we also wanted to ensure low end to end HCS latency via our gRPC API we had to sacrifice some of that speed. As a result, we could only ingest at about 3,000 transactions per second (TPS) before latency spiked above 10 seconds. This was entirely due to our use of PostgreSQL notify/listen to notify the gRPC API of new data.
In this release, we add a new notification mechanism without sacrificing speed or latency with our support for Redis pub/sub. With Redis, the mirror node can now ingest at least 10,000 TPS while still remaining under 10 seconds from submitting the topic message and receiving it back via the mirror node's streaming API. Redis is enabled by default, but it can be turned off if HCS latency is not a concern and you want to avoid another runtime dependency.
We also added support for the HAPI protobuf changes that are coming in v0.9.0. The protobuf is removing two deprecated fields while adding a new signedTransactionBytes field. Since the mirror node still needs to support historical transactions we retain support for parsing transactions that contain the old payload format.
Enhancements
- Bump versions for 0.20.0 #1200
- Bump version to rc1 from alpha1 w HTS Acceptance Tests #1137
- Bump versions for v0.20.0-alpha1 #1109
- Enable code scanning for vulnerabilities #1092
- Add Redis support to Helm #1091
- HTS: Support new balance file version #1090
- HTS: Add support for token transactions to Importer #1089
- HTS: Add token balances to accounts REST API #1088
- HTS: Add new fields to accounts, balances and transactions REST API #1087
- HTS: Minor updates to design and schema #1086
- Add topic messages and stateproof to monitor #1076
- Add a Redis HCS topic listener #1075
- stateproof: download record file from stored node #1074
- HTS: DB schema changes #1073
- Replace account_realm_num and account_num with account_id in account_balance table #1071
- Migrate account_balance table to use entity_id for account id #1070
- State proof alpha: download record file from the node used to verify and download the file in importer #1069
- HTS: Design document #1061
- allow individual maxLimit config in rest service monitor #1060
- Bump hedera-protobuf to 0.9.0-alpha2 and support new fields #1056
- HTS: Import token transactions #1051
- HTS: Support new balance file version #1049
- HTS: Database Schema #1048
- HTS: Add new fields to balances and transactions REST API #1047
- HTS: Design #1043
- Allow for configuration of specific checks on the monitor #1033
- Protobuf changes for 0.9.0 #1027
- Pre-OA Mirror Node #913
- Reduce subscribe topic latency #893
- Monitoring API lacks topics and stateproof endpoint coverage #667
- Aggregate monitor API should return 4xx if individual server fails #462
Bug Fixes
- Fix NPE parsing token transactions #1157
- NullPointerException parsing token dissociate #1156
- Fix memory leak in performance tests #1072
- Re add deleted metrics #1058
- gRPC notify thread blocked improvement #1057
- Fix polling cloud storage more often than the configured poll period #1054
- Fix race condition in check-state-proof #1053
- Check-state-proof error #1052
Documentation
- Fix wrong command in check-state-proof readme #1055
Dependency Upgrades
- Bump docker-maven-plugin from 0.33.0 to 0.34.1 #1100
- Bump jacoco-maven-plugin from 0.8.5 to 0.8.6 #1099
- Bump testcontainers from 4.0.9 to 4.3.0 in /hedera-mirror-rest #1097
- Bump eslint-plugin-import from 2.22.0 to 2.22.1 in /hedera-mirror-rest #1096
- Bump sinon from 9.0.3 to 9.1.0 in /hedera-mirror-rest #1093
- Bump spring-cloud-dependencies from Hoxton.SR7 to Hoxton.SR8 #1084
- Bump eslint-config-prettier from 6.11.0 to 6.12.0 in /hedera-mirror-rest #1083
- Bump aws-sdk from 2.756.0 to 2.761.0 in /hedera-mirror-rest #1082
- Bump supertest from 4.0.2 to 5.0.0 in /hedera-mirror-rest #1081
- Bump mathjs from 7.2.0 to 7.3.0 in /hedera-mirror-rest #1080
- Bump prettier from 2.1.1 to 2.1.2 in /hedera-mirror-rest #1079
- Bump mathjs from 7.2.0 to 7.3.0 in /hedera-mirror-rest/monitoring/monitor_apis #1078
- Bump supertest from 4.0.2 to 5.0.0 in /hedera-mirror-rest/monitoring/monitor_apis #1077
- Bump cucumber.version from 6.6.0 to 6.7.0 #1066
- Bump sdk from 1.2.0 to 1.2.2 #1065
- Bump aws-sdk from 2.747.0 to 2.756.0 in /hedera-mirror-rest #1064
- Bump testcontainers from 3.4.4 to 4.0.9 in /hedera-mirror-rest #1063
- Bump eslint-plugin-jest from 24.0.0 to 24.0.2 in /hedera-mirror-rest #1062
- Bump embedded.testcontainers.version from 1.81 to 1.82 #1041
- Bump testcontainers from 3.4.3 to 3.4.4 in /hedera-mirror-rest #1039
- Bump pretty-quick from 3.0.0 to 3.0.2 in /hedera-mirror-rest #1036
- Bump husky from 4.2.5 to 4.3.0 in /hedera-mirror-rest #1035
- Bump protobuf-java from 3.11.1 to 3.13.0 #1024
- Bump reporting-plugin from 4.0.77 to 4.0.84 #1023
Contrib...
v0.21.0-rc2
Bump v0.21.0-rc2 with missing columns to token info REST API (#1173) * Bump v0.21.0-rc2 with missing columns to token info REST API - Add modified_timestamp and name to token info response - Update spec and its with missing columns - Bump from rc1 to rc2 Signed-off-by: Nana-EC <[email protected]>
v0.21.0-rc1
Bump version for v0.21.0-rc1 (#1161) Signed-off-by: Nana-EC <[email protected]>
v0.20.0-rc2
Fix NPE parsing token transactions (#1157) - Fix NPE parsing token transactions - Bump versions for 0.20.0-rc2 Signed-off-by: Steven Sheehy <[email protected]>
v0.20.0-rc1
Bump version to rc1 from alpha1 w HTS Acceptance Tests (#1137) * Bump version to rc1 from alpha1 w HTS Acceptance Tests - Add Basic HTS coverage for Acceptance tests - Bump version from alpha1 to rc1 Signed-off-by: Nana-EC <[email protected]>
v0.20.0-alpha1
Bump versions for v0.20.0-alpha1 (#1109) Signed-off-by: Steven Sheehy <[email protected]>
v0.19.0
This release finishes the State Proof alpha REST API and makes it generally available. As part of this, we made a lot of improvements to the check-state-proof command line tool that queries the API and validates the files locally. We also now store the node account used to verify record file, ensuring greater accuracy as to the provenance of the state proof.
There's been some changes to the public Hedera environments lately and we've updated the mirror node to reflect that. We added support for the new previewnet environment and we updated the configuration to point to the new testnet bucket after its recent reset. Please ensure your mirror node has all of the data in the previous bucket before updating to this release, assuming you're not specifying the bucket name manually.
We added proper liveness and readiness probe endpoints for all our components. If you're not familiar with the concept of liveness and readiness probes, check out the Kubernetes documentation on the subject. Our new liveness endpoint now does not fail if external dependencies are down like the database, ensuring the application doesn't restart unnecessarily. Even if you're not using Kubernetes it would be worthwhile to look into to ensure your mirror node is using the appropriate endpoint for health checks, based upon your needs.
Enhancements
- Bump versions for 0.19.0 #1068
- Bump versions for v0.19.0-rc1 #1046
- Return only the current address book from state proof alpha REST API #1018
- Improved check-state-proof CLI support and documentation #1016
- Update testnet bucketname and add previewnet #1015
- Splitting health check into liveness and readiness checks #1014
- Update testnet bucket name and add previewnet #1010
- Store verification info in stream file tables #1009
- Composite entity listener #995
- Make HCS acceptance tests less dependent on maven downloads #967
- Fix acceptance tests so that the jar is not always built (#967) #994
- Store which node was used to verify a stream file #919
Bug Fixes
- Re-add metrics accidentally deleted in #1009 #1059
- Clear topic notifications on error #1031
- Memory leak in NotifyingEntityListener #1028
- Ignore NodeAddress fields not set in protobuf #1013
- Address book defaults node_id and port to 0 #968
Dependency Upgrades
- Bump node-fetch from 2.6.0 to 2.6.1 in /hedera-mirror-rest/check-state-proof #1030
- Bump software.amazon.awssdk:bom from 2.13.11 to 2.14.12 #1025
- Bump eslint-plugin-jest from 23.20.0 to 24.0.0 in /hedera-mirror-rest #1022
- Bump aws-sdk from 2.742.0 to 2.747.0 in /hedera-mirror-rest #1021
- Bump node-fetch from 2.6.0 to 2.6.1 in /hedera-mirror-rest #1020
- Bump testcontainers from 3.4.0 to 3.4.3 in /hedera-mirror-rest #1019
- Bump bl from 4.0.2 to 4.0.3 in /hedera-mirror-rest #1011
- Bump cucumber.version from 6.5.0 to 6.6.0 #1007
- Bump reactor-grpc-stub from 1.0.0 to 1.0.1 #1004
- Bump spring-cloud-gcp-dependencies from 1.2.4.RELEASE to 1.2.5.RELEASE #1003
- Bump mathjs from 7.1.0 to 7.2.0 in /hedera-mirror-rest #1002
- Bump pg from 8.3.2 to 8.3.3 in /hedera-mirror-rest #1001
- Bump aws-sdk from 2.738.0 to 2.742.0 in /hedera-mirror-rest #1000
- Bump prettier from 2.0.5 to 2.1.1 in /hedera-mirror-rest #999
- Bump mathjs from 7.1.0 to 7.2.0 in /hedera-mirror-rest/monitoring/monitor_apis #998
- Bump software.amazon.awssdk:bom from 2.13.11 to 2.14.3 #983
Contributors
We'd like to thank all the contributors who worked on this release!
v0.18.2
v0.18.1
Contains a small change to the State Proof Alpha REST API to only return the current address book for now.
Enhancements
- Bump versions for v0.18.1 #1032
- Return only the current address book from state proof alpha REST API #1029
Contributors
We'd like to thank all the contributors who worked on this release!
v0.18.0
Building upon the availability of the State Proof Alpha REST API in the last release, we've added sample code in JavaScript to retrieve the state proof from a mirror node and locally verify it. This allows users to obtain cryptographic proof that a particular transaction took place on Hedera. The validity of the proof can be checked independently to ensure that the supermajority of Hedera mainnet stake had reached consensus on that transaction. Similar to the promise of the state proofs on the main nodes, the user can trust this state proof alpha served by the mirror nodes, even when the user does not trust them.
Importer can now be configured to connect to Amazon S3 using temporary security credentials via AssumeRole. With this, a user that does not have permission to access an AWS resource can request a temporary role that will grant them that permission. See the configuration documentation for more information.
Importer also added two new properties to control the subset of data it should download and validate. The hedera.mirror.importer.startDate property can be used to exclude data from before this date and "fast-forward" to a point in time of interest. By default, the startDate will be set to the current time so mirror node operators can get up and running quicker with the latest data and reduce cloud storage retrieval costs. Note that this property only applies on the importer's first startup and can't be changed after that. The hedera.mirror.importer.endDate property can be used to exclude data after this date and halt the importer. By default it is set to a date far in the future so it will effectively never stop.
Breaking Changes
The aforementioned startDate property does change how the mirror node operators on initial start from previous releases. By defaulting to now, users standing up a new mirror node will no longer retrieve all historical data and will instead only retrieve the latest data. Current users upgrading to this release will not be affected even if their data ingest is not fully caught up since this property only applies if the database is empty like it is on first start. To revert to the previous behavior, a date in the past can be specified like the Unix epoch 1970-01-01T00:00:00Z.
Enhancements
- Bump versions for v0.18.0 #1012
- Bump versions for 0.18.0-rc1 #991
- Adding AWS AssumeRole support to Importer #971
- Performance test HighTPS HCS publish improvements #970
- State proof js demo #969
- Import latest property #966
- Moved RecordFileParser file system logic to RecordFilePoller #936
- AWS AssumeRole support #900
- JavaScript state proof sample code #866
- Import latest property #801
Bug Fixes
- Fix entity cache #990
- Fix TopicMessageServiceTest.incomingMessagesWithEndTimeBefore #986
- incomingMessagesWithEndTimeBefore fails #985
- Add a drop.sql script for OPS usage when resetting the mirror node to… #973
- Fix the issue gRPC notify thread blocked #965
- gRPC API notify thread blocks and cancels subscribers #945
- Duplicate key violation error after importer restart #899
Dependency Upgrades
- Bump sdk from 1.1.5 to 1.2.0 #982
- Bump reporting-plugin from 4.0.70 to 4.0.77 #981
- Bump embedded.testcontainers.version from 1.62 to 1.81 #980
- Bump axios from 0.19.2 to 0.20.0 in /hedera-mirror-rest #979
- Bump pretty-quick from 2.0.1 to 3.0.0 in /hedera-mirror-rest #978
- Bump pg from 8.3.0 to 8.3.2 in /hedera-mirror-rest #977
- Bump aws-sdk from 2.734.0 to 2.738.0 in /hedera-mirror-rest #976
- Bump jest from 26.4.0 to 26.4.2 in /hedera-mirror-rest/monitoring/monitor_apis #975
- Bump jest from 26.4.0 to 26.4.2 in /hedera-mirror-rest #974
- Bump jib-maven-plugin from 2.5.0 to 2.5.2 #964
- Bump grpc-netty-shaded from 1.27.2 to 1.30.2 #963
- Bump grpc.version from 1.31.0 to 1.31.1 #962
- Bump spring-boot-starter-parent from 2.2.7.RELEASE to 2.3.3.RELEASE #961
- Bump reporting-plugin from 4.0.66 to 4.0.70 #960
- Bump aws-sdk from 2.725.0 to 2.734.0 in /hedera-mirror-rest #959
- Bump jest from 26.2.2 to 26.4.0 in /hedera-mirror-rest #958
- Bump testcontainers from 3.2.0 to 3.4.0 in /hedera-mirror-rest #957
- Bump lodash from 4.17.19 to 4.17.20 in /hedera-mirror-rest #956
- Bump sinon from 9.0.2 to 9.0.3 in /hedera-mirror-rest #955
- Bump jest from 26.2.2 to 26.4.0 in /hedera-mirror-rest/monitoring/monitor_apis #954
Contributors
We'd like to thank all the contributors who worked on this release!