Skip to content

Commit

Permalink
Merge pull request #3471 from TrueBlocks/omahs-patch-4
Browse files Browse the repository at this point in the history
Omahs patch 4
  • Loading branch information
tjayrush authored Jan 7, 2024
2 parents d0689b2 + 4aed7a8 commit 0b6efd3
Show file tree
Hide file tree
Showing 64 changed files with 99 additions and 95 deletions.
2 changes: 1 addition & 1 deletion docs/content/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2100,7 +2100,7 @@ paths:
type: boolean
- name: remote
description: >
prior to processing, retreive the manifest from the Unchained Index smart contract
prior to processing, retrieve the manifest from the Unchained Index smart contract
required: false
style: form
in: query
Expand Down
20 changes: 11 additions & 9 deletions docs/content/chifra/admin.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ weight: 1600
toc: true
---
<!-- markdownlint-disable MD033 MD036 MD041 -->
The Admin gourp of commands allows you to query the status of the TrueBlocks system and
manage various apects including the Unchained Index. You may query the status; query for information
The Admin group of commands allows you to query the status of the TrueBlocks system and
manage various aspects including the Unchained Index. You may query the status; query for information
about TrueBlocks caches; control the creation, sharing, and pinning of the Unchained Index; and even
serve the data through an API.

Expand Down Expand Up @@ -165,7 +165,7 @@ To convert the options for a command line tool to an API call, do the following:

1. Any `--snake_case` argument to the command line should be converted to `camelCase`. For example, `--no_header` on the command line should be sent as `&noHeader` to the API server.
2. Any `switch` on the command line, (i.e., options whose presence indicates `true` and whose absence indicates `false`) should be sent as a `boolean` to the API server. For example, `--no_header` on the command line should be sent as `&noHeader=true` to the API server. If the option is `fales`, you do not need to send it to the API server.
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` seperating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` separating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).


## chifra scrape
Expand All @@ -179,7 +179,7 @@ multiplexer such as `tmux`. You may start and stop `chifra scrape` as needed, bu
scraper will not be keeping up with the front of the blockchain. The next time it starts, it will
have to catch up to the chain, a process that may take several hours depending on how long ago it
was last run. See the section below and the "Papers" section of our website for more information
on how the scraping process works and prerequisites for it proper operation.
on how the scraping process works and prerequisites for its proper operation.

You may adjust the speed of the index creation with the `--sleep` and `--block_cnt` options. On
some machines, or when running against some EVM node software, the scraper may overburden the
Expand Down Expand Up @@ -234,7 +234,7 @@ Each of the following additional configurable command line options are available

Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 respectively. See the specification for a justification of these values.

These items may be set in three ways, each overridding the preceeding method:
These items may be set in three ways, each overriding the preceding method:

-- in the above configuration file under the `[scrape.<chain>]` group,
-- in the environment by exporting the configuration item as UPPER&lowbar;CASE (with underbars removed) and prepended with TB_SCRAPE&lowbar;CHAIN&lowbar;, or
Expand All @@ -245,7 +245,7 @@ These items may be set in three ways, each overridding the preceeding method:
Each time `chifra scrape` runs, it begins at the last block it completed processing (plus one). With
each pass, the scraper descends as deeply as is possible into each block's data. (This is why
TrueBlocks requires a `--tracing` node.) As the scraper encounters appearances of address in the
block's data, it adds those appearance to a growing index. Periodically (after processing the the
block's data, it adds those appearances to a growing index. Periodically (after processing the
block that contains the 2,000,000th appearance), the system consolidates an **index chunk**.

An **index chunk** is a portion of the index containing approximately 2,000,000 records (although,
Expand Down Expand Up @@ -279,7 +279,7 @@ more information about running the scraper and building and sharing the index of
## chifra chunks

The chifra chunks routine provides tools for interacting with, checking the validity of, cleaning up,
and analyizing the Unchained Index. It provides options to list pins, the Manifest, summary data
and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data
on the index, Bloom filters, addresses, and appearances. While still in its early stages, this
tool will eventually allow users to clean their local index, clean their remote index, study
the indexes, etc. Stay tuned.
Expand All @@ -300,7 +300,7 @@ Flags:
-c, --check check the manifest, index, or blooms for internal consistency
-i, --pin pin the manifest or each index chunk and bloom
-p, --publish publish the manifest to the Unchained Index smart contract
-r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract
-r, --remote prior to processing, retrieve the manifest from the Unchained Index smart contract
-b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk
-F, --first_block uint first block to process (inclusive)
-L, --last_block uint last block to process (inclusive)
Expand Down Expand Up @@ -344,6 +344,7 @@ Links:
- [api docs](/api/#operation/admin-chunks)
- [source code](https://github.com/TrueBlocks/trueblocks-core/tree/master/src/apps/chifra/internal/chunks)


## chifra init

When invoked, `chifra init` reads a value from a smart contract called **The Unchained Index**
Expand All @@ -360,7 +361,7 @@ time you run it.

Certain parts of the system (`chifra list` and `chifra export` for example) if you have not
previously run `chifra init` or `chifra scrape`. You will be warned by the system until it's
satified.
satisfied.

If you run `chifra init` and allow it to complete, the next time you run `chifra scrape`, it will
start where `init` finished. This means that only the blooms will be stored on your hard drive.
Expand Down Expand Up @@ -397,3 +398,4 @@ Links:

- [api docs](/api/#operation/admin-init)
- [source code](https://github.com/TrueBlocks/trueblocks-core/tree/master/src/apps/chifra/internal/init)

5 changes: 3 additions & 2 deletions docs/readmes/admin-chunks.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
## chifra chunks

The chifra chunks routine provides tools for interacting with, checking the validity of, cleaning up,
and analyizing the Unchained Index. It provides options to list pins, the Manifest, summary data
and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data
on the index, Bloom filters, addresses, and appearances. While still in its early stages, this
tool will eventually allow users to clean their local index, clean their remote index, study
the indexes, etc. Stay tuned.
Expand All @@ -23,7 +23,7 @@ Flags:
-c, --check check the manifest, index, or blooms for internal consistency
-i, --pin pin the manifest or each index chunk and bloom
-p, --publish publish the manifest to the Unchained Index smart contract
-r, --remote prior to processing, retreive the manifest from the Unchained Index smart contract
-r, --remote prior to processing, retrieve the manifest from the Unchained Index smart contract
-b, --belongs strings in index mode only, checks the address(es) for inclusion in the given index chunk
-F, --first_block uint first block to process (inclusive)
-L, --last_block uint last block to process (inclusive)
Expand Down Expand Up @@ -66,3 +66,4 @@ Links:

- [api docs](/api/#operation/admin-chunks)
- [source code](https://github.com/TrueBlocks/trueblocks-core/tree/master/src/apps/chifra/internal/chunks)

2 changes: 1 addition & 1 deletion docs/readmes/admin-daemon.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,5 @@ To convert the options for a command line tool to an API call, do the following:

1. Any `--snake_case` argument to the command line should be converted to `camelCase`. For example, `--no_header` on the command line should be sent as `&noHeader` to the API server.
2. Any `switch` on the command line, (i.e., options whose presence indicates `true` and whose absence indicates `false`) should be sent as a `boolean` to the API server. For example, `--no_header` on the command line should be sent as `&noHeader=true` to the API server. If the option is `fales`, you do not need to send it to the API server.
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` seperating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` separating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).

3 changes: 2 additions & 1 deletion docs/readmes/admin-init.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ time you run it.

Certain parts of the system (`chifra list` and `chifra export` for example) if you have not
previously run `chifra init` or `chifra scrape`. You will be warned by the system until it's
satified.
satisfied.

If you run `chifra init` and allow it to complete, the next time you run `chifra scrape`, it will
start where `init` finished. This means that only the blooms will be stored on your hard drive.
Expand Down Expand Up @@ -52,3 +52,4 @@ Links:

- [api docs](/api/#operation/admin-init)
- [source code](https://github.com/TrueBlocks/trueblocks-core/tree/master/src/apps/chifra/internal/init)

6 changes: 3 additions & 3 deletions docs/readmes/admin-scrape.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ multiplexer such as `tmux`. You may start and stop `chifra scrape` as needed, bu
scraper will not be keeping up with the front of the blockchain. The next time it starts, it will
have to catch up to the chain, a process that may take several hours depending on how long ago it
was last run. See the section below and the "Papers" section of our website for more information
on how the scraping process works and prerequisites for it proper operation.
on how the scraping process works and prerequisites for its proper operation.

You may adjust the speed of the index creation with the `--sleep` and `--block_cnt` options. On
some machines, or when running against some EVM node software, the scraper may overburden the
Expand Down Expand Up @@ -65,7 +65,7 @@ Each of the following additional configurable command line options are available

Note that for Ethereum mainnet, the default values for appsPerChunk and firstSnap are 2,000,000 and 2,300,000 respectively. See the specification for a justification of these values.

These items may be set in three ways, each overridding the preceeding method:
These items may be set in three ways, each overriding the preceding method:

-- in the above configuration file under the `[scrape.<chain>]` group,
-- in the environment by exporting the configuration item as UPPER&lowbar;CASE (with underbars removed) and prepended with TB_SCRAPE&lowbar;CHAIN&lowbar;, or
Expand All @@ -76,7 +76,7 @@ These items may be set in three ways, each overridding the preceeding method:
Each time `chifra scrape` runs, it begins at the last block it completed processing (plus one). With
each pass, the scraper descends as deeply as is possible into each block's data. (This is why
TrueBlocks requires a `--tracing` node.) As the scraper encounters appearances of address in the
block's data, it adds those appearance to a growing index. Periodically (after processing the the
block's data, it adds those appearances to a growing index. Periodically (after processing the
block that contains the 2,000,000th appearance), the system consolidates an **index chunk**.

An **index chunk** is a portion of the index containing approximately 2,000,000 records (although,
Expand Down
4 changes: 2 additions & 2 deletions docs/templates/readme-groups/admin.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<!-- markdownlint-disable MD033 MD036 MD041 -->
The Admin gourp of commands allows you to query the status of the TrueBlocks system and
manage various apects including the Unchained Index. You may query the status; query for information
The Admin group of commands allows you to query the status of the TrueBlocks system and
manage various aspects including the Unchained Index. You may query the status; query for information
about TrueBlocks caches; control the creation, sharing, and pinning of the Unchained Index; and even
serve the data through an API.

Expand Down
2 changes: 1 addition & 1 deletion docs/templates/readme-intros/admin-chunks.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<!-- markdownlint-disable MD041 -->
The [{NAME}] routine provides tools for interacting with, checking the validity of, cleaning up,
and analyizing the Unchained Index. It provides options to list pins, the Manifest, summary data
and analyzing the Unchained Index. It provides options to list pins, the Manifest, summary data
on the index, Bloom filters, addresses, and appearances. While still in its early stages, this
tool will eventually allow users to clean their local index, clean their remote index, study
the indexes, etc. Stay tuned.
2 changes: 1 addition & 1 deletion docs/templates/readme-intros/admin-daemon.notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ To convert the options for a command line tool to an API call, do the following:

1. Any `--snake_case` argument to the command line should be converted to `camelCase`. For example, `--no_header` on the command line should be sent as `&noHeader` to the API server.
2. Any `switch` on the command line, (i.e., options whose presence indicates `true` and whose absence indicates `false`) should be sent as a `boolean` to the API server. For example, `--no_header` on the command line should be sent as `&noHeader=true` to the API server. If the option is `fales`, you do not need to send it to the API server.
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` seperating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).
3. Positionals such as the addresses, topics, and four-bytes for `chifra export`, must be prepended with their positional name. For example, `chifra export <address> <topic>` should be sent as `&addrs=<address>&topics=<topic>` to the API server. For some commands (experiment) you may send more than one value for a positional with `%20` separating the entries or by sending multiple positionals (i.e., `&addrs=<address1>&addrs=<address2>`).
2 changes: 1 addition & 1 deletion docs/templates/readme-intros/admin-init.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ time you run it.

Certain parts of the system (`chifra list` and `chifra export` for example) if you have not
previously run `chifra init` or `chifra scrape`. You will be warned by the system until it's
satified.
satisfied.

If you run `chifra init` and allow it to complete, the next time you run `chifra scrape`, it will
start where `init` finished. This means that only the blooms will be stored on your hard drive.
Expand Down
2 changes: 1 addition & 1 deletion docs/templates/readme-intros/admin-scrape.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ multiplexer such as `tmux`. You may start and stop `[{NAME}]` as needed, but doi
scraper will not be keeping up with the front of the blockchain. The next time it starts, it will
have to catch up to the chain, a process that may take several hours depending on how long ago it
was last run. See the section below and the "Papers" section of our website for more information
on how the scraping process works and prerequisites for it proper operation.
on how the scraping process works and prerequisites for its proper operation.

You may adjust the speed of the index creation with the `--sleep` and `--block_cnt` options. On
some machines, or when running against some EVM node software, the scraper may overburden the
Expand Down
2 changes: 1 addition & 1 deletion docs/templates/readme-intros/admin-scrape.notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Each time `[{NAME}]` runs, it begins at the last block it completed processing (plus one). With
each pass, the scraper descends as deeply as is possible into each block's data. (This is why
TrueBlocks requires a `--tracing` node.) As the scraper encounters appearances of address in the
block's data, it adds those appearance to a growing index. Periodically (after processing the the
block's data, it adds those appearances to a growing index. Periodically (after processing the
block that contains the 2,000,000th appearance), the system consolidates an **index chunk**.

An **index chunk** is a portion of the index containing approximately 2,000,000 records (although,
Expand Down
6 changes: 3 additions & 3 deletions sdk/typescript/yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -2486,9 +2486,9 @@ [email protected]:
vite "^3.0.0 || ^4.0.0"

"vite@^3.0.0 || ^4.0.0":
version "4.4.9"
resolved "https://registry.yarnpkg.com/vite/-/vite-4.4.9.tgz#1402423f1a2f8d66fd8d15e351127c7236d29d3d"
integrity sha512-2mbUn2LlUmNASWwSCNSJ/EG2HuSRTnVNaydp6vMCm5VIqJsjMfbIWtbH2kDuwUVW5mMUKKZvGPX/rqeqVvv1XA==
version "4.5.1"
resolved "https://registry.yarnpkg.com/vite/-/vite-4.5.1.tgz#3370986e1ed5dbabbf35a6c2e1fb1e18555b968a"
integrity sha512-AXXFaAJ8yebyqzoNB9fu2pHoo/nWX+xZlaRwoeYUxEqBO+Zj4msE5G+BhGBll9lYEKv9Hfks52PAF2X7qDYXQA==
dependencies:
esbuild "^0.18.10"
postcss "^8.4.27"
Expand Down
2 changes: 1 addition & 1 deletion src/apps/chifra/cmd/chunks.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ func init() {
chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Publish, "publish", "p", false, "publish the manifest to the Unchained Index smart contract")
chunksCmd.Flags().StringVarP(&chunksPkg.GetOptions().Publisher, "publisher", "P", "", "for some query options, the publisher of the index (hidden)")
chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().Truncate, "truncate", "n", 0, "truncate the entire index at this block (requires a block identifier) (hidden)")
chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Remote, "remote", "r", false, "prior to processing, retreive the manifest from the Unchained Index smart contract")
chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Remote, "remote", "r", false, "prior to processing, retrieve the manifest from the Unchained Index smart contract")
chunksCmd.Flags().StringSliceVarP(&chunksPkg.GetOptions().Belongs, "belongs", "b", nil, "in index mode only, checks the address(es) for inclusion in the given index chunk")
chunksCmd.Flags().BoolVarP(&chunksPkg.GetOptions().Diff, "diff", "f", false, "compare two index portions (see notes) (hidden)")
chunksCmd.Flags().Uint64VarP(&chunksPkg.GetOptions().FirstBlock, "first_block", "F", 0, "first block to process (inclusive)")
Expand Down
10 changes: 5 additions & 5 deletions src/apps/chifra/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ require (
github.com/spf13/cobra v1.7.0
github.com/spf13/viper v1.9.0
github.com/wealdtech/go-ens/v3 v3.5.2
golang.org/x/term v0.7.0
golang.org/x/term v0.15.0
golang.org/x/time v0.0.0-20220922220347-f3bd1da661af
google.golang.org/grpc v1.56.3
google.golang.org/protobuf v1.30.0
Expand Down Expand Up @@ -71,11 +71,11 @@ require (
github.com/tklauser/numcpus v0.4.0 // indirect
github.com/wealdtech/go-multicodec v1.4.0 // indirect
github.com/yusufpapurcu/wmi v1.2.2 // indirect
golang.org/x/crypto v0.6.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
golang.org/x/exp v0.0.0-20230811145659-89c5cff77bcb // indirect
golang.org/x/net v0.9.0 // indirect
golang.org/x/sys v0.11.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
gopkg.in/ini.v1 v1.63.2 // indirect
gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce // indirect
Expand Down
Loading

0 comments on commit 0b6efd3

Please sign in to comment.