adjust scripts for API v5's BNS import

[rafaelcr]

We're in the process of releasing Hiro API v5 which fixes a bunch of bugs related to BNS data.

One of those changes is that now we import BNS data (if available) before event replay, because the API needs to know which namespaces exist so it can calculate names' expiration block heights if it sees a name registered, updated or transferred in a tx during replay. Launching the API no longer imports BNS data as it did before.

This import is handled automatically i.e. there's no need for users to run a command other than import-events, however, this repo might need some adjusting (maybe just on docs) so it can consider the availability of BNS data before performing a replay.

[wileyj]

if i understand this correctly - the API env var BNS_IMPORT_DIR has no impact for v5 when launching the API?
and BNS_IMPORT_DIR has an effect when importing events?

maybe i'm a bit confused since your comment says we import BNS data (if available) before event replay, but then Launching the API no longer imports BNS data as it did before

are there docs you can share on how this is changing in the API rep?

[rafaelcr]

if i understand this correctly - the API env var BNS_IMPORT_DIR has no impact for v5 when launching the API?
and BNS_IMPORT_DIR has an effect when importing events?

That is correct. The event replay process will now be the one to look at BNS_IMPORT_DIR and do the import.

are there docs you can share on how this is changing in the API rep?

I modified a part of the README here: https://github.com/hirosystems/stacks-blockchain-api/blob/ed8b6f5839c0c28339bb123e0411902ec78a0209/readme.md#event-replay-instructions

[wileyj]

I think i understand - functionally, i don't see any changes to this repo outside of docs.
the bns option just runs a script to download/extract the legacy bns data - the API is in control of how/when that data is imported.

It's not clear to me from the link though - if you download/extract the data, and define the BNS_IMPORT_DIR env var, starting the API normally will not import the bns data, correct? a user would need to first run an import, despite not having an event-replay file?

[rafaelcr]

if you download/extract the data, and define the BNS_IMPORT_DIR env var, starting the API normally will not import the bns data, correct? a user would need to first run an import, despite not having an event-replay file?

That is correct. That is a good question, though, because IIRC the import will fail if you don't have a replay file even though you do have the BNS data.

[wileyj]

well, good thing i plan on testing this to see what happens!
I noticed the API docs are updated a bit too, i'll take a closer look there and see what adjustments need to be made

[wileyj]

if you download/extract the data, and define the BNS_IMPORT_DIR env var, starting the API normally will not import the bns data, correct? a user would need to first run an import, despite not having an event-replay file?

That is correct. That is a good question, though, because IIRC the import will fail if you don't have a replay file even though you do have the BNS data.

One issue i've found is if you're trying to run the import before having the tsv file...the process fails and the BNS data isn't imported. syncing for a few blocks, then re-running the import works as expected (with BNS data being imported).

I'm wondering if there can/should be a check in the API code if the tsv doesn't exist, but BNS_IMPORT_DIR is defined...import the bns data only and exit vs trying to import events.

[wileyj]

@rafaelcr

attaching a test i ran, where i built the API from master - downloaded the BNS data and defined the variable, with an empty postgres DB.
running the import fails as expected with a tsv file not found error (since the replay file doesn't exist yet).
touch file.tsv and then running the import (the goal being to only import BNS data) - still fails though.

 ➭ node ./lib/index.js import-events \
    --file file.tsv --wipe-db --force
{"level":"info","message":"Shutting down... received beforeExit.","timestamp":"2022-09-21T15:47:20.536Z"}
{"level":"info","message":"App shutdown successful.","timestamp":"2022-09-21T15:47:20.538Z"}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed. Please reopen if needed.