Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pindexer: error on undefined field #4999

Closed
conorsch opened this issue Jan 22, 2025 · 8 comments
Closed

pindexer: error on undefined field #4999

conorsch opened this issue Jan 22, 2025 · 8 comments

Comments

@conorsch
Copy link
Contributor

Describe the bug
pindexer errors out on mainnet db data due to a missing field

To Reproduce
Steps to reproduce the behavior:

  1. Run pindexer (at least v1.0.0) against mainnet events db
  2. Index up to block 3139617
  3. Observe crash

Expected behavior
pindexer ingests block 3139617 and continues without error.

Screenshots

Error message:

2025-01-21T21:37:58.584465Z DEBUG cometindex::indexer::indexing_state: local_rowid=23110210 type_str="block" height=3139617 tx_hash=None
2025-01-21T21:37:58.584787Z  WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT events.rowid, events.type, events.height, …" db.statement="\n\nSELECT\n  events.row
id,\n  events.type,\n  events.height,\n  tx_results.tx_hash,\n  events.attrs\nFROM\n  (\n    SELECT\n      (\n        SELECT\n          height\n        FROM\n          blocks\n        WHERE\n          b
locks.rowid = block_id\n      ) as height,\n      rowid,\n      type,\n      block_id,\n      tx_id,\n      jsonb_object_agg(attributes.key, attributes.value) AS attrs\n    FROM\n      events\n      LEF
T JOIN attributes ON rowid = attributes.event_id\n    WHERE\n      block_id >= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $1\n      )\n      AN
D block_id <= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $2\n      )\n    GROUP BY\n      rowid,\n      type,\n      block_id,\n      tx_id\n
  ORDER BY\n      rowid ASC\n  ) events\n  LEFT JOIN LATERAL (\n    SELECT\n      *\n    FROM\n      tx_results\n    WHERE\n      tx_results.rowid = events.tx_id\n    LIMIT\n      1\n  ) tx_results ON T
RUE\nORDER BY\n  events.rowid ASC\n" rows_affected=0 rows_returned=1 elapsed=1.445861129s elapsed_secs=1.445861129 slow_threshold=1s
Error: error reading from database

Caused by:
    0: error returned from database: field name must not be null
    1: field name must not be null

Additional context
Looks like the problem is caused by slow queries on the target database. Indeed, the database I'm talking to can be quite slow, due to heavy traffic. However, I don't think that's the source of the problem: I think we've got a syntax error in a query. We can rule this out by checking against a local db and verifying that the crash still happens.
@github-actions github-actions bot added the needs-refinement unclear, incomplete, or stub issue that needs work label Jan 22, 2025
@conorsch
Copy link
Contributor Author

To aid in debugging, I've uploaded a fresh copy of a raw cometbft event db, in custom postgres format, here: https://artifacts.plinfra.net/penumbra-1/cometbft-dbdump-height-3141978.dump. Plugging that into a local setup will allow rerunning pindexer quickly to 1) reproduce the problem; and 2) try candidate fixes if necessary.

@conorsch
Copy link
Contributor Author

conorsch commented Jan 22, 2025 •
edited
Loading

I was able to reproduce the failure from that dump, although I can't identify where the null field is coming from. Pulled another dump from a different nodes cometbft db, and also encountered a failure at a different block:

2025-01-22T06:11:00.685928Z DEBUG cometindex::indexer::indexing_state: local_rowid=22489362 type_str="block" height=3138970 tx_hash=None
2025-01-22T06:11:00.686163Z  WARN sqlx::query: slow statement: execution time exceeded alert threshold summary="SELECT events.rowid, events.type, events.height, …" db.statement="\n\nSELECT\n  events.rowid,
\n  events.type,\n  events.height,\n  tx_results.tx_hash,\n  events.attrs\nFROM\n  (\n    SELECT\n      (\n        SELECT\n          height\n        FROM\n          blocks\n        WHERE\n          blocks.
rowid = block_id\n      ) as height,\n      rowid,\n      type,\n      block_id,\n      tx_id,\n      jsonb_object_agg(attributes.key, attributes.value) AS attrs\n    FROM\n      events\n      LEFT JOIN at
tributes ON rowid = attributes.event_id\n    WHERE\n      block_id >= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $1\n      )\n      AND block_id <
= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $2\n      )\n    GROUP BY\n      rowid,\n      type,\n      block_id,\n      tx_id\n    ORDER BY\n
   rowid ASC\n  ) events\n  LEFT JOIN LATERAL (\n    SELECT\n      *\n    FROM\n      tx_results\n    WHERE\n      tx_results.rowid = events.tx_id\n    LIMIT\n      1\n  ) tx_results ON TRUE\nORDER BY\n  e
vents.rowid ASC\n" rows_affected=0 rows_returned=4220 elapsed=18.781166029s elapsed_secs=18.781166029 slow_threshold=1s
Error: error reading from database

Caused by:
    0: error returned from database: field name must not be null
    1: field name must not be null

which makes me suspect we've got a data integrity issue in the databases. In order to debug we'll need to update the reindexer to support v0.81.x migrations and rebuild from scratch. It's been a long while since we've done that on the dbs in question.

@conorsch conorsch removed the needs-refinement unclear, incomplete, or stub issue that needs work label Jan 23, 2025
@conorsch
Copy link
Contributor Author

Prepared a bump for the reindexer in penumbra-zone/reindexer#21, running that code now to get a fresh database.

@conorsch
Copy link
Contributor Author

After regenerating a cometbft database based on changes in penumbra-zone/reindexer#21, I was able to rerun pindexer against that db, and it completed successfully, with no code changes. Confirmed successful up to at least block 3186459 on penumbra-1. However, once I restarted the node to keep that cometbft db up to date, pindexer quickly crashed with the same error reported above.

It appears we've got a regression in pd v1.0.0 that affects event emission: by downgrading the indexing nodes to v0.81.3, but keeping pindexer on v1.0.0, I was able to keep the pipeline up. As soon as I hooked up a node running pd v1.0.0 to the database, pindexer crashed again. That clearly indicates we've got a regression, but I haven't yet identified the shape of the data problem in the db layer itself. I tried queries like

psql query to find any null values (thanks, claude) 
DO $$
DECLARE
    table_rec RECORD;
    column_rec RECORD;
    null_query TEXT;
    result_count INTEGER;
BEGIN
    FOR table_rec IN 
        SELECT schemaname, tablename 
        FROM pg_tables 
        WHERE schemaname = current_schema()
    LOOP
        FOR column_rec IN 
            SELECT column_name, data_type 
            FROM information_schema.columns 
            WHERE table_schema = table_rec.schemaname 
              AND table_name = table_rec.tablename
        LOOP
            null_query := format('
                SELECT COUNT(*) FROM %I.%I WHERE %I IS NULL', 
                table_rec.schemaname, 
                table_rec.tablename, 
                column_rec.column_name
            );
            
            EXECUTE null_query INTO result_count;
            
            IF result_count > 0 THEN
                RAISE NOTICE 'Table: %.%, Column: %, Null Count: %', 
                    table_rec.schemaname, 
                    table_rec.tablename, 
                    column_rec.column_name, 
                    result_count;
            END IF;
        END LOOP;
    END LOOP;
END $$;

But comparing between good/bad dbs didn't show anything significant. Perhaps if we revise the query to look for values that are null or an empty string, we might find more hits. Will continue to investigate the db layer now that indexing functionality is restored.

@conorsch
Copy link
Contributor Author

Ran some tests locally. Given that we know the regression appeared in v1, we can prepare local indexing setups based on both the v0.81.3 and v1.0.0, then inspect the databases that result from those. The error message that pindexer throws points to an undefined field, but doesn't point out where. Let's query across the whole cometbft db and find out where undefineds are lurking.

reusable SQL query to generate statements for searching for nulls 
SELECT 
  'SELECT ''' || table_name || '.' || column_name || ''', ' ||
  'COUNT(*) as total_rows, ' ||
  'COUNT(' || column_name || ') as filled_rows, ' ||
  'COUNT(*) - COUNT(' || column_name || ') as null_count, ' ||
  'CASE WHEN COUNT(*) = 0 THEN 0 ELSE ' ||
  'ROUND(100.0 * (COUNT(*) - COUNT(' || column_name || ')) / COUNT(*), 2) ' ||
  'END as null_percentage ' ||
  'FROM ' || table_name || ';'
FROM information_schema.columns 
WHERE table_schema = 'public';

The workflow is:

  1. run that query against the db
  2. capture the generated statements
  3. edit the statement output to be raw sql
  4. run the generated statements
  5. capture the reported summaries for null finds

Doing that against both databases, we have a clear culprit:

  • block_events.composite_key
  • block_events.key
  • block_events.value
  • event_attributes.composite_key
  • event_attributes.key
  • event_attributes.value

all show significant quantities (~75%) of null values. Notably event_attributes.tx_id is null 100% of the time in both dbs, so we should disregard that for the current investigation. Raw data here:

nulls found on known-good db (v0.81.3) 
       ?column?       | total_rows | filled_rows | null_count | null_percentage 
----------------------+------------+-------------+------------+-----------------
 tx_events.created_at |          0 |           0 |          0 |               0
(1 row)

  ?column?   | total_rows | filled_rows | null_count | null_percentage 
-------------+------------+-------------+------------+-----------------
 events.type |       3112 |        3112 |          0 |            0.00
(1 row)

    ?column?    | total_rows | filled_rows | null_count | null_percentage 
----------------+------------+-------------+------------+-----------------
 attributes.key |       7764 |        7764 |          0 |            0.00
(1 row)

      ?column?       | total_rows | filled_rows | null_count | null_percentage 
---------------------+------------+-------------+------------+-----------------
 block_events.height |       7764 |        7764 |          0 |            0.00
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 tx_results.index |          0 |           0 |          0 |               0
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 events.rowid |       3112 |        3112 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 event_attributes.type |       7764 |        7764 |          0 |            0.00
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 tx_results.rowid |          0 |           0 |          0 |               0
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 events.tx_id |       3112 |           0 |       3112 |          100.00
(1 row)

         ?column?          | total_rows | filled_rows | null_count | null_percentage 
---------------------------+------------+-------------+------------+-----------------
 event_attributes.block_id |       7764 |        7764 |          0 |            0.00
(1 row)

   ?column?    | total_rows | filled_rows | null_count | null_percentage 
---------------+------------+-------------+------------+-----------------
 blocks.height |        763 |         763 |          0 |            0.00
(1 row)

       ?column?       | total_rows | filled_rows | null_count | null_percentage 
----------------------+------------+-------------+------------+-----------------
 event_attributes.key |       7764 |        7764 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 tx_results.created_at |          0 |           0 |          0 |               0
(1 row)

        ?column?         | total_rows | filled_rows | null_count | null_percentage 
-------------------------+------------+-------------+------------+-----------------
 tx_events.composite_key |          0 |           0 |          0 |               0
(1 row)

        ?column?        | total_rows | filled_rows | null_count | null_percentage 
------------------------+------------+-------------+------------+-----------------
 event_attributes.value |       7764 |        7764 |          0 |            0.00
(1 row)

    ?column?    | total_rows | filled_rows | null_count | null_percentage 
----------------+------------+-------------+------------+-----------------
 tx_events.type |          0 |           0 |          0 |               0
(1 row)

       ?column?       | total_rows | filled_rows | null_count | null_percentage 
----------------------+------------+-------------+------------+-----------------
 tx_results.tx_result |          0 |           0 |          0 |               0
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 tx_events.height |          0 |           0 |          0 |               0
(1 row)

      ?column?      | total_rows | filled_rows | null_count | null_percentage 
--------------------+------------+-------------+------------+-----------------
 tx_events.chain_id |          0 |           0 |          0 |               0
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 block_events.key |       7764 |        7764 |          0 |            0.00
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 tx_events.value |          0 |           0 |          0 |               0
(1 row)

          ?column?          | total_rows | filled_rows | null_count | null_percentage 
----------------------------+------------+-------------+------------+-----------------
 block_events.composite_key |       7764 |        7764 |          0 |            0.00
(1 row)

            ?column?            | total_rows | filled_rows | null_count | null_percentage 
--------------------------------+------------+-------------+------------+-----------------
 event_attributes.composite_key |       7764 |        7764 |          0 |            0.00
(1 row)

      ?column?      | total_rows | filled_rows | null_count | null_percentage 
--------------------+------------+-------------+------------+-----------------
 tx_results.tx_hash |          0 |           0 |          0 |               0
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 tx_events.index |          0 |           0 |          0 |               0
(1 row)

        ?column?        | total_rows | filled_rows | null_count | null_percentage 
------------------------+------------+-------------+------------+-----------------
 event_attributes.tx_id |       7764 |           0 |       7764 |          100.00
(1 row)

     ?column?      | total_rows | filled_rows | null_count | null_percentage 
-------------------+------------+-------------+------------+-----------------
 blocks.created_at |        763 |         763 |          0 |            0.00
(1 row)

         ?column?         | total_rows | filled_rows | null_count | null_percentage 
--------------------------+------------+-------------+------------+-----------------
 attributes.composite_key |       7764 |        7764 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 block_events.chain_id |       7764 |        7764 |          0 |            0.00
(1 row)

      ?column?       | total_rows | filled_rows | null_count | null_percentage 
---------------------+------------+-------------+------------+-----------------
 tx_results.block_id |          0 |           0 |          0 |               0
(1 row)

      ?column?      | total_rows | filled_rows | null_count | null_percentage 
--------------------+------------+-------------+------------+-----------------
 block_events.value |       7764 |        7764 |          0 |            0.00
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 blocks.rowid |        763 |         763 |          0 |            0.00
(1 row)

   ?column?    | total_rows | filled_rows | null_count | null_percentage 
---------------+------------+-------------+------------+-----------------
 tx_events.key |          0 |           0 |          0 |               0
(1 row)

     ?column?      | total_rows | filled_rows | null_count | null_percentage 
-------------------+------------+-------------+------------+-----------------
 block_events.type |       7764 |        7764 |          0 |            0.00
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 attributes.value |       7764 |        7764 |          0 |            0.00
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 blocks.chain_id |        763 |         763 |          0 |            0.00
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 events.block_id |       3112 |        3112 |          0 |            0.00
(1 row)

      ?column?       | total_rows | filled_rows | null_count | null_percentage 
---------------------+------------+-------------+------------+-----------------
 attributes.event_id |       7764 |        7764 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 block_events.block_id |       7764 |        7764 |          0 |            0.00
(1 row)

Compare that with:

nulls found on known-bad db (v1.0.0) 
  ?column?   | total_rows | filled_rows | null_count | null_percentage 
-------------+------------+-------------+------------+-----------------
 events.type |      21112 |       21112 |          0 |            0.00
(1 row)

    ?column?    | total_rows | filled_rows | null_count | null_percentage 
----------------+------------+-------------+------------+-----------------
 attributes.key |       5175 |        5175 |          0 |            0.00
(1 row)

      ?column?       | total_rows | filled_rows | null_count | null_percentage 
---------------------+------------+-------------+------------+-----------------
 block_events.height |      21112 |       21112 |          0 |            0.00
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 events.rowid |      21112 |       21112 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 event_attributes.type |      21112 |       21112 |          0 |            0.00
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 events.tx_id |      21112 |           0 |      21112 |          100.00
(1 row)

         ?column?          | total_rows | filled_rows | null_count | null_percentage 
---------------------------+------------+-------------+------------+-----------------
 event_attributes.block_id |      21112 |       21112 |          0 |            0.00
(1 row)

   ?column?    | total_rows | filled_rows | null_count | null_percentage 
---------------+------------+-------------+------------+-----------------
 blocks.height |       5175 |        5175 |          0 |            0.00
(1 row)

       ?column?       | total_rows | filled_rows | null_count | null_percentage 
----------------------+------------+-------------+------------+-----------------
 event_attributes.key |      21112 |        5175 |      15937 |           75.49
(1 row)

        ?column?        | total_rows | filled_rows | null_count | null_percentage 
------------------------+------------+-------------+------------+-----------------
 event_attributes.value |      21112 |        5175 |      15937 |           75.49
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 block_events.key |      21112 |        5175 |      15937 |           75.49
(1 row)

          ?column?          | total_rows | filled_rows | null_count | null_percentage 
----------------------------+------------+-------------+------------+-----------------
 block_events.composite_key |      21112 |        5175 |      15937 |           75.49
(1 row)

            ?column?            | total_rows | filled_rows | null_count | null_percentage 
--------------------------------+------------+-------------+------------+-----------------
 event_attributes.composite_key |      21112 |        5175 |      15937 |           75.49
(1 row)

        ?column?        | total_rows | filled_rows | null_count | null_percentage 
------------------------+------------+-------------+------------+-----------------
 event_attributes.tx_id |      21112 |           0 |      21112 |          100.00
(1 row)

     ?column?      | total_rows | filled_rows | null_count | null_percentage 
-------------------+------------+-------------+------------+-----------------
 blocks.created_at |       5175 |        5175 |          0 |            0.00
(1 row)

         ?column?         | total_rows | filled_rows | null_count | null_percentage 
--------------------------+------------+-------------+------------+-----------------
 attributes.composite_key |       5175 |        5175 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 block_events.chain_id |      21112 |       21112 |          0 |            0.00
(1 row)

      ?column?      | total_rows | filled_rows | null_count | null_percentage 
--------------------+------------+-------------+------------+-----------------
 block_events.value |      21112 |        5175 |      15937 |           75.49
(1 row)

   ?column?   | total_rows | filled_rows | null_count | null_percentage 
--------------+------------+-------------+------------+-----------------
 blocks.rowid |       5175 |        5175 |          0 |            0.00
(1 row)

     ?column?      | total_rows | filled_rows | null_count | null_percentage 
-------------------+------------+-------------+------------+-----------------
 block_events.type |      21112 |       21112 |          0 |            0.00
(1 row)

     ?column?     | total_rows | filled_rows | null_count | null_percentage 
------------------+------------+-------------+------------+-----------------
 attributes.value |       5175 |        5175 |          0 |            0.00
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 blocks.chain_id |       5175 |        5175 |          0 |            0.00
(1 row)

    ?column?     | total_rows | filled_rows | null_count | null_percentage 
-----------------+------------+-------------+------------+-----------------
 events.block_id |      21112 |       21112 |          0 |            0.00
(1 row)

      ?column?       | total_rows | filled_rows | null_count | null_percentage 
---------------------+------------+-------------+------------+-----------------
 attributes.event_id |       5175 |        5175 |          0 |            0.00
(1 row)

       ?column?        | total_rows | filled_rows | null_count | null_percentage 
-----------------------+------------+-------------+------------+-----------------
 block_events.block_id |      21112 |       21112 |          0 |            0.00
(1 row)

The raw data is hard to read, but if you diffoscope them and vgrep the "null_percentage" column, then the differences will stand out. I couldn't figure out how build an SQL that would summarize the emitted null percentages, thus the visual inspection, but hey, it worked.

My operating intuition is that the cnidarium crate move (which also broke #5004) is the culprit here, given that it has Events of its own, but still need to confirm that as the missing datatype between both dbs.

@erwanor
Copy link
Member

erwanor commented Jan 28, 2025

Thanks for the write-up, and digging into it, this is helpful

My operating intuition is that the cnidarium crate move (which also broke #5004) is the culprit here, given that it has Events of its own, but still need to confirm that as the missing datatype between both dbs.

I don't understand why the crate move would cause this. At this point, I don't think it did. My sense is that the source of the problem is with the ecosystem crate bump, either we missed something during review, or a semantics change in an upstream, or both. There's an easy way to test this hypothesis: cnidarium has a published v0.80.0.

I can do a small pass over the ABCI event handling in the protocol later today, and report back.

@erwanor
Copy link
Member

erwanor commented Jan 28, 2025 •
edited
Loading

There was a small but important error in translating the ABCI event indexing layer while doing the tendermint ecosystem bump: #5009 that I missed during review.

@erwanor erwanor mentioned this issue Jan 28, 2025
Merged
2 tasks
conorsch added a commit that referenced this issue Jan 29, 2025
@conorsch
test(pindexer): add integration tests to validate db contents
b9284fd 
Adds some feature-gated integration tests, not yet hooked up to CI,
to sanity check that the ABCI events pipeline is working:

  pd -> cometbft -> postgres -> pindexer

The integration tests talk to local postgres databases and make
assertions about their contents. There's also a small change to the
process-compose orchestration, instructing pindexer to wait for the
fullnode services to come up, lest it error out early due to an empty
database. The error I observed on an empty db was:

  Error: error occurred while decoding column 0: unexpected null;
  try decoding as an `Option`
  Caused by:
    unexpected null; try decoding as an `Option`

Which is distinct from the error reported in #4999. By waiting
a bit longer we can ensure a clean start even on a fresh devnet.

Will follow up with subsequent PRs to make sure these tests run in CI.
@conorsch conorsch mentioned this issue Jan 29, 2025
Closed
5 tasks
conorsch added a commit that referenced this issue Jan 29, 2025
@erwanor @conorsch
app: fix broken event indexing (#5009)
bc912f0 
## Describe your changes

We missed this on reviewing #4963 which causes a series of unfortunate
events in the data pipeline: #4999

The important diff:

```
-set_index(false)
+set_index(true)
```

## Issue ticket number and link

## Checklist before requesting a review

- [x] I have added guiding text to explain how a reviewer should test
these changes.

I have not tested this change. To test it we can run `pd` and check if
events are making it to the comet pg indexer.

- [x] If this code contains consensus-breaking changes, I have added the
"consensus-breaking" label. Otherwise, I declare my belief that there
are not consensus-breaking changes, for the following reason:

  > event indexing layer

---------

Co-authored-by: Conor Schaefer <[email protected]>
@conorsch
Copy link
Contributor Author

Resolved via #5009.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@conorsch @erwanor