Panic in firehose while processing block

[let4be]

Network: BSC Test net
Firehose version: v1.3.1
Geth: bsc-v1.1.18-fh2.1-2

Steps to reproduce - start firehose sync, try doing an RPC request from substream code which results in an error like this
WARN (rpc-cache) retrying RPCCall on non-deterministic RPC call error {"error": "rpc error (code -32000): missing trie node cba260375b8a093ff89fa569a257fddee8ecec5178b4590a9238b64d71950030 (path )", "at_block": "2720c990fc3b507cd22f08ed2804f1e6c8c324e04b32e2ce70819ca40795dfe6", "endpoint": "http://localhost:8547"} and then stop substreams process(either ctrl-c or sigkill)

Firehose log during the panic: https://pastebin.com/HLzRjZ49
Probably this error should be handled more gracefully

[maoueh]

This is an error coming from your RPC node. It means mainly that you connected a non-archive node as your --substreams-rpc-endpoints value and your RPC node is no able to handle the "archive" request.

This being a non-deterministic error returned from the RPC node we have no choice to retry forever, because the error could resolved if an archive node is hit, so there is no other error to mention.

We could improve the log however to detect what seems like a non-archive node.

[maoueh]

I see the stack printed in your logs. Indeed, context canceled should not be printed as a panic. Seems unrelated to the actual problem described.

[let4be]

Due to my initial misunderstanding I was pointing it to an RPC exposed by geth(that's ran by firehose)

But no matter what - panic is a bad thing... Why do we see a panic in the log when panic can be handled gracefully?

I also don't think panic originates in geth, logs say it's in Firehose/substreams, correct me if I'm wrong

[let4be]

I see the stack printed in your logs. Indeed, context canceled should not be printed as a panic. Seems unrelated to the actual problem described.

Initial problem described is a step that leads to this panic... I described the exact steps to cause the panic in my first post

[maoueh]

But no matter what - panic is a bad thing... Why do we see a panic in the log when panic can be handled gracefully?

It's handled, we just print it fully, I just mean it's an handled panic. No panic should happen, and in this particular case, context canceled should not have generated such problem, definitely a bug.

Due to my initial misunderstanding I was pointing it to an RPC exposed by geth(that's ran by firehose)

It must point an archive node

Initial problem described is a step that leads to this panic... I described the exact steps to cause the panic in my first post

With an archive node correctly configured, nothing around this should happen anymore