isLiveTable check breaks Kinesis like streams

[npawar]

PR https://github.com/apache/pinot/pull/7756/files introduced a check for liveTable in the path of adding new partitionGroups.
This branch was introduced in setupNewPartitionGroup (prior to this PR we had just the else part)

if (isLiveTable) {
    startOffset = getPartitionGroupSmallestOffset(streamConfig, partitionGroupId);
 } else {
    startOffset = partitionGroupMetadata.getStartOffset();
 }

For new table, we go to the else, and for existing table detecting a new partitionGroup we go to if. Within the if, a call is made to getNewPartitionGroupMetadataList(streamConfig, Collections.emptyList());. For Kinesis like streams, the response from this call depends on the currentList passed. If a Kinesis like stream received empty list, it will only return the very first parent shards. This is because in Kinesis, the shards have a sequence (we started with 0, split it to 1, 2, so 1, 2 will only be returned if the current state tells it that 0 is done ingesting).

As a result of this PR change, no new shards can get detected in Kinesis.

[npawar]

Fixing in #8053 by using the patch that was suggested in an alternative PR #7743

Tagging @KKcorps , who detected this bug when enhancing integration tests for Kinesis. The current integration test misses checking this.
Tagging @KKcorps also to take note of this requirement when working on #7058

[npawar]

fyi @mcvsubbu @Jackie-Jiang

[Jackie-Jiang]

Have you verified that the approach in #7743 works? I feel it should work since it follows the current way of fetching offsets.
However, the issue for the current approach is that getPartitionGroupSmallestOffset() won't give correct offset for Kinesis, and I think that should be fixed because that is also used to fix the errored consuming segment.

Adding @mcvsubbu @KKcorps for more inputs

[KKcorps]

The computePartitionGroupMetadata in KinesisStreamMetadataProvider returns startingSequenceNumber of each shard in case an empty partitionGroupConsumptionStatuses list is passed. This is true for this scenario and hence correct offsets are being returned.

[npawar]

Yup 7743 should work as that one doesn't use empty list for current

[npawar]

Regarding, getPartitionGroupSmallestOffset(), also agree that this should be fixed regardless for Kinesis as it is used in validation manager. But that's a much smaller and infrequent case, so i think we should prioritize first fixing the bigger case of broken new partitiongroups.