[HUDI-9249]Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI

[wangyinsheng]

Change Logs

Support displaying InsertIntoHoodieTableCommand metrics in Spark Web UI

implement steps

• Change the return type of BaseHoodieWriteClient.commit from boolean to CommitStatus(contains a boolean status success and a HoodieCommitMetadata)
• update metrics of InsertIntoHoodieTableCommand from HoodieCommitMetadata

Before

After

Impact

Change the return type of BaseHoodieWriteClient.commit from boolean to CommitStatus(contains a boolean status success and a HoodieCommitMetadata)

Risk level (write none, low medium or high below)

low

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[wangyinsheng]

@hudi-bot run azure

[wangyinsheng]

@hudi-bot run azure

[hudi-bot]

CI report:

• 5afa0a4 UNKNOWN
• f42ddb2 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

The CommitStatus can be eliminated if we just return an option of HoodieCommitMetadata ?, empty means a failure.

[wangyinsheng]

In case of 'EmptyCommit', there is no HoodieCommitMetadata produced， but need return true

hudi/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java

Lines 222 to 229 in 93b4da3

	public boolean commitStats(String instantTime, List<HoodieWriteStat> stats,
	Option<Map<String, String>> extraMetadata,
	String commitActionType, Map<String, List<String>> partitionToReplaceFileIds,
	Option<BiConsumer<HoodieTableMetaClient, HoodieCommitMetadata>> extraPreCommitFunc) {
	// Skip the empty commit if not allowed
	if (!config.allowEmptyCommit() && stats.isEmpty()) {
	return true;
	}

[danny0405]

For this case, we can return null?

[wangyinsheng]

For this case, we can return null?

Many callers need to verify success or failure， return null for EmptyCommit case and return option for other cases will change simple boolean check to commitMetadataOpt == null || commitMetadataOpt.ifPresent, This is a bit weird，I Idon't think it is a good way。

hudi/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java

Lines 588 to 594 in 93b4da3

	boolean success = writeClient.commit(instant, writeResults, Option.of(checkpointCommitMetadata),
	tableState.commitAction, partitionToReplacedFileIds);
	if (success) {
	reset();
	this.ckpMetadata.commitInstant(instant);
	LOG.info("Commit instant [{}] success!", instant);
	} else {

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala

Lines 1002 to 1011 in 93b4da3

	val commitSuccess =
	client.commit(tableInstantInfo.instantTime, writeResult.getWriteStatuses,
	common.util.Option.of(new java.util.HashMap[String, String](metaMap.asJava)),
	tableInstantInfo.commitActionType,
	writeResult.getPartitionToReplaceFileIds,
	common.util.Option.ofNullable(extraPreCommitFn.orNull))
	if (commitSuccess) {
	log.info("Commit " + tableInstantInfo.instantTime + " successful!")
	}

[wangyinsheng]

@danny0405 PTAL

[wangyinsheng]

@danny0405 PTAL

[danny0405]

not sure how the metrics work here because the commit has been finished, does it still make sense to show them on the UI ? can we just read the commit metadata from timeline for these metircs instead of touching the core API now ?

[wangyinsheng]

now sure how the metrics work here because the commit has been finished, does it still make sense to show them on the UI ? can we just read the commit metadata from timeline for these metircs instead of touching the core API now ?

Reading commit metadata through the call command can indeed retrieve these metrics, However, displaying the corresponding commit metrics directly on the Spark UI can simplify the troubleshooting process to some extent, eliminating the need for additional query actions. Additionally, the data displayed in the Spark UI comes from the spark event log, and the Spark UI of long-past run instances can be restored through the event log, with these commit metrics being displayed accordingly. In contrast, querying commit matedata from a long time ago via the API can be relatively cumbersome.