[Core] Make encoder-decoder inputs a nested structure to be more composable

[Core] Make encoder-decoder inputs a nested structure to be more composable #20148