Tunneling: prepare(mediaSource) issue - weird "play/pause behaviour"

[kvillnv]

Hello,

Issue description

Weird simultaneous play/pause state In tunnelled mode after preparing the player.
Using the same code in non tunnelled mode works correctly.

Reproduction steps

As always, when I want to play a media I use these classic exoplayer methods:

playWhenReady(true)
prepare(mediaSource)

(works correctly in tunnelling)

And when I want to just prepare and not start playing, Instead of the previous code I usually just do :

prepare(mediaSource)

(ExoPlayerImpl's playWhenReady default value is "false" I don't need to set it)

• In non tunnelled mode: the behaviour is correct: my media is loaded and is in a paused state.
• In tunnelled mode: my media is loaded, but is in a weird state: the player looks like it is in between paused and playing: it plays a few frames then pauses and keeps going on repeating this weird play/pause (approximately 1second cycles).
  Also happens if I seekTo(mPosition) before or just after prepare(mediaSource)

According to logs, all values are the same in tunnelled and non-tunnelled, however I don't have the same behaviours. (The debugger shows the same information: playWhenReady:false, playbackState:ready)

Even if I try to use any of the methods of the EventListener I can't do anything because the onPlayerStateChanged gives me the same result then when I am paused in non-tunelled mode.

Also, I noticed after the bug if I press play, it plays correctly, then I wait a few seconds and manually pause, my player will be correctly paused. it only happens when I prepare the media and not setPlayWhenReady(true)

Do you have any idea what is causing this?
Thanks.

Version of ExoPlayer being used

2.9.3

Device(s) and version(s) of Android being used

Custom Sagemcom STB
Android 7.1.1

[andrewlewis]

It sounds like the video tunneling implementation is presenting video frames on the assumption that the audio track is in the playing state. I think it should ignore the hardware A/V sync timestamps in audio buffers until the corresponding audio played out (while the AudioTrack is in the playing state).

Can you reproduce this on any other devices that support video tunneling? If not, it's likely to be a device-specific bug and should be reported to the manufacturer of the "Custom Sagemcom STB" you're using.

[kvillnv]

@andrewlewis
I just reproduced it on another device.
Device: AMINO Amigo7
Android: 8.0.0
Detailed Specs: https://www.aminocom.com/products/amino-vu/client-devices/amigo-7x

Also, I was able to reproduce the bug on exoplayer2.demo from the release-v2 branch, using the default sources (for example all Youtube DASH)

Here is how to reproduce it:

com.google.android.exoplayer2.demo.PlayerActivity

• Enable tunneling
  line 202

-      trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder().build();
+      trackSelectorParameters = new DefaultTrackSelector
+              .ParametersBuilder()
+              .setTunnelingAudioSessionId(C.generateAudioSessionIdV21(getApplicationContext()))
+              .build();

• Cancel playWhenReady
  line 435

-      player.setPlayWhenReady(startAutoPlay);
+    //  player.setPlayWhenReady(startAutoPlay);

[kvillnv]

Hello @andrewlewis,
Please let me know if you need any additional information to solve this issue.
Thanks

[kvillnv]

We were able to reproduce the bug again on a third device.
I have just sent you the information privately by e-mail.

To sum up:

• This bug happened on 3 Android TV STBs from 3 different manufacturers.
• 2 are Android 7.1.1 and the other one Android 8.0.0.
• Happens when tunneling enabled.

Thanks for your help.

[kvillnv]

@andrewlewis, did you have a chance to try reproducing the steps I mentioned in the sample?
Please share your observations.

The issue is still tagged "need more info. I believe the provided information should demonstrate the player's unexpected behaviour.

[andrewlewis]

Sorry for the delay. It may still take a while before we get round to looking at this.

[kvillnv]

The issue was opened 1 month ago.
@andrewlewis can we have an ETA?
Thanks

[andrewlewis]

I wasn't able to reproduce this on two Sony Bravia ATVs (running L and N), so I think it needs to be reported to the relevant chipset manufacturer(s) or OEM(s). Most of the implementation of the video tunneling path is up to the device.

Side note: the Sony Bravia TVs don't render a first frame at all while paused, so in a sense they have the opposite problem to the devices you tried. They are not reading the A/V sync timestamp for the first unplayed frame in the AudioTrack's buffer so don't render anything until the audio track actually starts playing, while it sounds like the devices you've tried are reading several frames from the AudioTrack's buffer even before we've called AudioTrack.play.

[kvillnv]

@andrewlewis indeed I tried a Bravia, as you said the screen stays black until I play.

So far there is an issue on all the devices we tested. Even if it is slightly different, it is always about rendering the first frame on tunneling while playWhenReady=false.
There are too much devices to consider, it is not possible to ask each manufacturer to adapt their hardware for exoplayer. I have not seen a single device overcoming this tunneled first frame rendering problem.
I am pretty sure most people always set the playWhenReady=true, and I dont know how many people are using exoplayer in tunneling.

Can you confirm that some tweaking in the following classes will not change anything?
RenderersFactory / MediaCodecRenderer / EventListener / EventHandler

Here are the different tweaks I was thinking about:

• 1. Make Exoplayer handle waiting for the first frame to be rendered before pausing just like in non tunneled mode. There might be some missing observers, thus having every device reacting its own way for the same exoplayer event.
• 2. Or always render the first frame before pausing, with an option to show it or not.

[andrewlewis]

As a general point: I think this is not really about adapting devices to ExoPlayer, but instead implementing the APIs provided by the Android framework in a consistent way. With tunneling this seems to be particularly problematic because so much is left up to the device OEM and verifying behavior programmatically via an automated test is difficult.

That said, if ExoPlayer is using the APIs wrongly, we should absolutely fix that. If there is evidence that this is the case we need to investigate further.

For this specific issue, I wonder if the device should be inspecting the A/V sync headers queued at the start of unplayed audio in the AudioTrack and using that to output a single frame, to permit the "starting video playback paused" use case. It seems that your device is indeed looking at the unplayed audio, otherwise it wouldn't be outputting any frames until we call AudioTrack.play(), while the Sony TVs appear to be deciding what frames to show only based on played audio.

As you are probably aware, the app doesn't directly control when video frames are rendered when using tunneling mode. I thought about doing a workaround like you suggested, where we would start playback on the AudioTrack, wait until a frame is rendered, then pause it if needed. This is pretty bad though, for a variety of reasons: there's no guarantee that only one frame would be rendered, audio might be played out, it probably wouldn't fix the issue on your device if it's going to look at unplayed audio anyway, and (probably most importantly of all) it seems like it would add a lot of complexity.

I'll follow up with the media framework team to get more information on how this should work.

[kvillnv]

All right, thank you. Please keep us updated.

[kvillnv]

Hello,
I would like to know if you are going to make a fix for this?
All the devices tested so far have an issue.
Did you have any feedback from the media framework team?

[andrewlewis]

Unfortunately it seems that the behavior of tunneling while paused at startup is not verified by any automated tests, nor documented as a compatibility requirement. We have an internal tracking issue for the media framework team to provide guidance to OEMs about how this should work [Internal: b/126155094] and I hope this will be along the lines of my comment above (inspecting A/V headers in unplayed data while paused to decide what single frame to show).

I'm not currently planning to make a workaround for the reasons in my previous comment. If you have any suggestions for a good way to work around inconsistent behavior across devices, or if you can see something wrong with how we're using the APIs at the moment, please give some more details. Thanks!

[kvillnv]

All right thanks for pushing it to the media framework team. Providing the OEMs the correct guidance is probably the best solution.
Just to be clear, are you talking about Android's media framework team, or exoplayer's?
For sure if until now OEMs did not receive enough info, they all have their own implementation. So I agree that a workaround on our side would make no sense.

I will let you know if I have anymore information.
And please I'd like to be updated about this internal issue. I guess it is private, if not how can I see it?

[andrewlewis]

I mean the team that looks after lower-level components like MediaCodec, AudioTrack, etc.. I'll plan to keep this issue updated with any developments on the internal bug.