[Proposal] Adding an option to enable/disable X-TIMESTAMP-MAP in WebVTT

[ozt860071]

Please prefix your issue with one of the following: [BUG], [PROPOSAL], [QUESTION].

CCExtractor version (using the --version parameter preferably) : X.X

In raising this issue, I confirm the following (please check boxes, eg [X] - and delete unchecked ones):

• I have read and understood the contributors guide.
• I have checked that the bug-fix I am reporting can be replicated, or that the feature I am suggesting isn't already present.
• I have checked that the issue I'm posting isn't already reported.
• I have checked that the issue I'm porting isn't already solved and no duplicates exist in closed issues and in opened issues
• I have checked the pull requests tab for existing solutions/implementations to my issue/suggestion.
• I have used the latest available version of CCExtractor to verify this issue exists.

My familiarity with the project is as follows (check one, eg [X] - and delete unchecked ones):

• I have never used CCExtractor.
• I have used CCExtractor just a couple of times.
• I absolutely love CCExtractor, but have not contributed previously.
• I am an active contributor to CCExtractor.

Necessary information

• Is this a regression (did it work before)? [x] NO | [ ] YES - please specify the last known working version
• What platform did you use? [ ] Windows - [ ] Linux - [x] Mac
• What were the used arguments? -out=webvtt

Video links (replace text below with your links)

Additional information
Currently when WebVTT is specified as output, X-TIMESTAMP-MAP is automatically added. And if no subtitle text is found in the input, it generates an empty WebVTT, with just one line "WEBVTT" in the file.

There are potentially two issues with this implementation.
(a) X-TIMESTAMP-MAP is not a WebVTT requirement, but Apple HLS requirement, this may cause a problem if it is used in non-HLS environment. It should be an option to turn it off.
(b) On the other hand, Apple HLS spec says "an X-TIMESTAMP-MAP metadata header SHOULD be added to each WebVTT header", which includes the empty ones. The spec also says "If a WebVTT segment does not have the X-TIMESTAMP-MAP, the client MUST assume that the WebVTT cue time of 0 maps to an MPEG-2 timestamp of 0.". But unfortunately, not all clients follow this rule. It is safer to add X-TIMESTAMP-MAP in all VTT files, if it is enabled.

So, my proposal is to add an option for WebVTT, whether X-TIMESTAMP-MAP is enabled or disabled. And if it is enabled, it will be added in every VTT file it generates, including one with no caption cues.

(cf. "HTTP Live Streaming": https://tools.ietf.org/html/rfc8216 section 3.5)

[NilsIrl]

What happens if X-TIMESTAMP-MAP is disabled and the WebVTT cue time 0 doesn't map to an MPEG-2 timestamp of 0?

[NilsIrl]

What would be the reason to not have the X-TIMESTAMP-MAP?

[NilsIrl]

Wouldn't it be better to have the X-TIMESTAMP-MAP header present all the time?

[ozt860071]

You’ll lose caption, and in worst case, video freeze.

…

On Dec 4, 2019, at 3:34 PM, Nils ANDRÉ-CHANG ***@***.***> wrote: What happens if X-TIMESTAMP-MAP is disabled and the WebVTT cue time 0 doesn't map to an MPEG-2 timestamp of 0? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1127?email_source=notifications&email_token=AB4ZYEQBI7FXHMYUFSFJIE3QXA5B3A5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF64S7I#issuecomment-561891709>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4ZYERROT4I2YN2DRAMEYTQXA5B3ANCNFSM4JQUKKPA>.

[ozt860071]

Since it is HLS requirement, this may affect negatively in non-HLS use cases. I hope not. I don’t mind having it all the time, though.

…

On Dec 4, 2019, at 3:35 PM, Nils ANDRÉ-CHANG ***@***.***> wrote: What would be the reason to not have the X-TIMESTAMP-MAP? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1127?email_source=notifications&email_token=AB4ZYERUXAUFA2NTLMFRT43QXA5FLA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF64VVY#issuecomment-561892055>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4ZYERROQQFDJSCDSCW4I3QXA5FLANCNFSM4JQUKKPA>.

[ozt860071]

I am fine with this. But I don’t know how it affects in other use cases where this is not required. Thanks

…

On Dec 4, 2019, at 3:40 PM, Nils ANDRÉ-CHANG ***@***.***> wrote: Wouldn't it be better to have the X-TIMESTAMP-MAP header present all the time? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1127?email_source=notifications&email_token=AB4ZYEXAGJ4VZMFAJXCOZELQXA5V7A5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF65C2A#issuecomment-561893736>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4ZYEXCUV3XZKLDLRY7BXTQXA5V7ANCNFSM4JQUKKPA>.

[NilsIrl]

@cfsmp3 What should I do? I feel like the best here is to always use X-TIMESTAMP-MAP.

[NilsIrl]

There is also the general problem that ccextractor creates files even when it can't find anything making this problem more general and not limited to .vtt. @cfsmp3

[ozt860071]

But for HLS at least, I need an output even if no caption text is contained in the .vtt file. The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization. Thanks

…

On Dec 5, 2019, at 12:09 PM, Nils ANDRÉ-CHANG ***@***.***> wrote: There is also the general problem that ccextractor creates files even when it can't find anything making this problem more general and not limited to .vtt. @cfsmp3 <https://github.com/cfsmp3> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1127?email_source=notifications&email_token=AB4ZYEUVS7QIUBJILRT5MBLQXFNZBA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGB7OQQ#issuecomment-562296642>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4ZYERHWKLVUU5W2PXJVT3QXFNZBANCNFSM4JQUKKPA>.

[NilsIrl]

The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization.

why would you need to synchronize if you don't have subtitles?

[cfsmp3]

The client needs the get the timing info from X-TIMESTAMP-MAP for synchronization.

why would you need to synchronize if you don't have subtitles?

It's part of the HLS specification. If there are subtitles in WebVTT:

Subtitles

5. Subtitle requirements
   5.1. Subtitles MAY be provided.
   5.2. Subtitles MUST be WebVTT (according to the HLS specification) or IMSC1 in fMP4.
   5.3. WebVTT subtitles MUST be in text files, with an X-TIMESTAMP-MAP according to the HLS specification.

References:
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices
https://tools.ietf.org/html/rfc8216#page-13

Note that Apple there says MUST (in 5.2) but the RPC8216 says SHOULD.
But still, if we generate a .vtt file, even if it has no subtitles, it should have the required header. Otherwise players may refuse to play anything at all.

[NilsIrl]

The thing is that you can't synchronize between 2 things when you're missing 1 of the 2, in this case subtitles.

[cfsmp3]

An empty subtitle file is not the same as no subtitle file. If there's a subtitle file even if it has no actual subtitles inside you need to be able to know where it starts relative to the video.

In any case if the specs say that's the way it is then that's what we have to implement, unless you want to get in touch (you can) with the RFC8216 writers and discuss :-)

Otherwise we're just not standard compliant and that's not a good thing to be. In fact, if you read the RFC you will see that in a bunch of places it says that clients MUST refuse to play media that is not compliant. Not that they should do their best to play it anyway, but the opposite.

And well, there's a good reason for that: To force all programs generating media to follow the specs.

[NilsIrl]

If there's a subtitle file even if it has no actual subtitles inside you need to be able to know where it starts relative to the video.

1. As in where the subtitles are located in the video file in bytes.
2. Or as in the timestamp at which the subtitles start in seconds (or the duration unit that is used).

[cfsmp3]

You want to sync using time, not bytes. There's some reasons for this, one of them being that you might be starting the stream in the middle and not have access to anything before you started playing - for example, when you turn on your TV your TV needs to start playing the stream from the point you turn it on, Also, the stream never ends...

Second, even if we were talking about actual files with a beginning and an end, they will contain at the very least 1 video track and 1 audio track (possibly more of audio), so what byte offset would you use?

[NilsIrl]

In case 1, the value would just be metadata that would kinda be useless without the exact same video file.

For example in the East Enders file here is the header:

X-TIMESTAMP-MAP=MPEGTS:1090971027,LOCAL:00:00:00.000

MPEGTS is the duration since the start of the stream (before the start of the video file)?

[cfsmp3]

Not exactly. The MPEGTS is a clock that runs at 90,000 Hz and that rolls over roughly every 26 hours. Forget about the "start of the stream", for all purposes assume the stream doesn't have a beginning or an end, it's 24x7 (assuming we're talking about a TV channel). You need to sync the audio, video and packets. Of course packets interleaved in a way that they are playable without having a huge buffer (which would also cause a delay, and for example when watching sports you want the less possible latency), but still you need to sync everything. .srt and other video formats do start at 00:00 assuming that you are going to attach them to a file that also starts at 00:00. That's fine, we love .srt and CCExtractor produces timing relative to the start of a file :-) But professional use, in broadcast media, is just different. If someone from a media company tells us they need the MPEG clock to sync, they're probably know what they are doing.

…

On Sun, Jan 12, 2020 at 1:28 PM Nils ANDRÉ-CHANG ***@***.***> wrote: In case 1, the value would just be metadata that would kinda be useless without the exact same video file. For example in the East Enders file here is the header: X-TIMESTAMP-MAP=MPEGTS:1090971027,LOCAL:00:00:00.000 MPEGTS is the duration since the start of the stream (before the start of the video file). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1127?email_source=notifications&email_token=ABNMTWOWJ6GPZBRW7CK7IITQ5ODRDA5CNFSM4JQUKKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIXEWQQ#issuecomment-573459266>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNMTWNKZLSG6XMWWIYD3BDQ5ODRDANCNFSM4JQUKKPA> .

[kdrag0n]

Shouldn't this issue be closed since #1176 has been merged?

[NilsIrl]

Well the underlying issue is still present so it doesn't solve the original problem

(THE header doesn't show up all the time). I'm going to reopen my PR

[NilsIrl]

Is there a file I can test it with? (a file without subtitles).

[cfsmp3]

Is there a file I can test it with? (a file without subtitles).

You can use any file - just manually select --datapid pointing to a PID that is not present in the stream.

[NilsIrl]

You can use any file - just manually select --datapid pointing to a PID that is not present in the stream.

This shouldn't affect the header right?

[NilsIrl]

https://github.com/NilsIrl/ccextractor/blob/db646f50ac14a772de9f374b2b40e05dbc6c0da9/src/lib_ccx/general_loop.c#L991-L1006

If you look at the above snippet, data_node == NULL when a wrong -datapid is set. The thing is that in order to get the value for X-TIMESTAMP-MAP, data_node needs to be used. I don't know if the problem has to do with the fact -datapid is used. From what I understand it's not the fact we don't have subtitles that prevents from having a correct X-TIMESTAMP-MAP so it should be possible.

[cfsmp3]

OK so cheating with datapid is not an an option :-(

Check out our list of samples.
https://drive.google.com/drive/folders/0B_61ywKPmI0TUUk5LXJPeG1feFE?usp=sharing

Specifically the UK TV one has one that says "no subs"

[ozt860071]

Sorry not to see this in 0.89.
(from CHANGES.TXT)

• New: Added an option to disable timestamps for WebVTT (In response to issue [Proposal] Adding an option to enable/disable X-TIMESTAMP-MAP in WebVTT #1127)
  This is not implemented, either.

[cfsmp3]

Sorry not to see this in 0.89.
(from CHANGES.TXT)

• New: Added an option to disable timestamps for WebVTT (In response to issue [Proposal] Adding an option to enable/disable X-TIMESTAMP-MAP in WebVTT #1127)
  This is not implemented, either.

https://github.com/CCExtractor/ccextractor/pull/1176/files

[ozt860071]

As you mentioned,

Note that Apple there says MUST (in 5.2) but the RPC8216 says SHOULD.
But still, if we generate a .vtt file, even if it has no subtitles, it should have the required header. Otherwise players may refuse to play anything at all.

This needs to be resolved. I think the code is ready.