Other Language Audio

[shawna-slh]

From Estel·la:

One of the new accessible elements which is not in the list are:

Audio subtitles/Spoken subtitles - Provides a spoken version of the subtitles if the audio is in a foreign language and a subtitled translation is available. They are an audio alternative to people who are blind and others who cannot see and understand the subtitles. It also helps people with cognitive impairments to better understand what is written in the text.

[shawna-slh]

Some movies are available in different spoken languages. The visual film is the same, the speech is different. This is often called "dubbing" or "re-voicing".

Is that essentially what you are referring to? Or something different?

Are there new or other ways to provide this specifically related to accessibility? (for example, text-to-speech tools that read subtitle files in sync with the video)

[eoncins]

Audio subtitles/spoken subtitles/spoken captions are not products of re-voicing/ dubbing or voice-over. The latter are secondary audio version of a film or video produced by voice talent, translators, and dialogue writers in a language other than the original language of the film or video
Note 1 to entry: When dubbing, this is timed to match the voicing or mouth movements of animated characters or the original actors.

According to ISO/IEC TS 20071-21:2015(en) Information technology — User interface component accessibility — Part 21: Guidance on audio descriptions
Refers to audio subtitles/spoken subtitles/spoken captions under 2.3.4:

• spoken subtitles
• audio subtitles
• spoken captions

subtitles on audiovisual content that are read aloud and spoken over the audio in subtitled audiovisual content

2.3.5
spoken subtitles with audio description
audiovisual content which is both subtitled and audio described through the narrator reading the translation after stating that a subtitle appears
Note 1 to entry: Subtitles can also be identified through alternative voicing, synthetic speech, or sound indicator.
Note 2 to entry: Spoken subtitles often use a different voice to the voice of the audio describer to avoid confusion between the dialogue and the description.
Note 3 to entry: Spoken subtitles and audio description can be available to users separately or together.
Note 4 to entry: Spoken subtitles are synchronized with the text on screen.

https://www.iso.org/obp/ui/#iso:std:iso-iec:ts:20071:-21:ed-1:v1:en

[shawna-slh]

Thank you for an official definition.

Do I understand correctly that it is conceptually the same as dubbed a different language? The main difference is that the spoken subtitles are synchronized with the text on screen, rather than timed to match the voice/mouth movements? (And probably usually it is done afterwards with less-professional production, and specifically for accessibility?)

I would like to understand more how it is implemented in practice. :-)

Is it usually spoken version of intralingual subtitles only (that is, not including non-speech sounds)?

In practice, does it usually totally replace the main audio? Or, is the audio re-mixed so the main audio spoken words are quiet and the spoken subtitles are audible? Or, is the main audio included as is, and there are pauses for the spoken subtitles?

Can you provide some examples?

Any practical guidance for this that you know of -- either formal or informal?

[eoncins]

Hello Shawn,

1.- Spoken subtitles/Audiosubtitles are mainly aimed at blind and visually impaired audiences. In countries with a subtitling tradition like Finland and Holand, to name a couple, no dubbed version is provided, therefore blind people do not have access to the audiovisual content if they do not speak the language of the movie. Also they have the audio in one language and the audio description in another language. This is also a problem in terms of cognitive load. An implemented solution in these countries is to provide spoken subtitles to grant access. Yes, it has been conceived as an accessibility measure.

Here is a link to an explanation of spoken subtitles or audio subtitles provided in a MOOC that we developed for the scenic arts:
(https://www.coursera.org/lecture/accessibility-scenic-arts/audio-subtitling-gcQBd)

2.- Is it usually spoken version of intralingual subtitles only (that is, not including non-speech sounds)?
Yes that's right. It may depend if the non-speech is integrated in the subtitles.

3.- In terms of examples. There is an article that will to be published in (2020). Please notice that not all scenarios and workflows apply to the Guideline but I include them to provide a wider overview.

ORERO, Pilar, Mario MONTAGUD, Jordi MATA, Enric TORRES & Anna MATAMALA (2020) “Audio subtitles or spoken subtitles/captions: an ecological media accessibility service.” En: Mabel RICHART-MARSET & Francesca CALAMITA (eds.) 2020. Traducción y Accesibilidad en los medios de comunicación: de la teoría a la práctica. MonTI 12

SCENARIOS FOR AUDIO SUBTITLES:

Relevant situations in which audio subtitles become useful are:

a)Within news programmes, some interventions are in the original language with subtitles. Unless you can read subtitles, you do not understand what they are saying. An example could be Theresa May explaining the Brexit agreement in Catalan TV.

b)Watching the news with subtitles over catch up TV or in start over mode.

c)Watching different language/same language content in a smartphone where subtitles are too small.

d) Watching different language/same language audiovisual content in a public screen too far away. In this case, subtitles are not legible. Similar examples are live situations, when for example going to a public conference, or in a museum.

e)Watching a partially subtitled film, that is a film in its original or dubbed version with some subtitled dialogues, as another language is spoken. This may be especially relevant in multilingual productions.

WORKFLOWS:

Another issue to be taken into consideration regarding technology, beyond TTS or OCR, is the place where the synthetic audio is generated. This may be at the broadcaster (or server, in case of VoD platforms) or at the client side. In general, two solutions can be applied to all previously described cases:

1. Audio is generated at the broadcast/server side. In such a case, the audio can be sent via a broadcast (multiplexed) audio channel (e.g., audio description channel) or via an additional audio channel (e.g., by using a parallel broadband connection, taking advantage of the Hybrid Broadcast Broadband TV (HbbTV) standard (HbbTV standard). Apart from the audio generation and delivery processes, the mix can be also performed at the origin/server or receiver/client sides of the chain.

2. Audio is generated at the client side. In such a case, the audio can be generated from:
   • OCR of burned-in subtitles.
   • Streamed text with subtitles (i.e., separate subtitle tracks).
   • File with audio segments.

The workflow for all these cases may be any of the following.

WORKFLOW A. TTS of existing subtitle files
This workflow is the most commonly used, and it takes advantage of existing subtitle text files to generate the associated audio signal that will be read aloud on the end user device (either on main or companion devices). The TTS conversion can be done at the broadcast or at the client side. An example of the former situation is the YLE solution described in the next section.

WORKFLOW B. OCR for burnt in subtitles
In this case, the use of OCR technology serves to transform burnt in text into a text file that can be read aloud by a TTS engine, like in Workflow A. An example of this is the NPO solution described in the next section.

WORKFLOW C. Use of screen readers
Screen reader features are e.g. commonly available in smartphones, and can be applied to text on screen or subtitle files.
However, to our knowledge, this solution has not been developed for AST yet.

WORKFLOW D. Use of Timed Text Markup Language 2 (TTML2)
Timed Text Markup Language 2 (TTML2) is a new content type for timed text media which includes features to embed audio audio resources, with the associated time codes and metadata. TTML2 file format can have a big potential for both AD and AST.

WORKFLOW E. Use of HbbTV standard
Features from HbbTV standard can be used to deliver either extra audio tracks for TTS generation or already generated TTS at the broadcast side to the targeted clients (either main or companion devices).

WORFLOW F. Automatic translation plus TTS
It consists of performing an automatic translation of text subtitles and then generating the AST via TTS.
The appropriateness of combining the above described workflows, like D and E, or F with other ones, is also worth to explore.
In addition, it should be remarked that the above workflows targeted at the generation of AST must not replace existing workflows for AD, but must complement, and inter-operate with them.

[shawna-slh]

@eoncins - I have been processing all this information the last 3 weeks as I have been working on our resource.

I find Audio subtitles/Spoken subtitles interesting and challenging!

I just reviewed the Scenario/Use Case Examples and Tasks for this resource.

I'm now thinking that Audio subtitles/Spoken subtitles is probably out-of-scope for this resource -- because it focuses on practical guidance on current practices.

Possibly we might want to mention Audio subtitles/Spoken subtitles very briefly on the Planning Accessible Audio and Video Media page -- although without examples and additional information to point to, maybe it's still too early to include in this type of resource?

I look forward to your input on the approach for this resource.

[eoncins]

You are right in saying that Audio subtitles/spoken subtitles might be out of scope. To me it sounds reasonable to mention them in the Planning Accessible Audio and Video Media page<https://wai-media-guide.netlify.com/design-develop/media/planning/> (Sorry I just find out a typo in this page "Media Palyers, Skills and Tools<https://wai-media-guide.netlify.com/design-develop/media/player/#skills-and-tools>") Which type of example do you need? A description of a UC or a video example?

[shawna-slh]

Still thinking about a very simple way to mention this.

A try is at: https://wai-media-guide.netlify.com/design-develop/media/planning/#other-languages

Thoughts?

[shawna-slh]

@eoncins ping :-)

[eoncins]

I think that it short description is ok for now! I really think that spoken subtitles / audio subtitles will be developed in the coming years. Right now there is a problem with the media players (blind people can not turn caption/subtitles into audio) Best, Estel·la Oncins Noguer Post Doctoral Research Fellow TransMedia Catalonia Edifici MRA 126 - Campus UAB 08193 Bellaterra Barcelona T. +34 610 655 149 orcid.org/0000-0002-0291-3036<https://orcid.org/0000-0002-0291-3036><https://orcid.org/0000-0002-0291-3036> http://grupsderecerca.uab.cat/transmedia<http://grupsderecerca.uab.cat/transmedia/> <http://grupsderecerca.uab.cat/transmedia/> <http://grupsderecerca.uab.cat/transmedia/> <http://grupsderecerca.uab.cat/transmedia/><http://grupsderecerca.uab.cat/transmedia/>

…

________________________________ De: shawn_slh <[email protected]> Enviat el: divendres, 26 de juliol de 2019 17:57:51 Per a: w3c/wai-media-guide <[email protected]> A/c: Estel·la Oncins Noguer <[email protected]>; Mention <[email protected]> Tema: Re: [w3c/wai-media-guide] Other Language Audio (#11) @eoncins<https://github.com/eoncins> ping :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#11?email_source=notifications&email_token=AMGTUEHKJRHZWRW6UA72G7LQBMNH7A5CNFSM4HR4ZGG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25ANAA#issuecomment-515507840>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMGTUEA2KIGOMMWLGRZQOULQBMNH7ANCNFSM4HR4ZGGQ>.