Switch from Audio to AudioContext to play music

[martpie]

last edit: november 30th 2016 (specs added)

The Audio media element works well, but we need the power of AudioContext to add effects, play without interruptions, easy cross fade ect.

That's a huge work.

Tasks

Those are hints on how to accomplish this, these may change depending of our needs. More over, they more explain how "a player work" than "how it should be done"

• Re-implement the lib/player.js Interface (already existing)
  • play()
  • pause()
  • stop()
  • mute()
  • unmute()
  • setAudioVolume()
  • setAudioPlaybackRate()
  • Unload track and load another one (ex-setAudioSrc())
  • Seek to a certain time (ex-setAudioCurrentTime())
  • + the few utils of player.js
• Emits corresponding events (may not be needed anymore)
  • play: new track is being played
  • pause: the whole thing is paused
  • ended: the track ended
  • timeupdate: same than HtmlMediaElement
  • error: means what it means

Hints

• lib/player.js can use Audio tags as they can be used as sources for AudioContext

Help

• Example play/pause with AudioContext http://jsbin.com/lenapigo/1/edit?html,css,js,output

This issue is required to eventually do #127 #31 #114

[YurySolovyov]

@KeitIG if we're going to use AudioContext, or OfflineAudioContext, we might want to come up with some kind of custom buffering strategy.

This becomes especially important for ffmpeg, where we can't just wait for the entire file to be converted.

Something like:

1. Start playing first 20 seconds
2. On 15 second start buffering next 20 seconds
3. Swap/Update buffers
4. On navigation/seeking start over at p.1

[YurySolovyov]

Reference: http://72lions.github.io/PlayingChunkedMP3-WebAudioAPI/

[YurySolovyov]

One more: https://developers.google.com/web/updates/2015/06/Media-Source-Extensions-for-Audio#introduction

[YurySolovyov]

@KeitIG how do you expect this to happen? If I'm going to do this, you're expecting the PR with specs and without regressions, right?

[martpie]

@YurySolovyov I've updated the specs.

"Basically", the things is to rewrite lib/player.js

[martpie]

A few more things:

• About the chunkload, this is not required for this issue, we'll see that later if we use ffmpeg
• I keep myself assignee for now because I want to understand that API, I just have to find enough time, but you can work on this too, we may end up with different implementations of the same thing, that could be interesting to compare our work
• This is an off-release issue: it's out when it's ready

[YurySolovyov]

@KeitIG
How about this shipping plan:

1. Use WebAudio
2. Use Chunked Loading
3. Use FFmpeg

Where we can do any number of releases between the steps.
This would allow us test it properly at each step.

[martpie]

This issue is just about AudioContext, for other things, issues already exist or create another ones.

[martpie]

Added events specs.

[httnn]

is it really necessary to get rid of HTML5 audio though? it can still be used with the web audio API using createMediaElementSource(): https://developers.google.com/web/updates/2012/02/HTML5-audio-and-the-Web-Audio-API-are-BFFs

it's so much easier to control playback with HTML5 audio than it is with the web audio API. however, if you somehow want to convert and stream tracks from ffmpeg you'd probably have to use web audio API only... unless you first convert the entire audio file temporarily to disk but it gets tricky.

[martpie]

That's what was planned.

When I say "drop HTML element", I'm speaking about the whole player system, not the things that play/decode music.

This may change in the future though if we decide to use ffmpeg to decode audio, and not <audio>

[YurySolovyov]

@bodyflex @KeitIG it seems like if we just do the simplest thing possible to replace <audio /> with AudioContext, it won't be all that hard. The question is: what next?

For ffmpeg it would be useful to employ chinked encoding to avoid converting the whole file at once.

For other stuff, I'm not sure what benefit we can get from WebAudio at this point, though I think it is still important.

[martpie]

I'm not sure what benefit we can get from WebAudio at this point

All the issues depending on this ? (equalizer, crossfade ...)

[OutsourcedGuru]

For what it's worth, AudioContext.createAnalyser() could be used to create equalizer-like effects as in these examples.

[s0]

I've effectively implemented this in https://github.com/s0/museeks/blob/synesthesia/src/ui/lib/player.ts, as part of the work towards #31 (see #31 (comment)).

Hopefully I'll soon have a PR open that will address this issue and #31.