waveform seek bars

[httnn]

I've been thinking about building a music app with Electron for a while since I'm really missing a player that has waveform seek bars on MacOS (like the ultimate music player foobar2000 has on Windows). I just found museeks and really like the clarity of the code and the technology choices (esp. React, Redux, ES6) so contributing looks like a realistic alternative on by behalf.

a possible implementation would be to draw and cache waveforms on demand. with the snippet below, generating a waveform for a 3-4 minute track would take about 1-2 seconds. the images should be cached somewhere and only re-drawn if the content of the audio has changed (cached waveform images could be referenced to with hashes of the audio content). waveform seek bars should be optional and only available if a sufficient version of ffmpeg is detected on a users system.

I have experimented with different methods to draw waveforms with Node a lot, and have come to the conclusion that ffmpeg with node-fluent-ffmpeg is currently the best way. getting a waveform as a PNG stream is as easy as:

ffmpeg(audioInputStream)
  .format('image2pipe')
  .outputOptions(['-frames:v 1', '-vcodec png'])
  .complexFilter(
    'showwavespic=s=1000x200,format=rgba,colorkey=black,colorchannelmixer=rr=0:gg=0:bb=0'
  )
  .pipe();

would you be interested in this kind of functionality or should I start developing a completely separate version?

[YurySolovyov]

I'd say if we do this, let's do it via Web Audio API

[httnn]

actually yeah, good point. wavesurfer.js is one implementation that uses the Web Audio API. seems to be pretty quick as well!

[YurySolovyov]

This is probably still blocked by #128

[dkniffin]

This would be another feature that would make a great plugin (#260), because not everyone is going to want the waveform seekbar. I'm not against putting it directly in the core code for now though, as long as it's hidden away by a button, or disable-able via a setting.

[martpie]

Hello @bodyflex,

Thanks for your interest, Museeks is young and all contributions/ideas are welcome.

About the waveform bars, I kind of like it. It's useful for long songs, with some design, it could look nice and still be discreet enough (something like a light color bars in background of PlayingBar...).

I'm more a fan of the soundcloud than foobar's one for example.

There are some problems to solve first:

• Do we do something like covers, where waveforms are generated everytime we play a track? Or can we store them somewhere ?
• Left/Right or Mono ? For design reasons, I personally prefer a mono design
• ...

Adding the discussion label, because the idea is great, but not doable right now.

===

Refs

Foobar

Soundcloud

[YurySolovyov]

I'd say if possible let's do it every time, but in a separate thread if possible, this way we can start playing track right away, and fade in waveform when it is ready.

Might be useful: https://developer.mozilla.org/en-US/docs/Web/API/OfflineAudioContext

[httnn]

Do we do something like covers, where waveforms are generated everytime we play a track? Or can we store them somewhere ?

I think it would be good to cache them somewhere temporarily. if generation takes even one second it's much nicer to be able to show previously-generated waveforms instantly.

Left/Right or Mono ? For design reasons, I personally prefer a mono design

I'd definitely say mono. in waveforms images, both channels look virtually the same in the vast majority of music, so one summed channel should really be enough.

---

personally I dislike SoundCloud's (new) waveform, it is pretty much useless if you want to see what is happening in a song. but I agree that foobar2000 doesn't have the most polished design either. some kind of smoothed waveform but with a higher resolution than SoundCloud's would probably look best. would need to experiment with drawing waveforms a bit to find the best alternative though!

EDIT:
one metric for the perceived loudness of music is the RMS (root mean square) level. I wonder if, instead of the peak level, it could be used with some kind of sliding calculation window to draw waveforms that showed the "true" loudness of each part of a song in a more user-friendly and useful way.

[martpie]

I'd say if possible let's do it every time, but in a separate thread if possible, this way we can start playing track right away, and fade in waveform when it is ready.

I was thinking about that recently. There's a lot of things we could delegate to the main process. Even if we can only pass strings between Renderer and Main.

I think it would be good to cache them somewhere temporarily

We could also imagine to cache the generated wavebars somewhere in a JSON file in ~/.config.

---

About all the rest (RMS...), I have no particular opinion at the moment.

[YurySolovyov]

@KeitIG problem with main process is that we can't access the DOM from there, so no API calls possible.

[httnn]

we can only pass strings between Renderer and Main

this might be a problem for the caching. as the web audio API would only be available in the browser (right?), the waveforms would have to be transferred to the backend as strings. I have no clue how performant backend/frontend communication is in Electron.

storing waveform data as JSON might also be an option, but I wonder how useful it would be since the images would anyway be pretty static so it might result in unnecessary rendering overhead.

[martpie]

it would be since the images would anyway be pretty static

It won't be static and we won't generate images for wavebars. It'll be generated SVG, pure HTML or anything vectorial generated from data.

Only quality work here :|

[httnn]

I'm all about quality work, trust me! :) I just don't necessarily think waveforms are best represented vectorially. however, this completely depends on what kind of waveform is desired. I'm going to start experimenting with drawing some waveforms with the web audio API.

[YurySolovyov]

@KeitIG any ideas how to integrate them in current design?

AIMP kinda has what looks acceptable for me:

Might be a good reason to try out full-width progress bar.

[martpie]

@YurySolovyov I have some ideas indeed, but it will imply some PlayingBar redesign.

[httnn]

here's the result of my initial experimentation:
https://gist.github.com/bodyflex/e4f6c9ec0fdea9450fd9303dd088b96d

select an audio file with the form input to see the waveform and look into the console for some simple statistics. it uses the RMS method I mentioned but with a very small window, so not really that useful yet.

next I will try to draw the waveform with SVG and with some more smoothing.

[YurySolovyov]

@bodyflex can you console.time + timeEnd this thing for a particular file?

[httnn]

it already measures the durations (console.time wasn't flexible enough for my needs). an mp3 that is a bit short of four minutes long takes ~0.5s to calculate the values and render to canvas. a 7.5 minute mp3 takes ~1.5s.

EDIT: can now also render to SVG

[martpie]

With an animation and eventually a cache system, it should be fine

[httnn]

what kind of animation do you mean specifically? highlighting the current playback position in the SVG waveform is pretty trivial using masks at least.

[martpie]

I did not think about that specifically. With some data, I could work on the design and the flux implementation. But roughly, we could imagine a fade animation when the player enter in "play" mode, and then update the bars with some transition when a track changes (an example, even if we won't have the same scale/graph https://bl.ocks.org/RandomEtc/cff3610e7dd47bef2d01)

[martpie]

One question is "will it replace the current progress bar or will it just be a background information ?". Just things I have to play with

[httnn]

I'd say that if it's enabled it should replace the default seek bar. of course then the progress would need to be displayed on the waveform.

[YurySolovyov]

I'd say that if it's enabled it should replace the default seek bar.

@bodyflex think about how usable that would be

[httnn]

@YurySolovyov what usability problems do you see? I'd argue that it's even more usable since the clickable area is larger! :)

[YurySolovyov]

@bodyflex I'd say it might look unfamiliar to users. We should try to integrate it, no replace

[httnn]

I don't think that's a problem if it's an optional feature! once a user chooses to enable waveform seek bars in the settings they surely anticipate that something will change in the UI and the change shouldn't confuse them. but I agree that by default the "classic" seek bar should be used.

[YurySolovyov]

@bodyflex we could have modes like default, overlay, replace for it, so users can choose.

[martpie]

About options and all that stuff, please remember that Museeks is not meant to be modular, for now. It's meant to provide experience we think is the best for the user.

We're talking about a music player, not something you will spend 8 hours a days on.

edit: what I a, saying is we provide options for UX stuff (dark theme is UX for me), but we define the UI for the user.

[martpie]

here's the result of my initial experimentation:

great job here

[dkniffin]

not meant to be modular, for now

Although, I'd really like to discuss this in #260.

We're talking about a music player, not something you will spend 8 hours a days on.

But I do spend 8 hours a day (or more) listening to music

[httnn]

updated the gist now. it plays music with an <audio> element and visualises the progress on the waveform. you can also skip around the track. @KeitIG at this resolution, SVG seems like a suitable solution!

[YurySolovyov]

@bodyflex can you post a screenshot? Just curious

[httnn]

just so you know: you only need to save that gist in an HTML file and open it (in Chrome). it should work out of the box!

[martpie]

I'll try to test it this week-end :)

[martpie]

Maybe for 0.9 ? Just adding it to the milestone just for an overview of what we could add for this release.

[YurySolovyov]

https://vx-demo.now.sh/gallery