Source maps

[Rich-Harris]

Going to open this issue before someone else does - one-to-one file transformers need to be able to support source maps somehow.

Not sure what that looks like, but it'd be a shame if plugin authors were to forgo the benefits of file transformers (i.e. using directory transformers instead, so they could read and write source maps) when doing file -> file-plus-sourcemap transforms (or file-plus-sourcemap -> file-plus-sourcemap).

[lluchs]

Plugins using one-to-one transforms could support source maps today by inlining them with data URLs. Here's an article with sample code at the bottom.

[Rich-Harris]

@lluchs that's such a great idea! And that article is really helpful, thanks.

So I wonder if it's be possible to, as a convention, always keep source maps inline, but then maybe have something like this for production:

if ( gobble.env() === 'production' ) {
  // find sourcemap comments, and create separate .map files
  node = node.extractSourceMaps();
}

Going to have to do some experimentin'.

[rstacruz]

When you get there, this might come in handy: https://www.npmjs.org/package/exorcist

[Rich-Harris]

Yeah, I have a tab open with that link! I was doing a load of research into sourcemaps over the weekend. I came to a realisation - none of this stuff matters if you can't solve the multi-level sourcemap problem (e.g. CoffeeScript -> JavaScript -> concatenated -> minified... even if each step emits a sourcemap, your devtools will only map back to the penultimate step), and sourcemaps are complicated enough that expecting people to start writing compilers that both ingest and generate sourcemaps is a total non-starter.

So I started wondering if there's a simpler way. Here's what I came up with - sorcery.js. It's an early proof-of-concept but I think it has legs. The idea is that each transformation generates its own sourcemap (which can be inlined, using the technique in the article @lluchs mentions, or in a separate foo.js.map file) in blissful ignorance of what came before, and at the end of that process you generate a source map that describes the entire transformation from origin to output. (I tried to find a pre-existing package that did this, but came up empty.)

If it works (and early signs are encouraging), you'd be able to do this:

node = gobble( 'src/coffee' )
  .transform( 'coffee' )
  .transform( 'concat' )
  .transform( 'uglify' )
  .transform( 'sorcery' );

The final step would (say) read app.min.js, find its sourcemap app.min.js.map, analyse it, and create a new app.min.js.map sourcemap that traces mappings all the way back to the original coffeescript.

The nice thing about this is that you don't need to complicate the API by allowing one-to-one transforms to also potentially be two-to-two - the most complex case is one-to-two, because no transformation needs to care about what came before it. So you could still do

return compiledString;

but maybe you could also do

return {
  code: compiledString,
  map: sourceMap // gobble would need to fill in the `sources` property of the sourcemap
};

And the process of resolving sourcemap chains fails gracefully - if a file doesn't have a sourcemap, it's assumed to be an original source.

[Rich-Harris]

As of 0.6.0, file transformers have experimental support for source maps, using the API described above - they can either return the code, or an object with a code property and an optional map property. map must be a valid version 3 sourcemap, or JSON that represents one.

The sources and sourcesContent properties will be filled in by Gobble, since the compiler isn't expected to know that information (though the value of this inside the plugin is an object with src, dest and filename properties).

Will sit with this for a bit longer and update docs etc if it all pans out.

[insidewhy]

I was really interested in plumber because it transforms source maps as it processes the pipeline. If gobble can nail this then it'd be so great, plumber seems kinda buggy at the moment :(

[OliverJAsh]

@nuisanceofcats Please do keep filing issues on Plumber repositories. We will get back to you!

[Rich-Harris]

@nuisanceofcats I hear you. I'd love for sourcemaps to 'just work', and I think all the pieces are in place - it's just a matter of battle-testing, and ensuring that plugins can create sourcemaps. I've just updated the wiki to document the API for returning sourcemaps from file transformers.

So I'll close this issue, but this is still a good place to discuss any sourcemap-related stuff until a new issue arises.

FYI, the latest version of gobble-cli will attempt to use sourcemaps for debugging info. (The latest version of gobble needs to be installed locally for it to work.) The screenshot below has an example - some ES6 modules got concatenated with gobble-esperanto-bundle, which generated a sourcemap, but the subsequent transformation with gobble-es6-transpiler failed because es6-transpiler throws an error if it encounters a global variable it doesn't know about.

On encountering the error, Gobble will:

• try and determine where the error occurred - if the error has file, line, column properties it will use those, or it will look for a loc property (which e.g. acorn includes), or it will try to find information contained in the error message itself, as is the case here ('line 185'). (These heuristics are fallible, but can easily be improved)
• if it succeeds, it will print the code surrounding the error, and highlight it
• it then uses sorcery to attempt to find the origin of the error. In this example, it correctly locates the original line of code that contained the error, making it very easy to fix.

There are a lot of moving parts, so it's all done very much on a 'best effort' basis, but when it works it makes life much easier. If anyone out there tries this out, I'd be very interested to hear how you get on.