raster mark

[mbostock]

This is an alternative to the pixel mark #1185; the raster mark takes a set of discrete {x, y, fill} samples and produces a corresponding raster grid (image) using putImageData. The canvas is then converted to a URL for use as an svg:image, which is positioned using {x1, y1, x2, y2} abstract coordinates likewise bound to the x and y scales.

[mbostock]

I’m not sure why CI is failing; presumably it’s a non-visible difference in the result of canvas.toDataURL. 😞

[Fil]

I'm not sure this mark works for me, because I don't see the use case (besides copying a raster with the same dimensions (?)), and I don't see how it fits with the general approach of Plot's scales etc. To help clarify what we want to cover, I've started a list of use cases in this notebook: https://observablehq.com/@observablehq/pixel-or-imagedata--dev

Two unrelated remarks:

• Re: opacity, I think we might have to compose it with the A of rgbA color channel?
• I still want to pass the facet information (the actual value of fx, fy) somewhere in the call to mark.render (render API #501); this would help make this mark facet-dependent. I'm using "facet-dependent" and not "faceted", in the sense that they would not have a partition index, but could act differently depending on the facet they're drawing.

[mbostock]

Okay, here’s my third attempt… I hope you like it better. 🙏 It feels very convenient for the continuous f(x, y) case, and supports all (continuous) scale types. And it’s still convenient for the simple case of rendering an existing raster grid (e.g., volcano) with linear scales. I would like to implement a contour mark with the same type of x/y/value specification… but I’m not sure how much work I’ll do over Christmas.

[Fil]

would this be easier if the signature for fill was f({x,y}, i)?

[mbostock]

I don’t think this test needs to be “easy”—the easy form is shown above, and this test isn’t supposed to be representative of recommended usage. The point here is to test the explicit form where fill is a function of data.

[Fil]

I really like where this is going. As mentioned in the comments there's more we could do (in terms of performance, compatibility with opacity, faceting, projections…) but they could be done in follow-ups.

Note that Safari currently doesn't support image-rendering: pixelated on svg images which makes it a bit awkward for us. Should we expect Webkit to fix itself, or look for a workaround (foreignObject+img)?

I'm adding a bit of documentation. I must say that I'm still struggling to understand/explain how the x and y channels work in conjunction with width, x1 and x2 (?). I've been trying as an exercise to create the image of the volcano with x, y, fill as channels, but not very successfully (seems like I need to specify all of x1, x2, width…?).

[Fil]

Suggested change

	const {r, g, b} = rgb(F[i]);
	// TODO: memoize for performance? We'll usually have a maximum of 128 different shades, but 100x more pixels.
	const {r, g, b} = rgb(F[i]);

[mbostock]

I think it would be better to do the optimization earlier when the scale is applied, like how Yuri does here:

// Convenience function to create a cached color interpolator that
// returns cached rgb objects, avoiding color string parsing.
cacheInterpolator = (interpolator, n = 250) =>
  d3.scaleQuantize(d3.quantize(pc => d3.rgb(interpolator(pc)), n))

https://observablehq.com/@twitter/density-plot@4159

Like maybe there’s some hint that the mark wants {r, g, b} objects instead of color strings and can instruct Plot to materialize those efficiently when applying the color scale. But in any case I think we should do optimizations as follow-up, as you say.

[mbostock]

I've been trying as an exercise to create the image of the volcano with x, y, fill as channels, but not very successfully (seems like I need to specify all of x1, x2, width…?).

Yes, you must specify all those options. Like this:

Plot.plot({
  marks: [
    Plot.raster(volcano.values, {
      width: volcano.width,
      height: volcano.height,
      x1: 0,
      y1: 0,
      x2: volcano.width,
      y2: volcano.height,
      x: (_, i) => (i % volcano.width) + 0.5,
      y: (_, i) => Math.floor(i / volcano.width) + 0.5,
      fill: (d) => d
    }),
    Plot.frame()
  ]
})

Alternatively if you want the samples at integer locations, you have to offset the bounds by 0.5:

Plot.plot({
  marks: [
    Plot.raster(volcano.values, {
      width: volcano.width,
      height: volcano.height,
      x1: -0.5,
      y1: -0.5,
      x2: volcano.width - 0.5,
      y2: volcano.height - 0.5,
      x: (_, i) => (i % volcano.width),
      y: (_, i) => Math.floor(i / volcano.width),
      fill: (d) => d
    }),
    Plot.frame()
  ]
})

The x and y channels specify the centroids of the pixels, so you need x1, y1, x2, and y2 to specify the extent of the pixel grid; otherwise the raster mark cannot know how wide a pixel is in abstract coordinates. And you need to specify width and height because this is the size of the raster grid (in grid coordinates, i.e., pixel indexes) which might be a different resolution than the samples.

[mbostock]

If you would prefer that the samples are taken at integer locations by default (rather than offset by 0.5) then we could apply this patch:

% git diff -p
diff --git a/src/marks/raster.js b/src/marks/raster.js
index 71a0af1c..52864d10 100644
--- a/src/marks/raster.js
+++ b/src/marks/raster.js
@@ -19,10 +19,10 @@ export class Raster extends Mark {
       // If X and Y are not given, we assume that F is a dense array of samples
       // covering the entire grid in row-major order. These defaults allow
       // further shorthand where x and y represent grid column and row index.
-      x1 = x == null ? 0 : undefined,
-      y1 = y == null ? 0 : undefined,
-      x2 = x == null ? width : undefined,
-      y2 = y == null ? height : undefined,
+      x1 = x == null ? -0.5 : undefined,
+      y1 = y == null ? -0.5 : undefined,
+      x2 = x == null ? width - 0.5 : undefined,
+      y2 = y == null ? height - 0.5 : undefined,
       imageRendering,
       pixelRatio = 1,
       fill,
@@ -156,7 +156,6 @@ function sampleFill({fill, fillOpacity, pixelRatio = 1, ...options} = {}) {
     if (h === undefined) h = Math.round(Math.abs(y2 - y1) / pixelRatio);
     const kx = (x2 - x1) / w;
     const ky = (y1 - y2) / h;
-    (x1 += kx / 2), (y2 += ky / 2);
     let F, FO;
     if (fill) {
       F = new Array(w * h);

Do you have a preference? I think it looks better to have the grid start and end on integer boundaries.

[Fil]

Integer boundaries look better, I agree.

My difficulty is in precisely articulating how the various options must be combined, or on the contrary are "not compatible". For example, if you try to replace width: volcano.width with a "finger in the air" pixelRatio in the example above, you will get banding (unless pixelRatio=580 / 87). To document this, we need to mention that there is rounding, and that the pixelRatio is imputed from the width of the frame divided by the width of the data.

[mbostock]

My difficulty is in precisely articulating how the various options must be combined, or on the contrary are "not compatible".

It’s best to think about this as multiple coordinate systems.

First there are a discrete set of samples in abstract coordinates x and y with fill or fillOpacity (or both). These correspond to the raster mark’s data. (The samples are imputed when fill or fillOpacity is specified as a function and data is null.) These samples are typically in an axis-aligned grid but not necessarily so; arbitrary sample positions will be more useful in the future if and when the raster mark supports different methods of interpolation.

Second there is a raster grid (a.k.a. canvas) with its own pixel coordinates. The aforementioned samples are mapped to a canvas that is width pixels by height pixels, with the origin [0, 0] in the top-left corner. The extent of the canvas in pixel coordinates [0, 0, width, height] corresponds to an abstract extent [x1, y1, x2, y2] in the same abstract coordinate system as the samples. (In some cases x1 and x2 can be flipped, and likewise y1 and y2.) During rendering, the raster mark assigns (bins using Math.floor) each sample in x and y to a pixel.

Lastly there are screen coordinates (really Plot frame coordinates, in the range of the x and y scales). These are needed to place the svg:image in the correct position within the Plot frame.

If you specify an incorrect width in the volcano example, then you expect to see banding because you no longer have exactly one sample per pixel in the raster grid. For example, here three columns are missing samples:

Plot.plot({
  marks: [
    Plot.raster(volcano.values, {
      width: volcano.width + 3,
      height: volcano.height,
      x1: 0,
      y1: 0,
      x2: volcano.width,
      y2: volcano.height,
      x: (_, i) => (i % volcano.width) + 0.5,
      y: (_, i) => Math.floor(i / volcano.width) + 0.5,
      fill: (d) => d
    }),
    Plot.frame()
  ]
})

(They should really be transparent though… I think that’s a regression I introduced when adding support for fillOpacity. Fixed.) If and when the raster mark supports better interpolation—something smarter than just binning the samples into rectangular pixels—then we could e.g. fill those gaps with the closest sample, or a blend of nearby samples.

[mbostock]

Last step here is to figure out faceting, or at least document that it doesn’t work yet.

[mbostock]

@Fil The latest changes are in 1e22d3d...a745cc3; documentation aside, this should be ready to go! 🚢

[Fil]

I'm tempted to say "ship", but this morning I tried to plug in https://observablehq.com/@jobleonard/pseudo-blue-noise instead of randomLcg(42), and the improvement in quality is really remarkable:

walk-on-spheres (before, after)

barycentric (before, after)

contours with random walk (before, after)

[mbostock]

The pseudo blue noise looks nice but I would like to focus on documenting and releasing this and returning to the axis mark. I’d be willing to expose the built-in spatial interpolation methods as functions so that you can pass in a custom RNG.

[mbostock]

The latest two commits expose the built-in spatial interpolation methods

• Plot.interpolateNone
• Plot.interpolateNearest
• Plot.interpolatorBarycentric({random})
• Plot.interpolatorRandomWalk({random, minDistance, maxSteps})

So you can now provide your own RNG that uses pseudo blue noise e.g.