Streamtubes [WIP]

[monfera]

Goals

Streamtubes is a new trace type that gives plotly.js the ability to generate streamtubes. Inspirations:

• science, e.g. connectomics
• industry, e.g. energy
• general tensor-based visualizations, for flow or other things in 3D
• space filling or stick molecular visualization, molecular structure or molecular marking
• progress toward more general use in engineering or related presentation
• art

Direct motivations included:

• ability to do 3D scatterplots with curved lines, gradient colors and varying line width
• avoid the Z-fighting issue that's present in scatter3d when point markers are of a different color than the lines
• provide nice, realistic connections among markers and connecting lines
• benefit from our revamped lighting and reflection support for aesthetic or realistic rendering

Immediate goals: a streamtubes trace type, which

• has an API as close to scatter3d as possible
• generates realistic looking 3D markers (in this round, icospheres) and connecting lines (in this round, of spherical cross-section)
• each marker should be able to have a different color and size
• connecting lines should support varying radius, varying color (gradient or discrete)
• use splines for smoothness of a fine trace, or for curviness with low sample count, and for smoothly varying radius and color

Nice to have goals:

• provide text labeling of markers
• support the plane projection as in scatter3d
• support a color legent as in scatter3d

Previous steps

Development of #617 (now merged; example1 example2 example3 and see also the test cases in the changeset)
Receiving requirements on coloring, curvature and aesthetics (i.e. no Z-fighting)

The mentioned Z-fighting is an artifact that arises due to the pseudo-3D nature of scatter3d which uses much fewer polygons, but has inherent tradeoffs, see the arbitrary connections and glitches at the markers. Also observe the (sometimes desired) flat, non-3D like presentation and the sometimes muddy color gradients:

Solution

We run a test atop of mesh3d and this showed the basic feasibility (realistic lighting, gradient color, curves) and led to management request for streamlines, as opposed to just scatter3d with curves:

Splines

Colors, radii need to be interpolated between points. Also, even high-density trace lines suffer from an angular look, so interpolation is needed to give a smooth surface. Moreover, it's desirable to give the option to smoothen the curve even if there are few points supported. Therefore a spline implementation had to be chosen (though more could be added in the future). Due to its favorable properties, Centripetal Catmull-Rom was chosen for its relative simplicity, computational speed, desirability for data visualization (respects monotonicity i.e. doesn't overshoot points; goes across all points; doesn't form weird loops), intuitive use (no need for control points other than user-supported points), and looks good. Also supported by the now-standard [http://bl.ocks.org/mbostock/1705868](D3 4.0) library and may have eventual use in animation from which it stems.

It wasn't possible to use the preexisting Catmull-Rom spline in plotly.js or d3.js because they use Canvas and/or SVG and for efficiency of rendering, map Catmull-Rom to native splines (cubic, quadratic are native spanes). Also, we needed it to work in 3D, or more if we count the radius and color interpolations. (The current function is WIP and very un-DRY, but avoids memory allocation; to be refined later).

Geometry

Each aspect of geometry is discussed in separation. From a bird's eye, we have markers and connections.

Marker geometry

Markers are currently icospheres, arrived at via increasing the LoD (level of detail) on an initial seed icosahedron. They are costly in that lots of polygons are being used for smooth appearance (compactness todo below). It's mitigated by the fact that rendering speed is usually limited by fill speed rather than geometry size. Also, the examples worked on so far posed no speed issue on rotation/zooming etc. even with a very high number of markers:

Here's an intentionally low-poly icosphere:

Other markers (umm.. damaged icospheres) were quickly tried but not yet added:

The icosphere points are shared (cached as the increase in LoD generates shared vertices).

Because icospheres are made up of a lot of polygons (involving quite a few calculations), but they don't need rotation, the solution defines a unit icosphere and marker rendering basically pastes this in with scaling and translation transforms only (trivial math).

Connection geometry - ring element

The simplest connection line would be a long, thin cylinder between two marker centers. But we need to vary the radius (in the future: cross-section too), and it's built out of polygons, so the solution uses the (generally non-right) frustum as the basic ring element. Lots of these glued to one another form the connecting line. In general, the frustum is not right, i.e. a curve makes it necessary that the pyramid cutting planes are at some infinitesimally small angle, to follow the curve. The cutting planes themselves are not rendered, only the sides, so there's an opening at the end of the 'tubes'.

Here's an example of low-poly, very long rings, that demonstrate not only the triangles that make up the frustum sides, but also how the gradient coloring works not only as a different color from ring to ring, but also, there's blending inside a ring (which reduces somewhat the number of polygons necessary to create for the same gradient smoothness). This tessellation in general should be invisible due to the infinitesimally short rings and small color changes.

Connection geometry - fusing the adjacent rings to form a tube

In order to not waste (duplicate) vertices between neighboring ring elements, the second cutting plane polygon vertices are being used as the first cutting plane poligon vertices in the next ring, i.e. it will form a joined mesh. This requires that attention is paid to the polygon point order, because, if the points are not in a matching order, then the ring will be degenerate (see the next point).

Gymbal lock

Each connection line ring is transformed in space (rotated and translated to its place). The current rotation approach is rather basic; it uses a distinguished normal vector around which the rotation takes place. This axis can be arbitrary but there is the degenerate case when the rotation vector is collinear to an arbitrarily picked normal vector. Therefore currently we loop through unit normal vectors (actually, x, y, z unit vectors for simplicity) and pick the first one that doesn't yield a degenerate case.

This unfortunately means that as the curvature evolves, the rings in a section of the curve are rotated as per an x normal vector, then another section by a y normal vector, so the above mentioned criterion for matching polygon points is violated, resulting in one degenerate ring. With fine enough resolution, it is either unnoticeable or shows up as a thin line (band) on the otherwise smooth surface. But it means that the connection line surface uses more polygons than visually needed.

The solution will be a conversion to quaternion based rotation which isn't vulnerable to the gymbal lock effect.

Mesh geometry in general

The generated mesh geometry internally is currently a tuple of {[{x, y, z}], [{i, j, k}] represented as long numerical vectors where {x, y, z} are vertex coordinates and {i, j, k} are faces. i, j, k are the three vertices of a triangle. There's the straightforward optimization possibility that we generate a mesh rather than a set of triangles as currently, because it's easy to identify the adjacent polygons. In this case, each new vertex index is implied to form a triangle with the previous two vertex indices.

There are numerous other optimization possibilities for geometry, including adaptively changing the LoD on zooming in parallel with culling, or doing some of the work not in geometry but shaders, but this initial PR reuses much of existing code.

Also, since the icospheres and tubes are meant to be closed hulls, gl.CULL_FACE could be enabled for them.

Useful features from scatter3d: labeling and projection

There's some visual disconnect between the realistic shapes and the flat-looking wall projections, and the label placement is just PoC, but it's a start:

Some examples

As the PR is right now, it generates this example (labels, color legend also present):

These below examples were generated during making a PoC, for example, straight linear interpolation isn't added yet, just the Catmull-Rom; closed loops aren't implemented (isn't hard to do).

[monfera]

Notes on being a standalone streamtubes trace vs. integrating it with scatter3d

streamtubes started as PoC code atop of mesh3d but got transformed into a separate trace type that inherited some of its frontend from scatter3d and the back-end from the PoC, therefore mesh3d.

A) Pros for folding streamtubes into scatter3d

1. significant amount of shared code, especially 3D projection and similarity of inputs (attributes)
2. ability to handle both 3D streamtubes and 2.5D scatter connection lines within the same trace type
3. forces us to think deeply about unity of traces, e.g. whether surface3d and mesh3d needs to be a separate trace type, and what the challenges are if the meaning of a trace type is overloaded (more powerful API at the expense of API & coercion complexity)

B) Pros for a dedicated streamtubes trace type

1. common code can / should be factored out anyway -> separation of concerns
2. mixing 3D streamtubes and 2.5D scatter lines seems not terribly useful (what's the use case?)
3. mixing them will reintroduce Z-fighting issues (between 2.5D elements and also between 2.5D and 3D)
4. mixing them will result in confusing visuals - one is lighting dependent, the other isn't; different look & feel (2D-ish vs solid 3D); also, the scatter3d points, line widths grow and shrink relative to the domain coordinates with zooming, while streamtubes preserve constancy
5. the 3D streamtubes may actually be preferred for scatterplotting vs. the original scatter3d because it solves issues with it, but it would be more of an absorption of still missing, important scatter3d functions rather than an integration as it is (i.e. there may be no longer 2.5D lines); see also In certain cases, a scatter3d trace line disappears #691
6. the feature set only partly overlaps: scatter3d has no splines, and has constant line width for an entire trace, while streamtubes has no (ATM) line types such as dotted, dashed, or alternative marker types (only icosphere), and has no error bars or Delaunay surface; even seemingly little details differ a lot, e.g. the placement of the text labels (again, solvable the DRY way, time permitting)
7. work is not done and even basic projection needs to change because scatter3d anisotropic projection was OK without lighting, but it can result in distorted lighting for streamtubes, therefore per-dimension scaling factors will probably need to differ - and there may be other issues that need further specificity (again, the common parts of the code can be extracted out)
8. streamtubes and scatterplots are rather different visualizations even if there's overlap, and future user requirements may make it difficult to maintain either, if coupling between them is too high - as an example, consider that stream surfaces or tensor inputs, seed models would be a natural extension of streamtubes but they would be even more removed from the concept of scatter3d
9. if streamtubes and scatter3d are worth fusing, then by the same token, most plots could be fused together - in theory, everything can be visualized with a 3D mesh: mesh3d, surface3d, streamtubes, scatter3d, contours, and even 2D plots, just choose an orthogonal projection, imply z values of 0 look at the scene from {x:0, y:0, z: 1}
10. even with separate trace types, it might be possible to feature both of them in the same scene (haven't tried it yet)

In line with current time constraints, I feel there are benefits with completing streamtubes in a standalone way and at some point think about the potential for fusing plot types, not necessarily constrained to these two plots only. However it's perfectly possible to fuse streamtubes and scatter3d right now, but needs significant additional time for properly working out all the extra control attributes and defaults that would necessarily add complexity. Either way is doable, it's just quicker to release and iterate piecemeal. There's still a good amount of work with streamtubes as it is.

[etpinard]

good 👀

[monfera]

Recap of discussion on decisions taken (correct me on the plotly.js slack or in the absence of that, as a comment below, if there's a misunderstanding and I'll update this comment):

1. streamtubes gets renamed to streamtube and it'll be a trace type of its own
2. generated geometry will be spherical/cylindrical (rather than flattened) no matter what the x, y, z domain scales are (if I bump into some obstacle, I might need to enforce identical x, y, z scaling in the initial merge candidate)
3. the plan is to first bring the PR to a merge candidate (code DRYing etc.), then add the as yet missing streamtube features such as ellipsoid streamtube cross-section, with two radii and rotation (think of fusilli)
4. also, things like scaling in line with the x, y, z scale proportions, or regenerating the geometry on zooming, or increasing LoD (either that of the mesh, or via resolving a large streamtube into a bundle of finer tubes upon zooming), or defining tube/marker dimensions in screen units (rather than domain units) will be for future consideration
5. tube/marker dimensions will be in terms of domain data (i.e. corresponding to the scale the user understands based on the axis tick cadende)
6. if the x, y, z scales can be of non-uniform cadence (see point 2) then tube/marker dimensions will be in relation to a specific dimension of the user's picking (though, as mentioned in point 2, perhaps the initial version will enforce uniform scales and in this case there won't be a need for this setting)

[etpinard]

the plan is to first bring the PR to a merge candidate (code DRYing etc.), then add the as yet missing streamtube features such as ellipsoid streamtube cross-section, with two radii and rotation

Excellent.

[etpinard]

From @monfera

I tried to solve the issue with the non-cube shaped scene [...] but it didn't quite yield. Part of the reason is that first, the geometry is set up inside the streamtube trace, ​_then_​ the scene decides on the aspect ratio (checks if the largest dimension is >4x the smallest), and while this depends on the streamtube geometry, the values are ​_not_​ known ​_inside_​ streamtube, i.e. the geometry can't be compensated for the eventual aspect ratio (unless I were to duplicate above mentioned aspect ratio logic). So the current rendering pipeline isn't amenable to my first approach. What I did instead: if the user explicitly ​_specifies_​ an aspect ratio, then that information is taken into account, so a non-cubic streamtube can be created. Lighting is adjusted too, but due to the above reasons, the spheres, tubes will not be circular (e.g. tubes in 10x10x1 become flattened).

[etpinard]

@monfera what are the units here?

[monfera]

@etpinard it should be in terms of the data domain, e.g. in a 100 x 100 x 100 cube a radius of 1 should mean 1/100th of the box edge

[etpinard]

Great. This is perfectly reasonable mode (how should we call it? 'axis' maybe?) In future development we could add other modes e.g. 'constant' (which would make the streamtubes appear cylindrical regardless of the scene aspectratio) and 'data' (which would correspond to a data array).

By the way, @monfera would you be ok with renaming connectionradius -> size or just radius ?

[etpinard]

To clarify the current behaviour:

Starting from:

then Plotly.relayout(gd, 'scene.zaxis.range', [-10, 10]); yields:

[monfera]

Yes. In this case, the light reflection can be more of a distraction than a positive. So with this type of use, it's good to avoid using a specular light component.

[etpinard]

@monfera I've just noticed that the hover labels aren't being shown for every data point.

Is that a known issue?
Is it a side-effect of your recent rebase?
Is it something you want me to look at?

[monfera]

@etpinard I don't think hover points show up and streamtubes in general are more of a continuous thing (e.g. wind flow) where points mostly serve as guides for where it curves and how radius, color evolve, rather than something like with scatterplots, so for this reason I haven't prioritized it.

Currently, on hover, the crosshair lines wrap the mesh rather than focus on the central axis of the streamtube, which I believe would be the desired behavior (haven't yet had the time to do in the first iteration), and if that's done then on that basis it would probably be easy to cover the points (sphere centers) as well. So we could leave it with the mesh hugging crosshairs, or disable for now, or if it's a dependency, we could iterate on this.

[etpinard]

Will soon be superseded by https://github.com/gl-vis/gl-streamtube3d