Consider replacing newlines with empty space in text visualizer

[glopesdev]

The new text visualizer now strips line endings with empty strings so each value fits in a single row of the buffer. An easy way to preserve some extra readability would be to replace the line endings with a single white space.

[glopesdev]

Another option would be to replace all hidden control characters with a rendition of the respective control character, e.g. https://en.wikipedia.org/wiki/Unicode_control_characters#Control_pictures

[PathogenDavid]

I did a partial implementation of using control picture replacements. While it does work, legibility with the default fonts on Windows 10 is not ideal:

Not sure if it maybe looks better on Windows 11. (During dev club I noticed the control pictures did look different on your screen.) If you want to check here's my WIP branch and my test file.

I did notice that this visualizer uses RichTextBox, so in theory we could dim the glyphs which would probably help. That might complicate things for non-winforms UI frontends though and might be overkill.

Assuming it even matters here, it could also put us down a bad path performance-wise. (I didn't dig super far but RichTextBox does seem to have a completely separate path for handling plain text.)

[glopesdev]

@PathogenDavid good point, we've had issues with high-frequency streams before. We now buffer and batch updates automatically so the impact should be less pronounced but would be nice to have a performant code path by default.

I guess another option might be to just replace the control characters by their literal c# escape characters, i.e. tab with \t, newline with \n, etc.

@bruno-f-cruz any thoughts or feedback on this?

[bruno-f-cruz]

I have mixed feelings:

As a developer, I like the change as it would offer a more powerful way to inspect the data streams.

As someone who interacts with new users frequently, and given how steep the learning curve of bonsai is perceived to be, I am afraid that the proposed change will add complexity to one (if not the) most core visualizer of the IDE. We must consider that many of the users of the IDE have no idea what a control character even is.

My proposal would be to have some sort of control in the text editor that toggles between the two modes. Alternatively, we could also think about ways to have "decorator"-like functionality to the text visualizer. It would be interesting to think how this could be extended to support other forms of richer text visualization (thinking about hex viewer, regexp text highlight, etc...)

[glopesdev]

@bruno-f-cruz Yup, I am a bit torn on this one too. The first time I remember hitting this was with matrix representations, either from Math.NET or OpenTK where they would actually include new lines in the ToString method result. For that one it is definitely nicer to simply flatten the rows with a regular space separator (since each row is bracketed anyway).

The other case which is more problematic is NetMQ messages where strings can be returned with a \0 character which literally just kills the control, since we are appending strings and even a single \0 will prevent any other row from being rendered. In that case it's not clear whether we want to replace \0 with a space, especially if it ends up being a dump of some kind of buffered character array from a serial port or network protocol.

[PathogenDavid]

For the sake of posterity: We discussed this again earlier today during dev club and it was decided that the original proposal (replacing newlines with whitespace) is the ideal solution for now and later we will provide an "advanced" visualization via a package or something similar for those who know and care about control characters.

---

We didn't really talk about how we wanted to handle \0 in this case. In my current proposed implementation I opted to replace it (along with all the other unprintable control characters) with "�" the replacement character.

For reference here's what that looks like along with some additional new test strings:

Since I had it implemented on Friday and for the sake of comparison, here's what the C# escape sequences implementation would've looked like:

and for further comparison, here's control pictures with the new sample strings:

(The �s in this case are bring printed by Windows since those are unprintable C1 control characters and have no control pictures.)

For future reference, here are the prototypes pictured above: PathogenDavid@d9ac1d8