Implement optional rectangular prompt support

[tailhook]

This is a reimplementation of #338 which was reverted.

When Highligher::has_continuation_prompt returns true, rustyline
enables a special mode that passes the original prompt for every line to
the highlighter, and highlighter is free to change color or characters
of the prompt as long as it's length is the same (usual contract for the
highlighter)

Note:

1. Wrapped lines are not prefixed by prompt (this may be fixed in future
   releases either by prepending them, or by collapsing and scrolling
   lines wider than terminal).
2. Unlike Always use rectangular prompt for multi-line input #338 this PR doesn't change the meaning of cursor and end
   fields of the Layout structure.

[bjorn3]

I think it would be nice to see a screenshot of this.

[tailhook]

Force-pushed windows fix

Also, just to make it clear: #350 is fixed in this PR (thanks for the test).

[tailhook]

I think it would be nice to see a screenshot of this.

Here is how we use it in edgedb-cli:

[gwenn]

Thanks.
I will try to review your changes this week-end.

[gwenn]

I am afraid there will be no continuation prompt when colors are disabled (ColorMode::Disabled) or not supported (Windows < 10).

[tailhook]

I am afraid there will be no continuation prompt when colors are disabled (ColorMode::Disabled) or not supported (Windows < 10).

I'm okay with that. Do you think this is a problem?

• I think color is often disabled when you redirect input from pipe, and disabling as much escape codes as possible in that case is nice.
• Downgrading features on windows < 10 is fine too, if we already downgrade colors. I think users will miss colors much more than continuation prompt.

[tailhook]

Just fixed two bugs. One of them is because I've untested on windows previously. Now works fine.

Anything else to update?

[gwenn]

There is a draft here to support continuation prompt even if there is no highlighting.
For reference:

• https://python-prompt-toolkit.readthedocs.io/en/master/pages/reference.html?highlight=continuation#prompt_toolkit.shortcuts.PromptSession
• https://python-prompt-toolkit.readthedocs.io/en/master/pages/asking_for_input.html?highlight=continuation#multiline-input
• https://github.com/prompt-toolkit/python-prompt-toolkit/blob/bd5c368e106d8e3d15244206a92959acb2f70592/prompt_toolkit/layout/containers.py#L1454

Currently, we highlight the whole input at once. So line wrapping cannot interfere.
I will try to figure out how we can avoid to split_highlight.

[tailhook]

There is a draft here to support continuation prompt even if there is no highlighting.

Well, that was my original approach. But the problem is that you can't implement Prompt for &str (because latter is already a "fat" pointer). So it looses the simplicity of .readline("abc>") for simpler cases. Since I see continuation prompt as some kind downgradable enhancement of the input, similarly to colors, I am not concerned of the issue on non-colorized terminals. And since continuation prompt in my cases doesn't depend on initial prompt (except width), highlighter is a good place to put such enhancements.

Also, it may be worth mentioning, that zsh shell hints what is unclosed in continuation prompt (saying quote> if single quote ' wasn't closed and dquote> for double-quote ", etc), it depends on the actual input line. So it's a good idea to put that functionality in the Helper where you can reuse or cache pieces of completion/highlighting/hinting code to make that work faster. (Although, this isn't what I use in my use-cases yet).

[gwenn]

Well, that was my original approach. But the problem is that you can't implement Prompt for &str (because latter is already a "fat" pointer). So it looses the simplicity of .readline("abc>")

str implements Prompt and, &str should be fine with Prompt + ?Sized.

[tailhook]

Well, that was my original approach. But the problem is that you can't implement Prompt for &str (because latter is already a "fat" pointer). So it looses the simplicity of .readline("abc>")

str implements Prompt and, &str should be fine with Prompt + ?Sized.

As far as I understand you can do readline<T: Prompt + ?Sized>(s: &T) but you can't do readline(s: &dyn Prompt) or Box<dyn Prompt> for that (playground). This means that all functions/structures dealing with prompt must have to be generics. Which is huge inconvenience and large codegen. Not sure if there any hacks to avoid that.

[gwenn]

This means that all functions/structures dealing with prompt must have to be generics. Which is huge inconvenience and large codegen. Not sure if there any hacks to avoid that.

We already need to keep track of 'prompt lifetime. And, at some point, the Prompt is reified into a Cow<'prompt,str>.
But maybe after polishing the draft, I will agree with you...

[naalit]

Is this still being considered, or is there another way to get a continuation prompt? I'd like to implement a continuation prompt for my language's REPL.

[basile-henry]

I would also be interested in seeing this PR getting merged at some point. It looks like it would work perfectly for my use case. Anything I can do to help? 😄

[thomcc]

So it took a while for me to figure out what this function does (src is highlighted, offset is the byte offset in the unhighlighted version of `src), but I think it's pretty bug-prone as-is, and would hit issues in a lot of cases it doesnt need to.

That said, I the problem is that this api makes the problem harder than it needs to be. I think instead of passing in the offset, if you pass in the old unhighlighted line (e.g. the unhighlighted version of src up until the point when it needs to be split), you can identify new stuff since it's whats in src but not its unhighlighted counterpart (and you can then see if it looks like a SGR sequence for the prompt support for the splitting).

This gets you "off the hook" for most of the escape sequences (note that if you use terminfo to source your escapes (as ideally you should), you definitely end up with non \x1b[ stuff. Given the current API, it feels like you'd need to use something like vte to parse it, but... I think that would not even really solve the problem very well.

Obviously, this could fail if the highlight function also changed non-highlighting parts of the text, but that isn't a deal breaker — you still have the line end as sort of a "synchronization point".

I'm going to try to see if I can make it work and paste a gist/patch for it. I think this PR is a lot closer now than the last version, but still hits issues for me.

[tailhook]

By the way, my other PR, improves the function to use offset in the highlighted source instead: https://github.com/kkawakam/rustyline/pull/378/files#diff-9df84bc451a5211110d67dd9b47c627ded197d37c6e04e38b5a8d0fe098257d3R252
(I can backport it here, if this will help).

Comparing with the original is probably sub par because it's harder to optimize. I.e. with current semantics we can use memchr to skip until first \x1b char. While this PR doesn't do that, I should be improved later.

Also we replace characters by highlighter for continuation prompt, so we need to create another mechanism for that.

[thomcc]

By the way, my other PR, improves the function to use offset in the highlighted source instead

Hm, that's not quite what I mean either, I'll get back to you later today on that though, I got sidetracked yesterday.

Comparing with the original is probably sub par because it's harder to optimize

These lines tend to be small enough that the performance here isn't visible, and the slowest thing by far is the IO anyway.

I'm way more concerned about correctness than performance here, mostly because the bugs in this tend to be pretty annoying, both as a user of the library, and as a user of the end application...

Also we replace characters by highlighter for continuation prompt, so we need to create another mechanism for that.

Yeah, that's what I meant when i said: "and you can then see if it looks like a SGR sequence for the prompt support for the splitting". (That said, I also think we shouldn't bother with that in cases where the continuation prompt is either empty or all whitespace, but that's neither here nor there, and is relatively easy to implement no matter what).

[tailhook]

Another use case for different chars in highlighted and non-highlighted text if someone wants to highlight some whitespace. You can do that with just background color, but it takes too much attention and to hard to remember which colors mean what. I.e. in my vim config have this:

set list listchars=tab:·⁖,trail:¶,nbsp:×

Which means tabs are replaced by ·⁖, trailing characters are replaced by ¶ and non-breaking spaces are replaced by ×. This shows characters in more subtle and more recognizable way. So I can see these things in the code, but they are saved as original characters.

While I haven't implemented that in any rustyline use case yet, I can imagine it to be useful. And this has to be done in highlighter but not in original text.

[schungx]

I would also be interested in seeing this PR getting merged at some point. It looks like it would work perfectly for my use case. Anything I can do to help? 😄

Almost another year and still no continuation prompt... Any idea if this is going to show up soon?