Make Fuller Use of Rustyline

[George-lewis]

Rustyline is very powerful and provides a lot to us, most of which we are currently not using. By providing our own helper implemention, we can add:

A dirty and partial implementation of this can be found on the rustyline branch.

On sharing state between the helper and the main function: The variables and functions vectors can be wrapped in a RefCell. The helper will keep a reference to this cell, and the main function / helper will borrow as necessary and pass down a dereferenced borrow to the library, making the RefCell stuff entirely transparent to the lower layers. This allows us the benefits of dynamic borrow checking (we are asserting that the helper will never borrow while the library has a mutable reference) while avoiding poisoning the library with RefCell and Rc stuff.

• Autocomplete
  • This should be fairly simple for variables and functions:
    • Walk back from the caret position until you find a # or a $, which will indicate the start of a function or variable respectively. Then, slice from there to the caret position, and run that string through the existing search machinery (get_by_repr).
  • This may be more complicated for builtins like operators and constants, because they don't have a predefined starting symbol. This alone doesn't prevent easily autocompleting builtins, but combine that with there being no delimiter requirement in Rustcalc (you don't need to put spaces between symbols) and it quickly becomes apparent how difficult that might be solve. This might be possible:
    • Filter the builtins by prefix in reverse. Take the current character (start with the one under the caret) and find that character in the builtins, then start walking backwards on the input, and for each character check the builtins and filter if they stop matching. Example:
      • Input: "1 + 2 + rand|" (where "|" represents the caret). We begin by creating a list of all the builtins. Most of them will not contain the letter "d", and so will be filtered out. Here's what could possibly remain: ["randint", "randfloat", "divide"]. It probably makes sense to pop off characters as we get matching ones, so the list will become: ["ran": randint, "ran": randfloat, "": divide]. Divide is out of characters so we can probably rule that one out, as we likely want to go with the longest match (or we could just present all options?) anyway, we walk back another character in the input and get "n", we can then pop off a character from each item in the list and filter: ["ra": randint, "ran": randfloat]. The process continues until there are are no candidates left. As candidates run out of characters (but still match) we can add them to a real candidates list, and we can then present the options.
• Hinting (need to investigate what this means)
• Live coloring of input
  • This will require some modifications to key infrastructure. On each edit, we will need to tokenize the input and then color it. There are a few immediate problems here.
  • First of all, tokenizing fails fast for bad input and doesn't return anything particularly useful. We want to support partial tokenization, so that part of the input can be colored while an incomplete token is being typed out. This will necessitate a change in tokenize to return a partial list of tokens on error, and we somehow need to handle partial tokens (which can appear anywhere in the input!). We should probably also try and predict the type of partial tokens, so that they can be colored accordingly, but this isn't crucial.
  • Secondly, tokenization doesn't actually return a true tokenization of the input, it can insert tokens that aren't there, like implicit coefficients. This causes some real problems when all we really want to do is color some input, what ends up happening is that a longer string than was entered by the user is subbed in under them, very bad UX. Additionally, coefficient muls are added even when they're the last valid token, meaning that one partial inputs with a partial token at the end, you get a mul slipped in. It may be time to decouple the tokenize function a bit from the pure backend logic, as it looks like it's quickly going to become a core part of the frontend as well. It may thus also be time to split intermediary transformations like implicit coefficients into a distinct step.
  • Thirdly, tokens do not store any information about their original representation in the input string. Currently, when a list of tokens is stringified the function just takes the "ideal representation" of each token, which more often than not is not what the user input. This again demands that we make tokenize more general, and possibly construct a new type around token that also keeps information about its original representation, such that it can be colored while keeping the structure of the original input. This might also demand a generalization of the stringify function if we intend to be able to both colorize the input as-it-is and also generate an idealized version. Maybe a separate function should be written for stringifying while keeping the original input structure? This may be further motivated by an additional problem: The current stringify function inserts implicit parens as part of its idealized output. This is in the same category as implicit coefficients and is really bad for UX.
• Live validation
  • I think this would be fairly trivial to implement, but I question its utility. It may be desirable to implement a bracket balancing validator?

Proposal

fn stringify<'a>(tokens: &[Tok], colorize: Colorize) -> String
where:
  Tok: Into<Token<'a>> + Display,
  Colorize: Fn(&str, &Token<'a>) -> String

struct HighlightToken<'a> {
  token: Token<'a>,
  repr: String,
  start: usize
}

Stringify will thus accept both HighlightToken and Token. The function will leverage the Display implementation to decide how to format the string, for Token this can be the ideal representation, and for HighlightToken this can be repr, the token as it was entered by the user. Additionally the start index can be used to know the position of the token in the input line, possible useful for determining the location of whitespace. It may make sense to create a new trait, perhaps Stringable.

fn tokenize<'a,  >

...