JuliaSyntax parser-based REPL completions overhaul

[xal-0]

Overview

As we add REPL features, bugs related to the ad-hoc parsing done by
REPLCompletions.completions have crept in. This pull request replaces most of
the manual parsing (regex, find_start_brace) with a new approach that parses
the entire input buffer once, before and after the cursor, using JuliaSyntax.
We then query the parsed syntax tree to determine the kind of completion
to be done.

Changes

• New, JuliaSyntax-based completions mechanism.

• The complete_line interface now has the option of replacing arbitrary
  regions of text in the input buffer by returning a Region (Pair{Int, Int}
  for consistency with the convention in LineEdit, and pos being a 0-based
  byte offset). This is used to improve the handling of auto-inserted closing
  delimiters.. Leaving this unchanged for now.

• Fixes parsing-related bugs:

  • [REPL] Autocompletion of arguments inside ModuleName.function call is broken #55420
  • Tab completion not working with module specification in @time calls involving another macro #55429
  • [REPL] Autocompletion of ModuleName.@macro_name is broken #55518
  • [REPL] Autocompletion of @benchmark A .= A setup=(A= is broken #55520
  • [REPL] Autocompletion broken for macro names if there's other text after it #55842
  • Incorrect package-name autocompletion suggestion within begin-end block #56389
  • Regression in autocomplete for Julia 1.10 #57307
  • [REPL] Autocompletion of v = Module.Submodule.<TAB> is broken #57611
  • BoundsError when completing a Cmd-string in the REPL #57624

• Fixes some bugs that exist on 28d3bd5 that were found by fuzzing:

  • x \" + TAB throws a FieldError exception
  • String completion would sometimes delete the entire input buffer.
  • Completions should not happen inside comments.

• The duplicate code for path completion in strings, Cmd-strings, and the
  shell has been removed, causing paths to complete the same way for all three.
  Now, ~ is expanded in two situations:

  • If foo exists, or if foo does not exist but there are no
    possible completions:

    "~/foo/b|"     =TAB=>   "~/foo/bar|"
    "~/foo/bar|"   =TAB=>   "/home/user/foo/bar|"
       OR
    "~/foo/bar"|   =TAB=>   "/home/user/foo/bar"|
    

  • If the current path ends with a / and you hit TAB again:

    "~/foo/|"      =TAB=>   "/home/user/foo/|"
       OR
    "~/foo/"|      =TAB=>   "/home/user/foo/"|
    

Future work

• Method completions could be changed to look for methods with exactly the given
  number of arguments if the closing ) is present, and search for signatures
  with the right prefix otherwise.

  • It would be nice to be able to search by type as well as value (perhaps
    by putting ::T in place of arguments).

• Other REPL features could benefit from JuliaSyntax, so it might be worth
  sharing the parse tree between completions and other features:

  • Emacs-style sexpr navigation: C-M-f/C-M-b/C-M-u, etc.
  • Improved auto-indent.

• It would be nice if hints worked even when the cursor is between text.

• CursorNode is a slightly tweaked copy of SyntaxNode from JuliaSyntax that
  tracks the parent node but includes all trivia. It is used with seek_pos,
  which navigates to the innermost node at a given position so we can examine
  nearby nodes and the parent. This could probably duplicate less code from
  JuliaSyntax.

[IanButterworth]

This sounds great! I think we should backport it to at least 1.12 given it fixes so many issues.

[giordano]

@xal-0 first of all, speaking as the author of most of the issues linked to this PR, thank you for tackling this!

Also, can you please add a test for #57772 as well? I verified this works already (which is amazing!)

[giordano]

I'm not sure this quite cover all the broken cases reported in #55518 (there are a few scattered in the discussion)

[xal-0]

I've added a few more from the thread

[StefanKarpinski]

Bravo. Truly excellent quality-of-life PR 👏🏻

[giordano]

Bravo. Truly excellent quality-of-life PR 👏🏻

Best part is that this PR has a negative diff, depsite the fact it added many tests.

[giordano]

Should this be

Suggested change

	:nothing
	nothing

?

julia> macro ignoremacro(e...)
           :nothing
       end
@ignoremacro (macro with 1 method)

julia> @macroexpand @ignoremacro
:(Main.nothing)

julia> macro ignoremacro(e...)
           nothing
       end
@ignoremacro (macro with 1 method)

julia> @macroexpand @ignoremacro

julia> 

Edit: anyway this is for a test, so not particularly important to get perfectly right.

[xal-0]

I was actually unaware that so many things were self-quoting!

[IanButterworth]

This doesn't pass when run via Pkg.test("REPL") because Base.runtests puts all stdlibs on the load path, whereas Pkg.test only does REPL deps and test deps.

Suggested change

	let s = "begin\n using Linear"
	c, r = test_complete(s)
	@test "LinearAlgebra" in c
	@test r == 15:20
	@test s[r] == "Linear"
	end
	let s = "begin\n using Ran"
	c, r = test_complete(s)
	@test "Random" in c
	@test r == 15:17
	@test s[r] == "Ran"
	end

[IanButterworth]

I found two cases that can cause ispath to error, so I think we should make a safe_ispath

1. ArgumentError: embedded NULs are not allowed in C strings: "invalid\0"
   for let name = \"valid.txt\", name⁺ = \"invalid\\0.txt\"

2. IOError name too long (ENAMETOOLONG) for strings that are too long.

Something like this? Or just never throw and just return false?

function safe_ispath(path::AbstractString)
    try
        return ispath(path)
    catch err
        if err isa Base.IOError # i.e. name too long (ENAMETOOLONG)
            return false
        elseif err isa Base.ArgumentError && occursin("embedded NULs", err.msg)
            return false
        else
            rethrow()
        end
    end
end