Indent/Dedent

[hudson-ai]

• For JSON and Python: track indentation depth
• For Python: also track nesting level of parentheses

[mmoskal]

• extend lexer state (RegexVec) with these two numbers.
• add some configuration syntax in lark (%config { ... } or similar) - this would let one configure what is the 1-level indent string (eg. " ") and whether to track nesting

[mmoskal]

more thinking:

• indent level is fixed for the whole grammar (4 spaces or similar)
• special lexemes INDENT, DEDENT, KEEPDENT (NEWLINE in Python spec), KEEPDENT_LAZY
  they all imply a newline
• need to identify which lexemes open/close parentheses/etc if any
  (special lexemes not generated inside of parentheses (or other lexemes for that matter))
• what about %ignore ? don't ignore \n clearly; otherwise special lexemes are greedy so should be fine?
• for yaml, KEEPDENT can't be greedy:

foo: >
    line 1
        indented line
        another
      a bit less
    and back
bar: 2

string: single_string | double_string | folded_string

# "\n\s*" -> " ", "''" -> "'"
single_string: /'([^']|'')*'/

# "\\\n\s*" -> "", "\n\s*" -> " ",
# then interpret escapes normally
double_string: /"([^"\\]|\\([\nnrfbt]|x[0-9a-fA-F]{2}|u[0-9a-fA-F]{4}|U[0-9a-fA-F]{8}))*"/

folded_string: />\s*/ INDENT text (KEEPDENT_LAZY text)* DEDENT