Skip to content

Fix getTokensFromNode template scanning issue #1560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Aug 11, 2025

The getTokensFromNode function in internal/ls/signaturehelp.go was incorrectly tokenizing template expressions, producing NoSubstitutionTemplateLiteral tokens that spanned across template boundaries instead of recognizing proper template structure.

Issue

When scanning a template expression like:

tag`Hello ${value} world ${value}!`

The function would incorrectly produce a NoSubstitutionTemplateLiteral token from the closing backtick to the next token (e.g., ";" instead of separate "`" and ";" tokens). This happened because the scanner wasn't aware of template context when starting from a specific position within a template expression.

Root Cause

The scanner would interpret closing backticks as the beginning of new NoSubstitutionTemplateLiteral tokens rather than TemplateTail tokens, because it lacked the proper template context when initialized at arbitrary positions within template expressions.

Solution

Added special handling for TemplateExpression nodes in getTokensFromNode:

  • Recognizes when processing a template expression AST node
  • Uses the AST structure to generate proper tokens for template parts
  • Returns correct sequence: TemplateHead → expressions → TemplateMiddle → expressions → TemplateTail
  • Recursively processes expression parts within template spans

Verification

The fix correctly tokenizes the example above as:

  1. TemplateHead: "Hello ${"
  2. Expression: value
  3. TemplateMiddle: "} world ${"
  4. Expression: value
  5. TemplateTail: "}!"`

All existing tests pass and signature help for tagged templates now works correctly.

Fixes #1554.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] getTokensFromNode is wrong when the ast contains template Fix getTokensFromNode template scanning issue Aug 11, 2025
@Copilot Copilot AI requested a review from jakebailey August 11, 2025 22:44
Copilot finished work on behalf of jakebailey August 11, 2025 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

getTokensFromNode is wrong when the ast contains template
2 participants