About understanding how C/C++ extension actually implements/generates SemanticTokens

[REYNEP]

After searching through this repository for a day.... I finally am saying this, i need some help 🙂

SHORT

more specifically saying, how exactly is GetSemanticTokensRequest implemented?
and, i am guessing C/C++ extension generates a Map/Dictionary of functions, types, structs, classes, etc. how and where are those converted to SemanticTokens?

What I know/found/think.... [after a day]

everything about SemanticTokens that C/C++ extension provides seems to be in semanticTokensProvider.ts & LanguageServer/client.ts .... and the provider is calling languageClient.sendRequest(GetSemanticTokensRequest, params); at L33 & the GetSemanticTokensRequest seems to be defined in client.ts#L469

But I couldn't find any implementation of GetSemanticTokensRequest]. [I'm new to typescript]
at this point, i'm only assuming that, there's some other callback functions implemented for stuffs likeonDidChange*
[but couldn't find much references to those too]

Then i thought maybe there's some module or file 'cpptools/getSemanticTokens', but the only reference i could find to that was getSemanticTokens.ts file in microsoft/vscode repo....

Then I finally thought that i'm gonna go check the commits of 'Switching to Semantic Tokens'.... but, couldn't really track where and how all the SemanticTokens are generated.

Any kinda help is really appreatiated.... ❤️

[Colengms]

Hi @REYNEP . Semantic tokens are generated as a result of IntelliSense parsing. IntelliSense is implemented in a closed source component (cpptools and cpptools-srv processes). GetSemanticTokensRequest is a message sent to that component to request a list of semantic tokens from the IntelliSense engine.

[REYNEP]

ahhh, thats why i couldn't find 'cpptools/getSemanticTokens'

is there any way, that I could like call .sendRequest(GetSemanticTokensRequest) from a custom extension?
What if I import GetSemanticTokensRequest from main.js that is in the final distribution and create a javascript extension that has custom SemanticTokensProvider? i would still need to get the active/default client, right to .sendRequest?

Sorry if i'm asking really newbie questions on javascript 😅

[Colengms]

Can you explain why you'd like to do that? :) The interface to the closed source component could change. There may be a way to provide access to the SemanticTokensProvider (in TypeScript, in the open source component), by exposing an API other extensions could access. But, if you have a compelling scenario, you might consider opening a feature request against VS Code to provide access to the semantic tokens associated with an open file. That could potentially enable the scenario for other languages as well.

[REYNEP]

Uh, I actually wanted to modify Semantic Tokens, [you know, delete some, add some.] Currently only 1 SemanticTokensProvider can be registered for a language.... so thats why am thinking about calling GetSemanticTokensRequest inside custom SemanticTokensProvider

[REYNEP]

And yes, it would be cool, if there was functionality for different extensions to provide SemanticTokens for same file or language perhaps.... Maybe the higher priority one gets executed after the lower priority one, so that one can override another

🙂, Should I open a feature request in vscode repo on that?
seems that, someone also opened an issue on 'Single Provider' topic: microsoft/vscode#135580