Integration with Ren'py

[BlackRam-oss]

On the internet I found this library that allows you to run a ren'py file to get all the information of that file.
I think it can be used to give the possibility of making renpy files compatible with pixi vn.

https://github.com/damyo-scientists/renpy-js developed by @horace-velmont

I noticed that the repo is based on the official renpy parser in python:

Renpy: parser.py
https://www.renpy.org/static/wiki/parser.py

[BlackRam-oss]

Hi @blurymind I'm not very experienced with parsers, do you think it's possible to give the possibility to bind renpy files with pixi-vn? as we are doing with ink

[blurymind]

That would be a killer feature for sure... but I dont have alot of experience with parsers. I did some bug fixing on bondagejs (yarn file parser) in the past. That uses jison, not sure if its the best choice.
Inkjs doesnt seem to have a dependency, but their mechanism is to first compile the ink file into a json file and then parse that - its a different strategy than directly parsing it.

Its a cool idea because in theory it might help with directly converting renpy games to pixi-vn games!

[BlackRam-oss]

Update:

I tested https://github.com/damyo-scientists/renpy-js in the basic cases it manages to extrapolate something, but I'm afraid it doesn't have such an effective control.

But I thought of another way that could be effective even if a Parser is not used:
you can use https://github.com/renpy/vscode-language-renpy developed by @duckdoom4. the extension can extrapolate information from the current file to color and report errors.
you could use the same mechanism to extrapolate a model with the information from the file.

It doesn't seem like a simple process and it would need the help of one of the developers. but it would have the advantage that it would be continuously updated by renpy.

[duckdoom4]

Hi, I've started working on a full parser behind the scenes on different branches on that repo. One is on the 'parser' branch, and the other on the 'lsp-server' branch.

(With the lsp-server setup, it should be possible to interact with that from any other application as well.)

I'd love to help setup what you guys need, and get some help with that parser from you guys as well.

To give a quick overview of what is currently implemented:
The tokenizer (lexer) is already complete and fully functional, so it's already possible to extract a whole lot of information from that.

The parser is in it's early stages though, the parser is needed to get an AST (abstract syntax tree). The AST allows us to verify if a token sequence is valid renpy code grammar.

[BlackRam-oss]

Hi, I've started working on a full parser behind the scenes on different branches on that repo. One is on the 'parser' branch, and the other on the 'lsp-server' branch.

(With the lsp-server setup, it should be possible to interact with that from any other application as well.)

I'd love to help setup what you guys need, and get some help with that parser from you guys as well.

To give a quick overview of what is currently implemented: The tokenizer (lexer) is already complete and fully functional, so it's already possible to extract a whole lot of information from that.

The parser is in it's early stages though, the parser is needed to get an AST (abstract syntax tree). The AST allows us to verify if a token sequence is valid renpy code grammar.

yes i noticed that branch. in fact i planned to fork it and then create a library.

i can help, but it would be better to create an npm parser library external to the vscode extension project, in this way i can install it in my project and in this way it is easier to understand how it works.

[duckdoom4]

Hi, I've started working on a full parser behind the scenes on different branches on that repo. One is on the 'parser' branch, and the other on the 'lsp-server' branch.
(With the lsp-server setup, it should be possible to interact with that from any other application as well.)
I'd love to help setup what you guys need, and get some help with that parser from you guys as well.
To give a quick overview of what is currently implemented: The tokenizer (lexer) is already complete and fully functional, so it's already possible to extract a whole lot of information from that.
The parser is in it's early stages though, the parser is needed to get an AST (abstract syntax tree). The AST allows us to verify if a token sequence is valid renpy code grammar.

yes i noticed that branch. in fact i planned to fork it and then create a library.

i can help, but it would be better to create an npm parser library external to the vscode extension project, in this way i can install it in my project and in this way it is easier to understand how it works.

Yeah, so that's the idea behind the lsp-server. You basically build a completely separate application (I chose to build it in typescript as an npm package). That lsp-server can then be used outside of vscode as well. The 'client' is the vscode extension. (Read this if you want to learn more).

In your case you'd implement the client side in your application using the Language Server Protocol setup by Microsoft. The server will then do all the parsing and you'd only have to request the data you want.

[BlackRam-oss]

Hi, I've started working on a full parser behind the scenes on different branches on that repo. One is on the 'parser' branch, and the other on the 'lsp-server' branch.
(With the lsp-server setup, it should be possible to interact with that from any other application as well.)
I'd love to help setup what you guys need, and get some help with that parser from you guys as well.
To give a quick overview of what is currently implemented: The tokenizer (lexer) is already complete and fully functional, so it's already possible to extract a whole lot of information from that.
The parser is in it's early stages though, the parser is needed to get an AST (abstract syntax tree). The AST allows us to verify if a token sequence is valid renpy code grammar.

yes i noticed that branch. in fact i planned to fork it and then create a library.
i can help, but it would be better to create an npm parser library external to the vscode extension project, in this way i can install it in my project and in this way it is easier to understand how it works.

Yeah, so that's the idea behind the lsp-server. You basically build a completely separate application (I chose to build it in typescript as an npm package). That lsp-server can then be used outside of vscode as well. The 'client' is the vscode extension. (Read this if you want to learn more).

In your case you'd implement the client side in your application using the Language Server Protocol setup by Microsoft. The server will then do all the parsing and you'd only have to request the data you want.

LSP is the best goal to achieve for both me and you. I would also be interested in implementing a function that extracts a json with the information of a renpy file (with LSP it is certainly feasible). I have never created one but I can certainly help.

However, I insist on the fact that to create it it would be better to start from an empty project with a Submodules (temporary) that points to https://github.com/renpy/vscode-language-renpy/tree/parser (to obtain the elements that are currently in that repo)
in this way you can create a library to publish on npm, which in the future will be installed in the necessary projects.
also doing so would make it much easier for me to understand because in the project there would only be the files necessary for the parser.

I or you can create a new repo to start working on it (giving access to others), then eventually move it to renpy.

[duckdoom4]

I understand you completely, and I also avoided implementing the lsp elements for that reason. It adds a lot of 'unnecessary' overhead. (In my case it has become a necessity, because that's the only way to do proper multi-threaded processing. Allowing the server to be processing all files in the project without the client freezing).

I will say that it's just me doing this for fun in my free time (which is already limited), and a completely separate library doesn't align with my (personal) current goals for the project.

That said, I'm happy to tell you what files to look at and how to use them.
I already coded it in a modular way, so the tokenizer and parser sources don't use many vscode specific types. Only thing I can think of right now is the text document API. I'm even willing to make a couple extra abstractions if it turns out it does have some dependencies that can't be transferred, so it can be used outside of vscode

[BlackRam-oss]

Ok I can help with vscode-language-renpy implementation but I will need more detailed explanation to understand how to test the LSP.
anyway I will reuse the same code in my library where I need this implementation.

[duckdoom4]

The LSP is something I just started a couple of days ago. I'll need some time to link and move the previous implementation correctly. As far as I can tell, it won't take too much effort. Should be just a matter of calling the right API from the right events.

FYI, I'm in the middle of moving to a new place, so I will have very limited time until the 9th of August.

I think you'll be more interested in the parser branch rn. There is a single function called 'testParser()'. That will run the current implementation of the parser. What it does right now is run the tokenizer on the opened file, then run some simple parser rules and output some parser debug information to the console.

I was working on a formal grammar definition that you can find in the 'grammar' folder. That is very helpful for defining the grammar rules correctly in code. (The grammar file is about 60% complete, and it might be missing some of the new features.)

Quick overview of the functionality:

The tokenizer is fully mature and is used in production right now. It is able to output some 'unusual' tokens (it's sort of a parser itself). It is able to output meta tokens that basically give you a rudimentary AST. It's just not yet verified, which is what the parser is supposed to do.
The tokens are listed in the order they appear in the file, and they form a tree like structure. The main tokens are the individual components (eg. Keywords, variables, single character). Those won't overlap. The meta tokens will overlap and is what makes up the tree (eg. Comment, DefineStatement, SayStatement, etc.)

The tokenizer works equivalent to the one build into vscode, which they use for syntax highlighting. (Which is using this https://github.com/kkos/oniguruma). (My version is a completely custom implementation in ts though).

Sorry if that was too much info dumping, but let me know if you need more info :)

[BlackRam-oss]

The LSP is something I just started a couple of days ago. I'll need some time to link and move the previous implementation correctly. As far as I can tell, it won't take too much effort. Should be just a matter of calling the right API from the right events.

FYI, I'm in the middle of moving to a new place, so I will have very limited time until the 9th of August.

I think you'll be more interested in the parser branch rn. There is a single function called 'testParser()'. That will run the current implementation of the parser. What it does right now is run the tokenizer on the opened file, then run some simple parser rules and output some parser debug information to the console.

I was working on a formal grammar definition that you can find in the 'grammar' folder. That is very helpful for defining the grammar rules correctly in code. (The grammar file is about 60% complete, and it might be missing some of the new features.)

Quick overview of the functionality:

The tokenizer is fully mature and is used in production right now. It is able to output some 'unusual' tokens (it's sort of a parser itself). It is able to output meta tokens that basically give you a rudimentary AST. It's just not yet verified, which is what the parser is supposed to do. The tokens are listed in the order they appear in the file, and they form a tree like structure. The main tokens are the individual components (eg. Keywords, variables, single character). Those won't overlap. The meta tokens will overlap and is what makes up the tree (eg. Comment, DefineStatement, SayStatement, etc.)

The tokenizer works equivalent to the one build into vscode, which they use for syntax highlighting. (Which is using this https://github.com/kkos/oniguruma). (My version is a completely custom implementation in ts though).

Sorry if that was too much info dumping, but let me know if you need more info :)

Ok should I fork this or can you add me to the projects?

[duckdoom4]

Unfortunately I can't add you as a dev myself. (I'm not the owner of the repo with admin rights). So for now you'll need to fork and make a PR.

[BlackRam-oss]

ok I forked and invited you so we can collaborate.
could you create a Document mock to replace activeEditor.document, which points to a renpy file inside the repo? in this way I could run the parser without starting the vscode-language-renpy extension (also because I don't know how to start it)

[duckdoom4]

Doing this right now is a bit difficult due to my limited time until the 9th. I can try to set that up ASAP, but can't promise it can be done before that.

To launch the extension is relatively simple though. (There should be instructions on the repo).

You just open the repo using vscode, then you run npm install, and finally you just run the extension (through vscode's launch mechanism)

[BlackRam-oss]

ok, yes i understand how to execute it. now i will spend some time to understand how the parser actually works.

[duckdoom4]

Awesome, I've added a new section on the main page with the build instructions in a PR. (Also includes tips for usage btw).

I realized that it can be hard to find in the contributing link.

Let me know if you need any more help or if you'd like me to explain something. Chatting using my phone I can do while I'm not at my pc

[BlackRam-oss]

Hi, I took a look at the parser.
From what I understand the relevant part is in the parser-test file from line 32 to 43.
What I don't quite understand is what it does inside statementParser.test.

Anyway, what interests me most is when the parser "extracts information" in the rows that make up a label, has this step already been implemented?
If it hasn't been implemented, could you tell me more or less where and how this step should be implemented and what functions can be useful?

[duckdoom4]

Yes, you are correct. That's the core loop.

The statementParser.test() function is running the code here.

The way the parser is structured right now, is that I have defined a bunch of 'rules', which build 'nodes'.

So to give an example, the line linked above is a rule that loops through a list of rules. One of those rules is the DefineStatementRule. This rule has a test and a parse function. If test returns true, this means the rule should be parsed.

The loop (a bit back up the callstack) then triggers the parse method. This parse method will parse tokens.

Again to give an example, the DefineStatementRule checks the current token. In this case it requires that the first token is the keyword 'define'. The renpy syntax allows an optional integer, then an assignment operation. And lastly it expects an end of line.

In the comments above the class you can see a bit of EBNF grammar definition. Please read the comment in the grammars/renpy.grammar.ebnf file to understand the syntax.

[duckdoom4]

It appears I have not yet made the rule for labels, however I did already define the ebnf definition for labels, so it should be possible to implement the label rule with that definition.

I should note though, that I know some edge cases have not been handled by the parser yet. For example, I don't think I handle comments yet. And the error handling needs to be improved, but if we start with some valid syntax while building the parser we can deal with that later.

Also check out this file (and the others in that folder) for a complete list of all the tokens that the tokenizer can output.
There is a command to display the tokens in a debug view, which allows you to hover over them. I have it bounds to ctrl+alt+shift+T, but you might need to bind it manually

[duckdoom4]

Also, I hope to get some time before the 9th so we can work on it together, but I doubt I will have time before that. On the other hand, this gives you a chance to get familiar with the codebase and stuff so I guess it's not that big of a deal for those couple of days right?

[BlackRam-oss]

yes, don't worry. It will take a while before I understand how the parser works to be able to implement something. Every now and then I will ask you some questions

[duckdoom4]

I had a couple minutes when I took a break and I did a quick test run. Noticed a couple of bugs, so fixed them real quick. I pushed them to the origin branch.

I also added a simple parser_test.rpy file, which will run the define parse and then process the AST to get the defined 'e' symbol. (You can put some breakpoints to see what it does). I did notice I didn't hook it up to the document changed event, so currently it will only run on start

[duckdoom4]

Made a bunch of improvements again. It is now able to parse labels and has improved error handling

[BlackRam-oss]

@duckdoom4
Today I took a look at the changes you made.
But I'm not clear: after the parser reads a file, does it generate an element from which I can extract information?

I'm not sure what else needs to be implemented, however if you write me what needs to be implemented and more or less how you thought of doing it, I can help you

[duckdoom4]

Yeah, so I made some more changes on the main branch. I've merged them into your branch now.

How it works internally

So if we look at the file parser-test.ts, the idea is that the tokenizer runs on the document. This converts individual characters of a file into a sequence of tokens.
(See the DocumentParser constructor)

Then the parser uses this sequence to generate an AST (abstract syntax tree).
ast.process(program); This is a simplified form of logic that makes up the entire application. (See this link for an example of typescript AST. Just paste in some ts code).

We then process the AST (iterate the nodes) to create the 'RpyProgram' object using ast.process(program);, which you use to extract information.

As an example the following line searches for a symbol named 'e' in the global program scope (note that python doesn't really have local scopes, so might remove this later):
const sym = program.globalScope.resolve("e");
This returns a RpySymbol object which holds a definitionLocation and a list of
references. The definitionLocation is the source (file) location where the symbol is defined. And references are the source locations where the symbol is used. (Though I'm pretty sure this list isn't filled yet in the current implementation).

--

What we can do next

So I hope that makes it a bit more clear how it works. Let's talk about what to do next.

My latest addition is label definition processing. I'd recommend looking at the changes I introduced to renpy-grammar-rules.ts since this commit. (I should edit this link to compare with current branch)

What I think could be a really nice introduction for you is attempting to implement the 'jump' statement from renpy. This one is very simple.

Define the grammar

First we need the grammar definition. (I already made it, you can find it here.)
That definition describes how renpy parses the statement, so we can almost 1 to 1 copy the logic. (I also recommend looking at the official implementation to make sure we implement it correctly).

Implement the parser

With the parser implemented (let me know if you still want some more details about that), we can now integrate it. That's also pretty straight forward. If you search in the ebnf file, you will also find out where it's supposed to be used. (In this case only as part of the statement group). So adding it to this list will mean it's processed correctly.

Add the AST

Lastly, we will need to create and implement the CallStatementNodein ast-nodes.ts, which needs the required fields and a process method override to add it's LabelNameNode to the list of references of the label symbol. (And here we can also emit a compile error if the label wasn't defined here yet.)

Result

If we did everything correctly, we should then be able to go back to the parser-test.ts file and use the RpyProgram object to resolve a label name to a RpySymbol and see references added to the list. (Provided we added that rpy code to the file that was processed)

Let me know if you need more info or help with anything. Looking forward to your attempt! (And no worries if you don't quite get it yet, it took me weeks to wrap my head around compiler theory)

[duckdoom4]

Just in case you tried it today or yesterday; I just pushed a fix to the parser branch (yours as well). Apparently I broke it, but it's fixed now xd

[BlackRam-oss]

hello sorry for not writing to you, I've been very busy developing the integration with ink (another narrative language), I didn't expect ink to be very large.

as soon as I'm done with ink I'll switch to renpy.

[BlackRam-oss]

Hello @duckdoom4

I completed/tested the integration with ink (ink documentation is here https://pixi-vn.web.app/ink/ink.html), now I intend to work on integration with Ren'Py.

I quickly explain how this integration will work (maybe I had already hoped for it previously). Exactly as it was developed for integration with ink, I will create a function that receives the text of the .rpy file in input and returns a Json readable from the pixi-vn-json (https://pixi-vn.web.app/advanced/pixi-vn-json.html).

What I need to serve is a function that after giving input a text written with the language of Renpy gives me an element with information on the procedures written in that text. After I will take care of transforming that element into a Json readable by pixi-vn-json.

By rereading the old comments I saw that you had explained to me that in the extension of vs-code for Ren'py there was a class that I could use to do this. is it still like that? Is this class unrelated to the bookstore to create extensions in vs-code? If it is not, is it possible to disconnect it from the library to create extensions in vs-code?

The development of this integration will be divided into step. The first step would simply get the Label list in a file and for each of this understand which dialogues are present. For example in this case:

label sample1:
    "Here is 'sample1' label."

label sample2:
    "Here is 'sample2' label."

I would like to get:

{
    "labels": {
        "sample1": [
            {
                "dialogue": "Here is 'sample1' label.",
            }
        ],
        "sample2": [
            {
                "dialogue": "Here is 'sample2' label.",
            }
        ]
    }
}

Is it currently possible to get this information from your VS-Code extension?

[BlackRam-oss]

yes thank you.

in addition to being necessary for what I intend to create, with this modification I could use vitest files to debug only the features that interest me. without having to start/understand the rest of the project.
and in this way I can also help you.

[BlackRam-oss]

@duckdoom4 Sorry if I insist. But if you could make that little change I was telling you about, I can start with the development

[duckdoom5]

Hi, sorry for the delay. I've had zero time left lately. I'll see what I can do today

[duckdoom4]

Okay, I've pushed some changes that wrap the vscode classes/interfaces used in the tokenizer. You should now be able to run it on strings. (on your fork https://github.com/BlackRam-oss/vscode-language-renpy)

Sorry I can't be of more help rn. I have a lot on my plate currently 😅

[BlackRam-oss]

Great, thank you. I'll start working on it tomorrow.
I'm really interested in giving the possibility to use renpy.

[BlackRam-oss]

@duckdoom4 hello, sorry for the disturbance. I managed to make the parser work using a string. I created a small project https://github.com/DRincs-Productions/pixi-vn-renpy

I can run the code of the following function. But I don't understand now that I have the "parser" object how can I get the information of the text written in renpy that I passed as input.

import { RpyProgram } from "../vscode-extension/src/interpreter/program";
import { LogCategory, logCatMessage } from "../vscode-extension/src/logger";
import { AST } from "../vscode-extension/src/parser/ast-nodes";
import { DocumentParser } from "../vscode-extension/src/parser/parser";
import { RenpyStatementRule } from "../vscode-extension/src/parser/renpy-grammar-rules";
import { LogLevel, TextDocument } from "../vscode-extension/src/utilities/vscode-wrappers";

export async function testParser() {
    let document = new TextDocument(
        `
label start:
    "Hello, World!"
`,
        "start.rpy"
    );

    const parser = new DocumentParser(document as any);
    await parser.initialize();

    const statementParser = new RenpyStatementRule();
    const ast = new AST();

    while (parser.hasNext()) {
        parser.skipEmptyLines();

        if (statementParser.test(parser)) {
            ast.append(statementParser.parse(parser));
            parser.expectEOL();
        }

        if (parser.hasNext()) {
            parser.next();
        }
    }

    const errors: any[] = [];
    for (const error of parser.errors) {
        logCatMessage(LogLevel.Error, LogCategory.Parser, parser.getErrorMessage(error));
        // errors.push(error.errorRange.toVSRange(activeEditor.document));
    }

    logCatMessage(LogLevel.Debug, LogCategory.Parser, ast.toString());

    const program = new RpyProgram();
    ast.process(program);

    for (const error of program.errorList) {
        logCatMessage(LogLevel.Error, LogCategory.Parser, error.message);

        if (error.errorLocation !== null) {
            errors.push(error.errorLocation.range);
        }
    }

    // activeEditor.setDecorations(errorDecorationType, errors);
}

[BlackRam-oss]

these are the logs

[Tokenizer] > Running tokenizer on document: "start.rpy"

stdout | tests/initial.test.ts > Pixi.js Application
[Tokenizer] > Tokenizer completed in 12.24ms

stdout | tests/initial.test.ts > Pixi.js Application
[Parser] > [{
  type: LabelStatementNode
  labelName: {
    type: LabelNameNode
    srcLocation: location
    globalName: 
label start:
    "Hello, World!"

    localName: null
  }
  parameters: null
  hide: false
  block: {
    type: SayStatementNode
    who: null
    attributes: null
    temporaryAttributes: null
    what: {
      type: LiteralNode
      value: 
label start:
    "Hello, World!"

    }
  }
}]

also if you want to debug, i added a little documentation on how to download and run the project https://github.com/DRincs-Productions/pixi-vn-renpy . i also invited you if you want to make changes

[duckdoom4]

Interesting, it seems the AST is parsed correctly, but it's failing to get the correct string.

I think it's a bug with the vscode type wrappers I built specifically for you. I'll have a look tomorrow/later tonight

[duckdoom4]

There, fixed it for ya :)

[BlackRam-oss]

Yes, it works! 🚀
Now I should just map the data that is in ast.nodes.buffer to get the label information and make it compatible with pixi-vn json

[BlackRam-oss]

I should be able to create a first version of this library very easily

[BlackRam-oss]

Hi @duckdoom4 , I managed to get through the library.

but I have doubts about the following texts:

label start:
	"Hello, World!"
	"This is a test."
	"This is a test."
label end:
	"Goodbye, World!"

In this case starting from start renpy will show the following dialogs, since start does not have a return and immediately after it there is end:

"Hello, World!"
"This is a test."
"This is a test."
"Goodbye, World!"

Is there a way to get the information that after start a jump to end will be done?

label back_in_london:
    "We arrived into London at 9.45pm exactly."
    call hurry_home

label hurry_home:
    "We hurried home to Savile Row as fast as we could."
    return

The following text gives me some errors. but I'm not sure why. is it because it doesn't have a label called start?

( if you want to debug them in the usual project I have inserted some test files )

[duckdoom4]

Syntax error: Unexpected end of line.
at: (L4:C5) -> (L5:C0)

Yeah, so this means the parser was in some state where it was attempting to parse some expression/statement but found a \n (end of line character) instead.

You should look at Line 4, character 5 (and another at (L8:C5)( of your input source and see what statement it's attempting to parse. Then we can have a look at the source and/or parser implementation to see where it went wrong.

what I didn't understand is what the peekAnyOf and peek function do?

So the parser has a current()and a peek() function. current is the token we currently look at, and peek is the next token.
The reason it's called peek is because we want to look at the next token, without moving to it.
If we do want to consume the token (meaning we do move to the next one) we use next(). (For example to skip whitespace).

peekAnyOf(list) is then a version of peek where we check if the next token is any of the tokens in the list

[BlackRam-oss]

ok so what it is analyzing is this

label back_in_london:
    "We arrived into London at 9.45pm exactly."
    call hurry_home

label hurry_home:
    "We hurried home to Savile Row as fast as we could."
    return

in my case line 4 would be

    call hurry_home

but I don't understand the letter 5. do I also have to count the spaces before "call"?

I should have time to take a look at it tomorrow

[duckdoom4]

I can check as well. I think I programmed the error message incorrectly. You mentioned it comes from the expectEOL function, which makes me think the error message should actually be Expected newline token @ L4:C5, but got 'X' instead.

My guess is that I have not programmed the call statement into the parser, which makes it output a syntax error.

Same for line 8 with the return statement.

Edit: In fact, I already know this is true since you send me the AST output and it didn't produce a node for the call nor return statements.

So to add those, we have to add a new rule for call, like the one I made for label here.
The rule needs to follow the official renpy grammar syntax rules which I have already defined here.
Same for return (might want to start with return, since that's a more simple statement)

[duckdoom4]

👋 I've used claude to generate a bunch of the missing parser rules. It's currently only on the official branch since I want to make sure I don't break yours. You can just merge it in and you should have a much improved parser then :)

[BlackRam-oss]

Ok thanks. the AI can be useful in these cases.

I also saw that you implemented that part I needed. As soon as I have time I can continue with the library in order to do some tests.