Automatically generate definition files

[viperML]

It may be interesting to have some mechanism to autogenerate definition files to be consumed by lua-language-server from rust code:

https://luals.github.io/wiki/definition-files

A usecase for example: exposed a function from rust, and get type hints when editing lua code that uses this function.

[mikavilpas]

I started playing with an implementation for writing lua language server definition files. Maybe it could be used here.

I was thinking it could be an opt-in crate separate from this crate, but it would be interesting to discuss various options!

[mikavilpas]

Unfortunately my attempt, while possible to do, proved to perhaps not provide enough value as it does not have a way to type check the annotations (or to generate them from rust code).

Such an approach came up in discussion here: wezterm/wezterm#3132 (comment)

The idea, as I understand it, is to write a new feature to mlua that would allow exposing metadata about modules, functions, variables, etc. I think the best thing would be to be able to generate this at compile time with e.g. macros, so no runtime cost would have to be made. Then the feature could maybe be completely opt-in.

Does this sound like a possible approach?

[CodeDoctorDE]

I can only speak for myself but generating on compile time sounds like a good idea.
If you make modding support for a game and want to export definition files, there is no really need to do this on runtime

[mikavilpas]

I ran into what look like promising existing toolkits fit for the purpose of generating some sort of FFI (foreign function interface) bindings from rust to other languages:

1. UniFFI (https://github.com/mozilla/uniffi-rs) which seems to be the most popular (according to github stars 😄 ). What is notable is that it says it's used at Mozilla in the Firefox project.
2. Diplomat (https://github.com/rust-diplomat/diplomat) is another option but seems to focus heavily on C language bindings as the shared interface. I don't know much about mlua and C, but maybe someone else might know if this is a viable option.
3. Interoptopus (https://github.com/ralfbiedert/interoptopus) is a "polyglot bindings generator for your library"

Here are some suggestions for possible solutions to this problem:

1. Pick option 1 (UniFFI), and add a feature to mlua (e.g. using proc macros) where a UniFFI Interface Definition file can be generated. If this is successful, a decoupled project could be built (e.g. the one I have started) to process these Interface Definition Files into LuaLS type annotation files. In theory, any other possible lua type information system could be supported without requiring changes to mlua.
   • UniFFI design diagram https://mozilla.github.io/uniffi-rs/
   • examples of using procedural macros https://mozilla.github.io/uniffi-rs/proc_macro/index.html
2. Option 3 (Interoptopus) might also work with the same idea.

[Shelim]

Run into the very same question :)

I am looking to integrate lua via mlua into game framework and my strict requirement is to provide vscode intellisense and (less strictly) debugging capabilities. The first one is supplied by Lua Definition Files, so I am having my eyes open here.

As of API design, I would recommend looking into serde's schemars - it is designed around macros, and docs-comments around actual struct- and methods definitions and it allows generating schema via macro, which can be then deployed into specific file as part of build- or runtime process. We actually use that in runtime, as we can supplement it with additional data from the current project files.

Any progress on this feature?

[mikavilpas]

From my side, I looked at all the options I listed in my previous comment, and it seems they could kind of work. All of them are designed for languages that are compiled into runtime code, which is not the case here (types are not executed at runtime). This is likely not a big issue, but in some cases it means we have to conform to a common api that is not designed for lua.

Ref: mozilla/uniffi-rs#2144

Another interesting finding is that these FFI generation packages are designed to work based on either rust functions and data types (with some special restrictions), or an additional schema definition that's separate from rust source code. I felt my goal of having easy type definitions with minimal code changes by users (developers) cannot be achieved so easily.

Some more random notes:

• mlua seems to have an imperative/procedural api ("create a new function") rather than a declarative, reusable api ("this module should have functions based on this")
• schemars seems to define schemas for data only (I think json schemas only support data, not functions), but the idea is very good

---

I also have some ideas towards potential solutions:

Option 1: "fancy solution"

In this option, the types are automatically generated from the rust code. @Shelim, your idea about a schemars like api seems like the best option. In addition to data, functions would have to be supported.

I think this could be supported in a backward compatible manner if the user opts in to using it. It might work if they move their mlua invocations under a new trait whose methods have a signature that is compatible with whatever code generation tool is picked. Maybe the method calls are optimized away automatically to prevent a runtime cost, or maybe this can be done by inlining - I'm not sure.

Option 2: "simple solution"

This option is extremely simple:

• the developer writes the lua types into the rust code manually as comments, e.g.

  /// ```lua
  /// ---@param x number
  /// function foo(x) end
  /// ```
  pub fn create_mlua_method_foo(x: f64) {
      // call mlua here to create the method at runtime
  }

• a new tool is created that is able to extract these comments and generate the types
• no type checking is done for these types, but there are still some benefits with having the types be very near the implementation (colocation)

[Shelim]

@mikavilpas

Option 1 is a dream-like scenario. I believe it can be implemented as separate crate, atop of mlua (and just call some imperative mlua api); The cost of indirect function calls would remain through (or maybe not, if for some clever usage of Rust macros...?)

[lenscas]

Just going to drop this here then:

https://github.com/lenscas/tealr
https://github.com/lenscas/tealr_doc_gen

should support:

A: Generating a site containing any documentation and signatures of exposed functions
B: definition files for the lua language server
C: definition files for teal
D: And... custom definition files if you so desire.

Sadly, I don't have as much time to work on it as I would like, but I happily accept PR's and it should be a good starting point if someone wants to get more things working.

[Drewol]

I have also done some work related to this, though mostly for simpler "data-only" types:
https://github.com/Drewol/luals-gen

Which is more meant to be used with the serialize feature of mlua. It also has a tealr based luals definition generator.

A crate I've been looking at maybe adding luals support to is https://github.com/specta-rs/specta but again that'd maybe more in conjunction with the serialize feature.

@mikavilpas I've actually gone with option 2 for my project, didn't realize other people had ran into this issue as well! If anyone would like a quick and dirty implementation of option 2, have a look at this.

The basic gist is that this code will scan the system/ directory in the src/ folder in search of any special comment and automatically generate the corresponding LuaLS code. So, for example:

/* entry
{
    "version": "1.0.0",
    "name": "quiver.general.deserialize",
    "info": "Deserialize a given JSON string as a Lua value.",
    "member": [
        { "name": "value", "info": "String to deserialize.", "kind": "string" }
    ],
    "result": [
        { "name": "value", "info": "The value, in Lua value form.", "kind": "any" }
    ]
}
*/
fn deserialize(lua: &Lua, value: String) -> mlua::Result<LuaValue> {
    let value: serde_json::Value = serde_json::from_str(&value).map_err(|e| mlua::Error::runtime(e.to_string()))?;
    lua.to_value(&value)
}

will become:

---Deserialize a given JSON string as a Lua value.
---@param value string # String to deserialize.
---@return any value # The value, in Lua value form.
---
--- ---
---[Source Code Definition](https://github.com/sockentrocken/quiver/tree/main/src/system/general.rs#L113)
function quiver.general.deserialize(value) end