Support Farm Rust Plugin

[wre232114]

Farm is now 1.0 stable, the Rust plugin(https://www.farmfe.org/docs/plugins/writing-plugins/rust-plugin) are available for community plugin developers too.

We want to develop a Rust vue plugin based on this project, can we have a collaboration. Relative issue: farm-fe/farm#125

[phoenix-ru]

What I immediately see looks like a huge limitation for implementing a native plugin: https://www.farmfe.org/docs/plugins/writing-plugins/rust-plugin#using-swc-in-plugin

(BTW Amazing docs!)

I am using swc_ast and swc_common all over the project. Even though I can make guarantees that the plugin will be compatible with Farm (by fixing versions for example), the

Cause SWC stores the global state in the process, may cause dead lock when you use SWC in your plugin

doesn't seem very promising. What global state do you mean? I know that Atom is now thread-local, what else is there? Fervid doesn't use anything else except for parsing and AST visitors.

I also can't really switch to using farmfe_core::swc_ast as that means having farmfe_core as a dependency for the core crate and all other crates, which is undesirable, because there are other means of distribution (Wasm, raw Napi, unplugin, with Deno and Bun planned). From experience with SWC I can say that some of its crates are incompatible with Wasm, which means that using Farm will make Fervid incompatible with Wasm sadly.

The rust toolchain is defined in rust-toolchain.toml, it should not be modified manually

https://www.farmfe.org/docs/plugins/writing-plugins/rust-plugin#choosing-rust-toolchain

This is also a limitation for native plugins I was expecting tbh. Again, I can pin Farm version and manually synchronise the toolchain.

---

Early thoughts: however tempting it seems to use a native plugin, no-one can really ignore DLL incompatibility. At the moment I am thinking of providing a stable Vite/unplugin version of the project with a potential Farm plugin later down the road.

@wre232114 Please tell me if I am wrong and it is actually doable.

[wre232114]

First question:

I am using swc_ast and swc_common all over the project. Even though I can make guarantees that the plugin will be compatible with Farm (by fixing versions for example), the

Cause SWC stores the global state in the process, may cause dead lock when you use SWC in your plugin

doesn't seem very promising. What global state do you mean? I know that Atom is now thread-local, what else is there? Fervid doesn't use anything else except for parsing and AST visitors.

Dead lock happens only when the plugin reuses the ast parsed by Farm and access HygieneData(ast_node.span.ctxt, for example, which uses Globals, a global variable shared between threads with Mutex).

There are 2 effective methods to avoid dead lock:

1. Do not shared ast with Farm, means the plugin can implement transform hook, parse/transform/generate code in the transform hook, which is the same as current @vitejs/plugin-vue. It's still far faster than js-plugins(vite/unplugin) cause .vue files can be handled in parallel and pure rust parse/transform/generate are fast, the performance won't be a big difference even if you do not share the ast with Farm. In this way, you can use any swc version in Fervid.
2. Make sure Fervid does not access HygieneData, and share ast with Farm, it may be faster but less flexible.

I prefer method 1, do not share ast with Farm, then Farm and Fervid are totally independent.

Question 2:

I also can't really switch to using farmfe_core::swc_ast as that means having farmfe_core as a dependency for the core crate and all other crates, which is undesirable, because there are other means of distribution (Wasm, raw Napi, unplugin, with Deno and Bun planned). From experience with SWC I can say that some of its crates are incompatible with Wasm, which means that using Farm will make Fervid incompatible with Wasm sadly.

As talked above, if you do not share ast with Farm, you can use any swc versions, it's independent.

Question 3:

The rust toolchain is defined in rust-toolchain.toml, it should not be modified manually

https://www.farmfe.org/docs/plugins/writing-plugins/rust-plugin#choosing-rust-toolchain

This is also a limitation for native plugins I was expecting tbh. Again, I can pin Farm version and manually synchronise the toolchain.

This limitation only applies to the Rust plugin, you can build Fervid with any rust toolchains. I think the architecture should be:

Fervid compiler is toolchain independent, based on Fervid compiler, different toolchains can be used to build isolate products

I think Farm Rust plugin is actually doable, and can greatly speedup Vue compilation, and the limitation is reasonable, @phoenix-ru what do you think

[phoenix-ru]

@wre232114 Thank you for your detailed input. I will try the "don't share AST" at first, and then maybe go further with a shared AST, this is a good tip.

Didn't know I can have an independent toolchain. Could you please advise if Farm plugin needs to be in its own repository or it can be a part of a monorepo?

cause .vue files can be handled in parallel

Yes, you are absolutely right. Fervid already supports true multithreading in Napi and it's significantly faster than @vue/compiler-sfc.
I believe we'd see the same results in Farm.

Overall, I think I can start with a plugin very soon

[wre232114]

Didn't know I can have an independent toolchain. Could you please advise if Farm plugin needs to be in its own repository or it can be a part of a monorepo?

Farm plugin can be a part of a monorepo, the RUSTUP_TOOLCHAIN environment variable can be used to specify different toolchains inside a monorepo, see https://rust-lang.github.io/rustup/overrides.html?highlight=rust-toolchain

cause .vue files can be handled in parallel

Yes, you are absolutely right. Fervid already supports true multithreading in Napi and it's significantly faster than @vue/compiler-sfc. I believe we'd see the same results in Farm.

Do you mean Fervid can enable multi-threading itself? Farm enables maximum parallel for multiple files too.

Looking forward to cooperate on the full new Rust vue plugin, it would be a start of a full new future of vue projects compilation!

[phoenix-ru]

Do you mean Fervid can enable multi-threading itself?

Not Fervid itself, but rather Node.js with libuv:

fervid/crates/fervid_napi/src/lib.rs

Lines 33 to 44 in 676293e

	#[napi]
	pub fn compile_async(
	&self,
	source: String,
	signal: Option<AbortSignal>,
	) -> AsyncTask<CompileTask> {
	let task = CompileTask {
	compiler: self.to_owned(),
	input: source,
	};
	AsyncTask::with_optional_signal(task, signal)
	}

And the bench result: https://github.com/phoenix-ru/fervid?tab=readme-ov-file#is-it-fast

Looking forward to cooperate on the full new Rust vue plugin, it would be a start of a full new future of vue projects compilation!

Me as well, I think native tooling would make Vue and Nuxt way more appealing :)

[phoenix-ru]

@wre232114 Looks very promising from half an hour of experimentation:

Thank you for making the setup quite easy!

I would still play around more, can you point me at how to emit virtual modules in Farm? I basically want to use something like import 'virtual:some-file.css' in the emitted code and let Farm handle the rest as if it was a normal import (the virtual file does not exist on the file system).
I am thinking about

context.emit_file(EmitFileParams {
    resolved_path: "virtual:some-file.css".into(),
    name: "virtual:some-file.css".into(),
    content: ".bg-red { background: red }".into(),
    resource_type: farmfe_core::resource::ResourceType::Css,
});

Is this correct?

[wre232114]

No, context.emit_file is used to emit additional static assets like images.

For virtual modules, you use use resolve and load to support resolve and load a virtual module, example pseudocode:

// resolve
fn resolve(self, param) -> .. {
    // support resolve virtual module
    if param.source == "virtual:some-file.css" {
       return "virtual:some-file.css"
    }
}

// load
fn load(self, param) -> .. {
    if param.resolvedPath == "virtual:some-file.css" {
         // return the virtual module code
         let code = self.descriptors.get("virtual:some-file").cssCode;
         return code;
    }
}

[wre232114]

I think the js vite plugin vue is a good example for implement a vue plugin: https://github.com/vitejs/vite-plugin-vue/blob/main/packages/plugin-vue/src/index.ts#L224

[wre232114]

BTW, is there a repository of the rust vue plugin, we would like to contribute if we have time

[phoenix-ru]

No, context.emit_file is used to emit additional static assets like images.

Thank you for clarifying it. It would be of great help if the rust code was documented so that plugin authors don't have to guess. I saw that you have a far bigger API than mentioned in the docs.
Additionally, I believe that docs.rs to your API or a Plugin trait should be one of the first links someone sees in this page: https://www.farmfe.org/docs/plugins/writing-plugins/rust-plugin
Another point: this page https://www.farmfe.org/docs/quick-start is missing "raw" installation, i.e. using Farm in an existing project.
I can help you with writing the docs.

For virtual modules, you use use resolve and load to support resolve and load a virtual module

This is helpful, however, how do I save one? I tried attaching it to a hash map self.virtual_modules, but it's only a read borrow &self, and I absolutely do not want to use async hash maps of any sort just to cache virtual files. context is my next candidate, how should I do that?
Vitejs plugin by itself uses a global cache, which is obviously not a solution here:
https://github.com/vitejs/vite-plugin-vue/blob/fff40f67f05763d24e8c752fa98bcd08e19f7c82/packages/plugin-vue/src/utils/descriptorCache.ts#L80

---

Another point I found is — load hook is synchronous. I think you took its inspiration from Vite, however, I believe they made a mistake and it should be async. This line proves my point: https://github.com/vitejs/vite-plugin-vue/blob/fff40f67f05763d24e8c752fa98bcd08e19f7c82/packages/plugin-vue/src/index.ts#L240
The impact it has on Farm, however, is more devastating, because you force to use sync filesystem instead of relying on Futures (and therefore block threads which could be doing useful work). I don't know what runtime you use under the hood, but I would assume that you could have used tokio::fs and given a convenience method like context.read_file_to_string(param.resolved_path) which can also cache the calls to it (to avoid double-read by plugins).
Another point: it will allow loading modules from the remote location.
I can help you implementing and benchmarking it, but it is a breaking change.

BTW, is there a repository of the rust vue plugin

It is currently in the same repo. I can commit my raw findings if you want.
Fervid uses a bit different approach to the official compiler, e.g. doing the minimum possible work in js, exposing smarter API, etc.
The integration is completely different than Vitejs plugin

[wre232114]

I can help you with writing the docs.

Thank you for the suggestions and PR welcome! You are right, the writing rust plugins guide is just a small start of writing plugins, more APIS need to be documented.

This is helpful, however, how do I save one? I tried attaching it to a hash map self.virtual_modules, but it's only a read borrow &self, and I absolutely do not want to use async hash maps of any sort just to cache virtual files.

We uses Mutex<Xxxx> to cache and share state, example:

pub struct FarmPluginCss {
  css_modules_paths: Vec<Regex>,
  // cache and share css ast
  ast_map: Mutex<HashMap<String, (Stylesheet, CommentsMetaData)>>,
  content_map: Mutex<HashMap<String, String>>,
  sourcemap_map: Mutex<HashMap<String, String>>,
}

In this case, we should release the lock as quick as possible to avoid potential thread block.

Another point I found is — load hook is synchronous

Yes, all hooks must be synchronous, thanks for the advise, I think we can support async hook in next major version(2.0). reading local files is not bottleneck, but for loading remote modules, that would block the thread and affect performance. I think it's ok for now for sync hooks

[phoenix-ru]

@wre232114 Keeping you updated, I have committed a very basic working Farm example integration here:

https://github.com/phoenix-ru/fervid/tree/master/node/examples/farm

And plugin code here:
https://github.com/phoenix-ru/fervid/tree/master/crates/fervid_farmfe

[wre232114]

Awesome! Can not wait to use rust vue plugin in production!

I will add official farm fervid template as soon as the plugin is feature complete