Plugins Support for C/C++

docs/plugins/c-cpp.md

@@ -0,0 +1,109 @@
+# Plugins Development in C/C++
+
+Chipmunk's plugin system, built on WebAssembly and the Component Model, supports developing plugins using C/C++. This document will help you get started with C/C++ plugin development.
+
+## Prerequisites
+
+Developing plugins in C/C++ requires having the following tools and SDKs on the developer's machine:
+
+### wit-bindgen
+
+The [wit-bindgen CLI tool](https://github.com/bytecodealliance/wit-bindgen) generates C/C++ types and function declarations from [WASM Interface Format (WIT)](https://component-model.bytecodealliance.org/design/wit.html) files. 
+Although the pre-generated C header is included in the [C/C++ templates](https://github.com/esrlabs/chipmunk/tree/master/plugins/templates/c-cpp/), `wit-bindgen` is still necessary to generate `parse.c` and `parse_component_type.o` files. This process also regenerates the pre-generated the header file.
+
+### Chipmunk Plugins WIT Definitions
+
+To generate the specific C/C++ functions and types that interface with Chipmunk, developers need to have the [Chipmunk plugins WIT files](https://github.com/esrlabs/chipmunk/tree/master/plugins/plugins_api/wit) available locally on their machines.
+
+### WASI SDK
+
+The [WASI SDK](https://github.com/webassembly/wasi-sdk) is required for compiling C/C++ code into WASM/WASI binaries. Developers need to download the latest version locally and provide its path to the `Makefile` within the plugin templates.
+
+### wasm-tools
+
+The [wasm-tools](https://github.com/bytecodealliance/wasm-tools) suite is required for converting compiled WASM binaries into WASM Component binaries, which embed the WIT interfaces. `wasm-tools` also provides various useful commands for WebAssembly development.
+
+### WASI Reactor
+
+A WASI adapter is required to provide WASI functionalities when creating a WASM component ([more information](https://github.com/bytecodealliance/wit-bindgen/tree/main?tab=readme-ov-file#creating-components-wasi)). `Wasmtime` provides its own adapter, `wasi_snapshot_preview1.reactor.wasm`, which can be downloaded from their [latest releases](https://github.com/bytecodealliance/wasmtime/releases/latest). When developing plugins using the provided templates, the `Makefile` will automatically download the latest version into the plugins directory if a local path is not supplied.
+
+## Plugin Template Starters
+
+Starting plugin development in C/C++ can be complex. To simplify this, we provide [starter templates](https://github.com/esrlabs/chipmunk/tree/master/plugins/templates/c-cpp/) for each plugin type, available for both C and C++. The C++ templates are configured to work with the generated C headers for the API bindings but doesn't provide another abstraction layer between plugins' implementation and the generated bindings. These templates offer a pre-configured structure to organize your source files, generated bindings, vendor dependencies, and build scripts.
+They also include fully working plugin examples that demonstrate all functionalities provided by the Chipmunk host to plugins, as well as the functionalities plugins need to implement, serving as simple showcases.
+
+Each template includes a well-documented `Makefile` that streamlines the compilation process into a final WebAssembly (WASM) component. This `Makefile` handles several key steps automatically:
+
+* **Generating Bindings:** Creates the necessary Wasmtime bindings from your WIT definitions.
+* **WASI Reactor Management:** Downloads the WASI reactor (`wasi_snapshot_preview1.reactor.wasm`) if a local path isn't provided.
+* **Component Conversion:** Transforms the intermediate WASM module into a fully-fledged WASM component, embedding its WIT interfaces.
+
+### Makefile Configuration
+
+The `Makefile` requires you to specify the paths for the mandatory `WASI SDK` and `Chipmunk WIT Files`. You can also optionally provide the path to your locally installed `wasi_snapshot_preview1.reactor.wasm` file to prevent redundant downloads for each plugin.
+
+## Manual Plugin Development: Step-by-Step
+
+This section details the manual steps involved in compiling a C/C++ plugin into a WebAssembly Component. This information is useful for understanding the underlying mechanics automated by the provided templates, or for developing plugins from scratch without relying on them.
+
+### 1. Generate WIT Bindings
+
+The first step is to generate the C/C++ bindings from Chipmunk's WIT Definitions. Developers use the [wit-bindgen CLI tool](https://github.com/bytecodealliance/wit-bindgen) for this purpose, specifying the path to the WIT root for the desired API version and the 'world' (plugin type) for which to generate bindings.
+
+Here is an example for generating bindings for a parser plugin with API version `0.1.0`:
+
+```sh
+wit-bindgen c {path_to_plugins_api_crate}/wit/v0.1.0/ -w chipmunk:parser/parse --out-dir .
+```
+
+This command generates `parse.c`, `parse.h`, and `parse_component_type.o` in the current directory. All generated types and functions from the WIT files can then be referenced and used from `parse.h`.
+
+### 2. Compile to WebAssembly Module
+
+After generating the bindings, compile your C/C++ source code into an intermediate WebAssembly module using the `clang` or `clang++` binaries from the [WASI SDK](https://github.com/webassembly/wasi-sdk). Ensure you set the `--sysroot` flag to the `wasi-sysroot` provided by the SDK and `-mexec-model` to `reactor`.
+
+#### Compiling a Parser Plugin in C
+
+This command compiles the source code of a parser plugin into an intermediate WebAssembly module file:
+
+```sh
+{path_to_wasi_sdk}/bin/clang --sysroot={path_to_wasi_sdk}/share/wasi-sysroot parse.c parse_component_type.o my_parser.c -o my_parser_intermediate.wasm -mexec-model=reactor
+```
+
+#### Compiling a Parser Plugin in C++
+
+C++ plugins require an extra step because `clang` may warn about mixing C and C++ files. First, compile `parse.c` separately:
+
+```sh
+{path_to_wasi_sdk}/bin/clang --sysroot={path_to_wasi_sdk}/share/wasi-sysroot parse.c -o parse.o
+```
+Then, compile the C++ source code into the intermediate WebAssembly module using the compiled binary `parse.o`:
+```sh
+{path_to_wasi_sdk}/bin/clang++ --sysroot={path_to_wasi_sdk}/share/wasi-sysroot parse.o parse_component_type.o my_parser.c -o my_parser_intermediate.wasm -mexec-model=reactor
+```
+
+### 3. Convert to WebAssembly Component
+
+The final step is to convert the compiled WebAssembly module into a WebAssembly Component file. This component includes embedded WIT metadata. This step requires [wasm-tools](https://github.com/bytecodealliance/wasm-tools) and the `wasi_snapshot_preview1.reactor.wasm` adapter (which should be downloaded locally, as detailed in the [WASI Reactor](#wasi-reactor) section).
+
+The command to convert the module and generate the final component in our example is:
+
+```sh
+wasm-tools component new my_parser_intermediate.wasm -o my_parser.wasm --adapt=${path_to_wasi_reactor_file}
+```
+This command generates the `my_parser.wasm` file, which is ready for integration with Chipmunk.
+
+## Memory Ownership
+
+Unfortunately, the generated bindings don't explicitly specify memory ownership. For details, refer to the [wit-bindgen documentation for C/C++](https://github.com/bytecodealliance/wit-bindgen/blob/main/crates/c/README.md).
+
+For convenience, here's a brief summary of the key rules:
+- **Incoming Data:** When a host calls a WASM function, the WASM module takes ownership of the memory passed as arguments. The module is responsible for freeing this memory using the `*_free` functions.
+- **Outgoing Data:** When a WASM function returns a value (like a string or a list), the host takes ownership of that memory and is responsible for freeing it.
+
+## Plugin Integration
+
+For comprehensive information on integrating plugins with Chipmunk, refer to the following documents:
+
+* The integration and metadata sections in the [Plugins Development Guide](./development-guide.md).
+* The [Plugin Integration in Chipmunk UI](./integration-ui.md) guide.

docs/plugins/development-guide.md

@@ -86,6 +86,12 @@ To integrate this compiled plugin with Chipmunk, you have two primary methods:
 
 ---
 
+## Developing Plugins with C/C++
+
+Please refer to [C/C++ Plugins Development](./c-cpp.md) for detailed info developing plugins in C/C++.
+
+---
+
 ## Plugin Configuration
 
 Plugins can define their own configuration schemas. These schemas are presented to users so they can provide the necessary settings. The configuration is then delivered back to the plugin during the session initialization phase. For details on schema definitions, refer to:
@@ -114,7 +120,7 @@ The [`plugins-api`](https://github.com/esrlabs/chipmunk/tree/master/plugins/plug
 *  Use the "Add" function in the Chipmunk UI Plugins Manager, as described in the [Building and Integrating Plugins](#building-and-integrating-plugins) section.
 *  Alternatively, you can manually create a directory at `<HOME>/.chipmunk/plugins/parsers/<plugin-name>/` and copy the compiled WASM file (and optionally metadata TOML and README.md files) into this directory.
 
-To get started quickly, you can use the provided [parser template](https://github.com/esrlabs/chipmunk/tree/master/plugins/templates/parser_template). Simply copy the `parser_template` directory and modify it to implement your custom parser.  
+To get started quickly, you can use the provided [parser template](https://github.com/esrlabs/chipmunk/tree/master/plugins/templates/rust/parser). Simply copy the `parser` directory and modify it to implement your custom parser.  
 
 For reference, see the `string_parser` and `dlt_parser` examples.
 
@@ -176,13 +182,13 @@ Run the benchmark with the following command, replacing `{path_to_input_file}` w
 ```sh
 cargo chipmunk bench core plugin_praser_producer -i {path_to_input_file} -c {path_to_plugin_conig_file}.toml 
 ```
-For a working example of a parser plugin configuration, refer to the [DLT parser config file](https://github.com/esrlabs/chipmunk/tree/master/plugins/examples/dlt_parser/bench_config.toml). This example will help you understand how to structure your configuration for the parser plugin.
+For a working example of a parser plugin configuration, refer to the [DLT parser config file](https://github.com/esrlabs/chipmunk/tree/master/plugins/examples/rust/dlt_parser/bench_config.toml). This example will help you understand how to structure your configuration for the parser plugin.
 
 ---
 
 ## Additional Resources
 
-- **WIT Specifications:** [WIT on GitHub](https://component-model.bytecodealliance.org/design/wit.html)
+- **WIT Specifications:** [WIT References](https://component-model.bytecodealliance.org/design/wit.html)
 - **Cargo Component:** [cargo component GitHub Repository](https://github.com/bytecodealliance/cargo-component)
 - **Wasm-tools:** [Wasm-tools GitHub Repository](https://github.com/bytecodealliance/wasm-tools)
 - **Plugins API Crate Documentation:** [Plugins API](./plugins-api.md)

docs/plugins/plugins-api.md

@@ -25,7 +25,7 @@ Each plugin type is associated with a feature in this crate. A feature must be e
 * To develop a parser plugin, enable the `parser` feature in `Cargo.toml`.
 * Implement `Parser` trait on your struct to define a parser plugin.
 * Use `parser_export!()` macro with your struct to generate the necessary bindings for integration with Chipmunk.
-* Please refer to the [examples](https://github.com/esrlabs/chipmunk/blob/master/plugins/examples) and [parser template](https://github.com/esrlabs/chipmunk/blob/master/plugins/templates/parser_template) provided in Chipmunk repo to get started.
+* Please refer to the [rust examples](https://github.com/esrlabs/chipmunk/blob/master/plugins/examples/rust) and [parser template](https://github.com/esrlabs/chipmunk/blob/master/plugins/templates/rust/parser) provided in Chipmunk repo to get started.
 
 
 ### Byte-Source Plugins:

mkdocs.yml

@@ -15,6 +15,7 @@ nav:
     - Development Guide: plugins/development-guide.md
     - Integration in Chipmunk UI: plugins/integration-ui.md
     - Plugins API Crate: plugins/plugins-api.md
+    - C/C++: plugins/c-cpp.md
   - Development: 
     - Chipmunk Core:
       - Overview: development/core/core.md

plugins/examples/c-cpp/README.md

@@ -0,0 +1,3 @@
+# Plugin Examples in C/C++
+
+This directory contains examples of Chipmunk plugins developed using C/C++ programming language.

plugins/examples/rust/README.md

@@ -0,0 +1,3 @@
+# Plugin Examples in Rust
+
+This directory contains examples of Chipmunk plugins developed using Rust programming language.

plugins/examples/dlt_parser/Cargo.toml → plugins/examples/rust/dlt_parser/Cargo.toml

@@ -6,7 +6,7 @@ authors = ["Ammar Abou Zor <[email protected]>"]
 description = "An example of a Chipmunk parser that replicate the dlt parser that is built-in into Chipmunk"
 
 [dependencies]
-plugins_api = {path = "../../plugins_api", features = ["parser"] }
+plugins_api = {path = "../../../plugins_api", features = ["parser"] }
 dlt-core = { version = "0.20", features = ["fibex"] }
 chrono = "0.4"
 chrono-tz = "0.10"

plugins/examples/dlt_parser/bench_config.toml → plugins/examples/rust/dlt_parser/bench_config.toml

@@ -7,7 +7,7 @@
 
 # Path to the plugin WebAssembly (WASM) file.
 # Can be absolute or relative to the `indexer` directory within the Chipmunk repository.
-binary_path = "../../../../plugins/examples/dlt_parser/target/wasm32-wasip1/release/dlt_parser.wasm"
+binary_path = "../../../../plugins/examples/rust/dlt_parser/target/wasm32-wasip1/release/dlt_parser.wasm"
 
 # Plugin-specific configurations:
 

plugins/examples/file_source/Cargo.toml → plugins/examples/rust/file_source/Cargo.toml

@@ -8,7 +8,7 @@ authors = ["Ammar Abou Zor <[email protected]>"]
 description = "An example of a Chipmunk bytesource plugin that reads byte data from a file and provides it to the Chipmunk application."
 
 [dependencies]
-plugins_api = {path = "../../plugins_api", features = ["bytesource"]}
+plugins_api = {path = "../../../plugins_api", features = ["bytesource"]}
 
 [lib]
 crate-type = ["cdylib"]

plugins/examples/file_source/bench_config.toml → plugins/examples/rust/file_source/bench_config.toml

@@ -2,7 +2,7 @@
 
 # Path to the plugin WebAssembly (WASM) file.
 # Can be absolute or relative to the `indexer` directory within the Chipmunk repository.
-binary_path = "../../../../plugins/examples/file_source/target/wasm32-wasip1/release/file_source.wasm"
+binary_path = "../../../../plugins/examples/rust/file_source/target/wasm32-wasip1/release/file_source.wasm"
 
 # Plugin-specific configurations:
 # TODO:

plugins/examples/protobuf/Cargo.toml → plugins/examples/rust/protobuf/Cargo.toml

@@ -6,7 +6,7 @@ authors = ["Dmitry Astafyev"]
 description = "An example of a Chipmunk parser that replicate the dlt parser that is built-in into Chipmunk"
 
 [dependencies]
-plugins_api = {path = "../../plugins_api", features = ["parser"]}
+plugins_api = {path = "../../../plugins_api", features = ["parser"]}
 prost = "0.13"
 prost-reflect = { version = "0.15", features = ["text-format"]}
 thiserror = "2.0"

plugins/examples/protobuf/bench_config.toml → plugins/examples/rust/protobuf/bench_config.toml

@@ -2,7 +2,7 @@
 
 # Path to the plugin WebAssembly (WASM) file.
 # Can be absolute or relative to the `indexer` directory within the Chipmunk repository.
-binary_path = "../../../../plugins/examples/protobuf/target/wasm32-wasip1/release/protobuf.wasm"
+binary_path = "../../../../plugins/examples/rust/protobuf/target/wasm32-wasip1/release/protobuf.wasm"
 
 # Plugin-specific configurations:
 # TODO:

plugins/examples/string_parser/Cargo.toml → plugins/examples/rust/string_parser/Cargo.toml

@@ -8,7 +8,7 @@ authors = ["Ammar Abou Zor <[email protected]>"]
 description = "An example of a Chipmunk parser that parses the bytes into valid UTF-8 strings line by line"
 
 [dependencies]
-plugins_api = {path = "../../plugins_api", features = ["parser"]}
+plugins_api = {path = "../../../plugins_api", features = ["parser"]}
 memchr = "2.7"
 
 [lib]

plugins/examples/string_parser/bench_config.toml → plugins/examples/rust/string_parser/bench_config.toml

@@ -2,7 +2,7 @@
 
 # Path to the plugin WebAssembly (WASM) file.
 # Can be absolute or relative to the `indexer` directory within the Chipmunk repository.
-binary_path = "../../../../plugins/examples/string_parser/target/wasm32-wasip1/release/string_parser.wasm"
+binary_path = "../../../../plugins/examples/rust/string_parser/target/wasm32-wasip1/release/string_parser.wasm"
 
 # Plugin-specific configurations:
 

plugins/templates/c-cpp/README.md

@@ -0,0 +1,28 @@
+# Plugin Templates in C/C++
+
+This directory contains ready-to-use templates for Chipmunk plugins written in C/C++. 
+Plugin developers can simply copy a template directory to quickly start a new plugin project.
+
+## Step-by-Step Guide
+
+1. **Prerequisites**: Ensure all necessary tools, SDKs, and Chipmunk WIT files are available locally. Refer to the [C/C++ Plugins Development Guide](https://esrlabs.github.io/chipmunk/plugins/development-guide/) for detailed setup instructions.
+
+2. **Configuration**: Provide required paths either directly within the `Makefile` or by passing them as arguments to the `make` CLI command.
+
+3. **Compilation**: Run `make all {arguments}` to compile the template plugins. This command performs the following actions:
+  * Generates bindings from Chipmunk WIT files.
+  * Downloads the latest `wasi reactor` if its path is not specified.
+  * Generates the plugin WASM component file in the `dist` directory. For parser templates, the file will be named `my_parser.wasm`.
+
+4. **Plugin Integration**: To integrate the compiled plugin into Chipmunk:
+  1.  Create a directory for your plugin within the Chipmunk plugins directory. For parser plugins, this path is `~/.chipmunk/plugins/parsers`. The directory name for this template example would be `my_parser`.
+  2.  Copy the compiled plugin WASM component file into the newly created directory.
+  3.  For more detailed integration steps, refer to the [Plugins Integration Documentation](https://esrlabs.github.io/chipmunk/plugins/development-guide/#building-and-integrating-plugins).
+
+5. **Using Plugins in Chipmunk**:
+  1.  Navigate to the **Plugins Manager** in the Chipmunk UI to verify that your plugin is integrated and validated successfully.
+  2.  Start a session, for example, a CLI command session from *Terminal/Execute Command*.
+  3.  Select your plugin from the dropdown menu. For the parser template, it will be named `my_parser`.
+  4.  The configuration schemas defined by the plugin will be displayed on the UI, allowing you to provide necessary inputs.
+  5.  Start the session to view the parsed messages. If Chipmunk is running in a terminal, you can also check logs printed to standard output.
+  6.  For additional details and demonstrations, refer to the [Plugins Integrations in Chipmunk UI](https://esrlabs.github.io/chipmunk/plugins/integration-ui/).

plugins/templates/c-cpp/parser_c/.gitignore

@@ -0,0 +1,6 @@
+vendor/*
+build/*
+target/*
+bindings/*.c
+*.o
+*.wasm