Add optional metadata generation

AGENTS.md

@@ -0,0 +1,37 @@
+# Avlos – Agent and contributor guide
+
+This file gives AI and contributors consistent context for the Avlos codebase. For user-facing docs, see [README.md](README.md) and [docs/index.rst](docs/index.rst).
+
+## What Avlos is
+
+Avlos is a code generator: a single YAML device specification is turned into C firmware code, C++ client code, RST documentation, and CAN DBC. It is used for embedded protocols (e.g. [Tinymovr](https://tinymovr.com)).
+
+## Key directories
+
+| Directory | Purpose |
+|-----------|---------|
+| `avlos/definitions/` | Node types (RemoteNode, RemoteAttribute, RemoteFunction, RemoteEnum, RemoteBitmask, RootNode) and Marshmallow schemas |
+| `avlos/generators/` | Generator modules (`generator_c.py`, `generator_cpp.py`, etc.) and [filters.py](avlos/generators/filters.py) (e.g. `avlos_endpoints(instance)`) |
+| `avlos/templates/` | Jinja templates for C, C++, RST, DBC |
+| `avlos/datatypes.py` | `DataType` enum and C name/size maps |
+| `tests/definition/` | YAML specs and [avlos_config.yaml](tests/definition/avlos_config.yaml) |
+| `tests/outputs/` | Generated outputs (often gitignored) |
+
+## Conventions
+
+- **Device spec**: YAML with `name` and `remote_attributes` (nested). Endpoints have `dtype`, `getter_name` / `setter_name` / `caller_name`, and `arguments` for callables.
+- **Config**: `generators.<name>.enabled`, `generators.<name>.paths.<path_key>`. Paths are relative to the config file directory.
+- **C generator**: The same endpoint order is used everywhere (enums, header, impl, metadata). Use `instance | endpoints` for the canonical list.
+
+## Extending the C generator
+
+- Add optional path keys in config (do not add them to `required_paths` if they are optional).
+- In `generator_c.process()`: validate and render new templates using the same `instance` and `instance | endpoints`; append generated file paths to the list that gets clang-format applied.
+
+## Testing
+
+- Unittest/pytest in `tests/`. C generator tests use `deserialize()` + `generator_c.process()` and often run cppcheck on generated C. Outputs go under `tests/outputs/`.
+
+## Docs
+
+- Sphinx in [docs/](docs/); [docs/index.rst](docs/index.rst) is the root.

ARCHITECTURE.md

@@ -0,0 +1,86 @@
+# Avlos architecture
+
+This document describes the structure and data flow of Avlos. For contributor and AI guidance, see [AGENTS.md](AGENTS.md).
+
+## High-level flow
+
+1. **Input**: A device spec YAML file and an Avlos config file (`avlos_config.yaml`).
+2. **CLI or API**: The CLI (`avlos from file <spec>`) or programmatic use loads the YAML and calls the processor. See [avlos/cli.py](avlos/cli.py) and [avlos/processor.py](avlos/processor.py).
+3. **Processor** ([avlos/processor.py](avlos/processor.py)): Loads the config, resolves all paths to absolute (relative to the config file directory), then calls each enabled generator’s `process(instance, module_config)`.
+4. **Deserializer** ([avlos/deserializer.py](avlos/deserializer.py)): Builds the in-memory object tree via `RootNodeSchema`. The schema in [avlos/definitions/remote_node.py](avlos/definitions/remote_node.py) (`RemoteNodeSchema.post_load`) decides whether each node is a RemoteNode, RemoteFunction, RemoteAttribute, RemoteBitmask, or RemoteEnum, and assigns `ep_id` via [avlos/counter.py](avlos/counter.py).
+5. **Generators**: Each generator receives the same `instance` (root of the tree). The C generator uses `avlos_endpoints(instance)` to get one flat list of endpoints; templates iterate that list to produce the enums header, endpoints header, endpoints implementation, and optionally the endpoint metadata files.
+
+```mermaid
+flowchart LR
+    subgraph inputs [Inputs]
+        SpecYAML[device.yaml]
+        ConfigYAML[avlos_config.yaml]
+    end
+    subgraph avlos [Avlos]
+        CLI[CLI or API]
+        Proc[Processor]
+        Deser[Deserializer]
+        Gen[Generators]
+    end
+    subgraph outputs [Generated files]
+        Enums[tm_enums.h]
+        Header[fw_endpoints.h]
+        Impl[fw_endpoints.c]
+        Meta[metadata .h/.c optional]
+    end
+    SpecYAML --> CLI
+    ConfigYAML --> CLI
+    CLI --> Deser
+    Deser --> Proc
+    Proc --> Gen
+    Gen --> Enums
+    Gen --> Header
+    Gen --> Impl
+    Gen --> Meta
+```
+
+## Object model
+
+- **RootNode** ([avlos/definitions/remote_root_node.py](avlos/definitions/remote_root_node.py)): Root of the tree; extends RemoteNode.
+- **RemoteNode** ([avlos/definitions/remote_node.py](avlos/definitions/remote_node.py)): Interior node with `remote_attributes` (ordered dict of children). Can contain more RemoteNodes or endpoint leaves.
+- **Endpoint leaves** (same schema, different `post_load` result):
+  - **RemoteAttribute**: Has `dtype`, `getter_name`, `setter_name`. Used for read/write values.
+  - **RemoteFunction**: Has `caller_name`, `arguments` (list of name + dtype), `dtype` (return type).
+  - **RemoteEnum**: Has `options` (enum); exposes `dtype` as UINT8.
+  - **RemoteBitmask**: Has `flags` (bitmask); exposes `dtype` as UINT8.
+
+All endpoints have `ep_id`, `endpoint_function_name`, and a `dtype` (or equivalent). The **canonical traversal order** is given by `avlos_endpoints(root)` in [avlos/generators/filters.py](avlos/generators/filters.py): depth-first over `remote_attributes`, collecting any node that has `getter_name`, `setter_name`, or `caller_name`.
+
+### Endpoint kind (for metadata)
+
+When generating endpoint metadata, each endpoint is classified into one of:
+
+| Condition | Avlos_EndpointKind |
+|-----------|---------------------|
+| getter only, no setter, no caller | READ_ONLY |
+| setter only, no getter, no caller | WRITE_ONLY |
+| getter + setter, no caller | READ_WRITE |
+| caller, `arguments` empty | CALL_NO_ARGS |
+| caller, `arguments` non-empty | CALL_WITH_ARGS |
+
+```mermaid
+flowchart LR
+    subgraph kinds [Endpoint kinds]
+        RO[READ_ONLY]
+        WO[WRITE_ONLY]
+        RW[READ_WRITE]
+        CNA[CALL_NO_ARGS]
+        CWA[CALL_WITH_ARGS]
+    end
+```
+
+## C generator in detail
+
+- **Required paths**: `output_enums`, `output_header`, `output_impl`. See [avlos/generators/generator_c.py](avlos/generators/generator_c.py).
+- **Optional paths** (when both present, metadata is generated): `output_metadata_header`, `output_metadata_impl`.
+- **Templates**: `tm_enums.h.jinja`, `fw_endpoints.h.jinja`, `fw_endpoints.c.jinja`, and optionally `avlos_endpoint_metadata.h.jinja`, `avlos_endpoint_metadata.c.jinja`.
+- **Filters** (registered in `process()`): `endpoints`, `enum_eps`, `bitmask_eps`, `as_include`, and for metadata `avlos_ep_kind`, `avlos_metadata_dtype`. The endpoint list is identical for header, implementation, and metadata.
+
+## Config and paths
+
+Config is per generator under `generators.<name>`. Paths are relative to the directory containing the config file; the processor converts them to absolute before calling the generator. New outputs (e.g. metadata) are added as new optional path keys; the C generator only generates metadata when both `output_metadata_header` and `output_metadata_impl` are set.

README.md

@@ -98,6 +98,7 @@ The output config defines the output modules that will be used and their options
             paths:
                 output_header: outputs/header.h
                 output_impl: outputs/header.c
+                # optional: output_metadata_header + output_metadata_impl for type-aware UART/ASCII
             header_includes:
             - src/header.h
             impl_includes:

avlos/generators/filters.py

@@ -2,6 +2,25 @@
 from copy import copy
 from typing import List
 
+from avlos.datatypes import DataType
+
+# Avlos_Dtype enum names for metadata (reduced set for UART/ASCII parsing)
+_AVLOS_DTYPE_MAP = {
+    DataType.VOID: "AVLOS_DTYPE_VOID",
+    DataType.BOOL: "AVLOS_DTYPE_BOOL",
+    DataType.UINT8: "AVLOS_DTYPE_UINT8",
+    DataType.INT8: "AVLOS_DTYPE_UINT8",
+    DataType.UINT16: "AVLOS_DTYPE_UINT32",
+    DataType.INT16: "AVLOS_DTYPE_INT32",
+    DataType.UINT32: "AVLOS_DTYPE_UINT32",
+    DataType.INT32: "AVLOS_DTYPE_INT32",
+    DataType.UINT64: "AVLOS_DTYPE_UINT32",
+    DataType.INT64: "AVLOS_DTYPE_INT32",
+    DataType.FLOAT: "AVLOS_DTYPE_FLOAT",
+    DataType.DOUBLE: "AVLOS_DTYPE_FLOAT",
+    DataType.STR: "AVLOS_DTYPE_STRING",
+}
+
 
 def avlos_endpoints(input) -> List:
     """
@@ -101,3 +120,50 @@ def capitalize_first(input: str) -> str:
         String with first character capitalized
     """
     return input[0].upper() + input[1:]
+
+
+def avlos_ep_kind(ep) -> str:
+    """
+    Return the Avlos_EndpointKind enum name for an endpoint (for metadata generation).
+
+    Args:
+        ep: Endpoint object (RemoteAttribute, RemoteFunction, RemoteEnum, or RemoteBitmask)
+
+    Returns:
+        String like AVLOS_EP_KIND_READ_ONLY, AVLOS_EP_KIND_CALL_WITH_ARGS, etc.
+    """
+    has_getter = getattr(ep, "getter_name", None) is not None
+    has_setter = getattr(ep, "setter_name", None) is not None
+    has_caller = getattr(ep, "caller_name", None) is not None
+    if has_caller:
+        num_args = len(getattr(ep, "arguments", None) or [])
+        if num_args == 0:
+            return "AVLOS_EP_KIND_CALL_NO_ARGS"
+        return "AVLOS_EP_KIND_CALL_WITH_ARGS"
+    if has_getter and has_setter:
+        return "AVLOS_EP_KIND_READ_WRITE"
+    if has_getter:
+        return "AVLOS_EP_KIND_READ_ONLY"
+    if has_setter:
+        return "AVLOS_EP_KIND_WRITE_ONLY"
+    return "AVLOS_EP_KIND_READ_ONLY"  # fallback
+
+
+def avlos_metadata_dtype(value) -> str:
+    """
+    Map a DataType or an object with .dtype (endpoint or argument) to Avlos_Dtype enum name.
+
+    Used for value_dtype and arg_dtypes in endpoint metadata. Narrowing (e.g. 64-bit to
+    32-bit) is applied where the metadata enum set is smaller than DataType.
+
+    Args:
+        value: Either a DataType enum member or an object with a .dtype attribute (endpoint, argument)
+
+    Returns:
+        String like AVLOS_DTYPE_UINT32, AVLOS_DTYPE_FLOAT, etc.
+    """
+    dtype = getattr(value, "dtype", value)
+    try:
+        return _AVLOS_DTYPE_MAP[dtype]
+    except KeyError:
+        return "AVLOS_DTYPE_UINT32"  # safe fallback

avlos/generators/generator_c.py

@@ -4,13 +4,60 @@
 from jinja2 import Environment, PackageLoader, select_autoescape
 
 from avlos.formatting import format_c_code, is_clang_format_available
-from avlos.generators.filters import as_include, avlos_bitmask_eps, avlos_endpoints, avlos_enum_eps
+from avlos.generators.filters import (
+    as_include,
+    avlos_bitmask_eps,
+    avlos_endpoints,
+    avlos_enum_eps,
+    avlos_ep_kind,
+    avlos_metadata_dtype,
+)
 from avlos.validation import ValidationError, validate_all
 
 env = Environment(loader=PackageLoader("avlos"), autoescape=select_autoescape())
 
 
+def _generate_metadata_if_requested(instance, config):
+    """Generate endpoint metadata .h/.c when both paths are in config. Returns extra paths or []."""
+    paths = config["paths"]
+    if "output_metadata_header" not in paths or "output_metadata_impl" not in paths:
+        return []
+    meta_header_path = paths["output_metadata_header"]
+    meta_impl_path = paths["output_metadata_impl"]
+    metadata_header_basename = os.path.basename(meta_header_path)
+    os.makedirs(os.path.dirname(meta_header_path), exist_ok=True)
+    template_meta_h = env.get_template("avlos_endpoint_metadata.h.jinja")
+    with open(meta_header_path, "w") as f:
+        print(template_meta_h.render(), file=f)
+    os.makedirs(os.path.dirname(meta_impl_path), exist_ok=True)
+    template_meta_c = env.get_template("avlos_endpoint_metadata.c.jinja")
+    with open(meta_impl_path, "w") as f:
+        print(
+            template_meta_c.render(
+                instance=instance,
+                metadata_header_basename=metadata_header_basename,
+            ),
+            file=f,
+        )
+    return [meta_header_path, meta_impl_path]
+
+
 def process(instance, config):
+    # Validate config has required paths
+    required_paths = ["output_enums", "output_header", "output_impl"]
+    if "paths" not in config:
+        raise ValidationError(
+            "Config validation failed: Missing 'paths' section in avlos config.\n"
+            "Please add a 'paths' section with: output_enums, output_header, output_impl"
+        )
+
+    missing_paths = [p for p in required_paths if p not in config["paths"]]
+    if missing_paths:
+        raise ValidationError(
+            f"Config validation failed: Missing required paths in avlos config: {', '.join(missing_paths)}\n"
+            f"Please add these paths to the 'paths' section of your avlos config file."
+        )
+
     # Validate before generation
     validation_errors = validate_all(instance)
     if validation_errors:
@@ -21,6 +68,8 @@ def process(instance, config):
     env.filters["enum_eps"] = avlos_enum_eps
     env.filters["bitmask_eps"] = avlos_bitmask_eps
     env.filters["as_include"] = as_include
+    env.filters["avlos_ep_kind"] = avlos_ep_kind
+    env.filters["avlos_metadata_dtype"] = avlos_metadata_dtype
 
     template = env.get_template("tm_enums.h.jinja")
     os.makedirs(os.path.dirname(config["paths"]["output_enums"]), exist_ok=True)
@@ -54,17 +103,16 @@ def process(instance, config):
             file=output_file,
         )
 
-    # Post-process with clang-format if available
-    format_style = config.get("format_style", "LLVM")
-
-    generated_files = [
+    base_files = [
         config["paths"]["output_enums"],
         config["paths"]["output_header"],
         config["paths"]["output_impl"],
     ]
+    generated_files = base_files + _generate_metadata_if_requested(instance, config)
 
+    # Post-process with clang-format if available
+    format_style = config.get("format_style", "LLVM")
     for file_path in generated_files:
         success = format_c_code(file_path, format_style)
         if not success and is_clang_format_available():
-            # Only warn if clang-format is installed but failed
             print(f"Warning: clang-format failed for {file_path}", file=sys.stderr)

avlos/generators/generator_cpp.py

@@ -12,6 +12,21 @@
 
 
 def process(instance, config):
+    # Validate config has required paths
+    required_paths = ["output_helpers", "output_header", "output_impl"]
+    if "paths" not in config:
+        raise ValidationError(
+            "Config validation failed: Missing 'paths' section in avlos config.\n"
+            "Please add a 'paths' section with: output_helpers, output_header, output_impl"
+        )
+
+    missing_paths = [p for p in required_paths if p not in config["paths"]]
+    if missing_paths:
+        raise ValidationError(
+            f"Config validation failed: Missing required paths in avlos config: {', '.join(missing_paths)}\n"
+            f"Please add these paths to the 'paths' section of your avlos config file."
+        )
+
     # Validate before generation
     validation_errors = validate_all(instance)
     if validation_errors:

avlos/templates/avlos_endpoint_metadata.c.jinja

@@ -0,0 +1,34 @@
+/*
+* This file was automatically generated using Avlos.
+* https://github.com/tinymovr/avlos
+*
+* Any changes to this file will be overwritten when
+* content is regenerated.
+*/
+
+#include "{{ metadata_header_basename }}"
+
+{% set eps = instance | endpoints %}
+const Avlos_EndpointMeta avlos_endpoint_meta[] = {
+{% for ep in eps %}
+    /* {{ ep.endpoint_function_name }} */
+    [{{ ep.ep_id }}] = {
+        .kind = {{ ep | avlos_ep_kind }},
+        .value_dtype = {{ ep | avlos_metadata_dtype }},
+        .num_args = {% if ep | avlos_ep_kind == "AVLOS_EP_KIND_CALL_WITH_ARGS" %}{{ ep.arguments | length }}{% else %}0{% endif %},
+        .arg_dtypes = {
+            {%- for i in range(4) %}
+            {%- if ep | avlos_ep_kind == "AVLOS_EP_KIND_CALL_WITH_ARGS" and i < (ep.arguments | length) %}
+            {{ ep.arguments[i] | avlos_metadata_dtype }}
+            {%- else %}
+            AVLOS_DTYPE_VOID
+            {%- endif %}
+            {%- if i < 3 %}, {% endif %}
+            {%- endfor %}
+        }
+    }{% if not loop.last %},{% endif %}
+
+{% endfor %}
+};
+
+const uint8_t avlos_endpoint_meta_count = sizeof(avlos_endpoint_meta) / sizeof(avlos_endpoint_meta[0]);