Convert from JNI to FFM/Panama API based on log-c

.github/workflows/ci.yml

@@ -28,31 +28,30 @@ jobs:
       - name: Checkout opendata-java
         uses: actions/checkout@v4
 
+      - name: Checkout opendata
+        run: git clone --depth 1 https://github.com/opendata-oss/opendata.git ../opendata
+
       - name: Install Rust toolchain
         uses: dtolnay/rust-toolchain@stable
 
+      - name: Compute opendata cache key
+        id: opendata-hash
+        run: echo "cargo-lock-hash=$(sha256sum ../opendata/Cargo.lock | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"
+
       - name: Cache cargo registry
         uses: actions/cache@v4
         with:
           path: |
             ~/.cargo/registry
             ~/.cargo/git
-            log/native/target
-          key: ${{ runner.os }}-cargo-${{ hashFiles('log/native/Cargo.lock') }}
+            ../opendata/target
+          key: ${{ runner.os }}-cargo-opendata-${{ steps.opendata-hash.outputs.cargo-lock-hash }}
           restore-keys: |
-            ${{ runner.os }}-cargo-
-
-      - name: Check formatting
-        working-directory: log/native
-        run: cargo fmt --all -- --check
-
-      - name: Run clippy
-        working-directory: log/native
-        run: cargo clippy --all-targets -- -D warnings
+            ${{ runner.os }}-cargo-opendata-
 
-      - name: Build native JNI library
-        working-directory: log/native
-        run: cargo build --release
+      - name: Build opendata-log C library
+        working-directory: ../opendata
+        run: cargo build --release -p opendata-log-c
 
       - name: Set up Java 24
         uses: actions/setup-java@v4

AGENTS.md

@@ -4,14 +4,14 @@
 
 ## Project Overview
 
-OpenData Java provides Java bindings for [OpenData](https://github.com/opendata-oss/opendata) systems via JNI. Each binding module contains both Rust JNI glue code and Java interface classes.
+OpenData Java provides Java bindings for [OpenData](https://github.com/opendata-oss/opendata) systems via Panama FFM (Foreign Function & Memory). Each binding module uses jextract to generate Java bindings from C headers exposed by the OpenData native libraries.
 
-**Important**: This project depends on a sibling clone of the `opendata` repository. The Rust JNI code references OpenData crates via relative paths (`../../../opendata/`).
+**Important**: This project depends on a sibling clone of the `opendata` repository. The C library headers are referenced via relative paths (`../../opendata/`).
 
 ### Modules
 
 - **common**: Shared Java utilities, configuration records, and exceptions (`dev.opendata.common` package)
-- **log**: Java bindings for OpenData Log with Rust JNI bridge (`dev.opendata` package)
+- **log**: Java bindings for OpenData Log via Panama FFM (`dev.opendata` package)
 
 ### Directory Structure
 
@@ -23,63 +23,59 @@ opendata-java/
 │       ├── ObjectStoreConfig.java  # Sealed interface: InMemory, Aws, Local
 │       └── OpenDataNativeException.java
 ├── log/
-│   ├── native/
-│   │   ├── Cargo.toml              # Rust JNI crate
-│   │   └── src/lib.rs              # JNI implementation
 │   └── src/main/java/dev/opendata/
 │       ├── LogDb.java              # Main write API
 │       ├── LogDbReader.java        # Read-only API
+│       ├── RecordBatch.java        # Zero-copy batch builder for writes
+│       ├── Record.java             # Single record (key + value + timestamp)
+│       ├── LogScanIterator.java    # Heap-copying iterator over scan results
+│       ├── LogScanRawIterator.java # Zero-copy iterator (native memory views)
+│       ├── NativeInterop.java      # Panama FFM interop layer
 │       ├── LogDbConfig.java        # Configuration record
 │       └── ...
 ├── build.gradle                    # Gradle multi-module build
 └── settings.gradle
 ```
 
-## JNI Architecture
+## Panama FFM Architecture
 
-### Native Method Pattern
+### jextract Bindings
 
-Java classes declare native methods and load the shared library:
+The `log/build.gradle` uses the `de.infolektuell.jextract` Gradle plugin to generate Java bindings from the C header at `opendata/log/c/include/opendata_log.h`. Generated code lives in the `dev.opendata.ffi` package under the `Native` class.
 
-```java
-public class LogDb implements AutoCloseable {
-    static {
-        System.loadLibrary("opendata_log_jni");
-    }
+### NativeInterop Layer
 
-    private static native long nativeCreate(LogDbConfig config);
-    private static native void nativeClose(long handle);
-}
-```
+`NativeInterop.java` is the package-private interop layer that wraps the generated FFM bindings:
 
-Rust implements JNI functions following the naming convention `Java_<package>_<class>_<method>`:
+```java
+// Handle-based resource management with AtomicBoolean for thread safety
+static abstract class NativeHandle implements AutoCloseable {
+    private final AtomicBoolean closed = new AtomicBoolean(false);
+    private volatile MemorySegment segment;
+    // ...
+}
 
-```rust
-#[no_mangle]
-pub extern "system" fn Java_dev_opendata_LogDb_nativeCreate(
-    mut env: JNIEnv,
-    _class: JClass,
-    config: JObject,
-) -> jlong { ... }
+// Concrete handles: LogHandle, ReaderHandle, IteratorHandle, ObjectStoreHandle
 ```
 
-### Handle-Based Resource Management
+### Error Handling
+
+Every C API call returns `opendata_log_result_t`. The `checkResult()` method inspects the error kind and throws the appropriate Java exception:
 
-- Native resources are represented as `long` handles in Java
-- Rust stores actual objects (e.g., `LogDb`, Tokio runtime) behind the handle
-- Java classes implement `AutoCloseable` to ensure cleanup via `nativeClose`
+- `QUEUE_FULL` → `QueueFullException`
+- `TIMEOUT` → `AppendTimeoutException`
+- All others → `OpenDataNativeException`
 
-### Async Bridge
+### Write Paths
 
-The Rust JNI layer bridges synchronous JNI calls to async OpenData operations:
+There are two write paths through `NativeInterop`:
 
-- Uses two Tokio runtimes: one for user operations, one for SlateDB compaction (prevents deadlock)
-- JNI methods call `runtime.block_on(async_operation)` to wait for results
-- Error handling converts Rust errors to `OpenDataNativeException`
+- **`Record[]`** — `doAppend()` copies each record's `byte[] key` and `byte[] value` into arena-allocated segments at append time. Simple but involves per-record allocation.
+- **`RecordBatch`** — `doAppendBatch()` slices into the batch's pre-built contiguous segments (zero-copy pointer building). The batch builder (`RecordBatch`) accumulates records into two contiguous `MemorySegment`s (keys and values) with timestamp headers already prepended, so the append path only needs to build the pointer arrays.
 
 ### Timestamp Header
 
-The JNI layer prepends an 8-byte timestamp to values for latency measurement:
+The Java layer prepends an 8-byte timestamp to values for latency measurement:
 
 ```
 ┌─────────────────────┬──────────────────────┐
@@ -92,44 +88,30 @@ The JNI layer prepends an 8-byte timestamp to values for latency measurement:
 
 ### Prerequisites
 
-- Rust stable toolchain
+- Rust stable toolchain (for building the C library)
 - Java 24+
 - Sibling clone of `opendata` repository
 
 ### Building
 
 ```bash
-# Build native JNI library
-cd log/native
+# Build the C library
+cd ../opendata/log/c
 cargo build --release
 
-# Build Java modules
-cd ../..
+# Build Java modules (generates FFM bindings via jextract)
+cd ../../../opendata-java
 ./gradlew build
 ```
 
 ### Testing
 
 ```bash
-# Rust tests
-cd log/native
-cargo test
-
-# Java tests (requires native library)
+# Java tests (requires C library built)
 ./gradlew test
-```
-
-### Formatting and Linting
 
-Always run before committing:
-
-```bash
-# Rust
-cd log/native
-cargo fmt
-cargo clippy --all-targets -- -D warnings
-
-# Java uses standard conventions (4-space indent)
+# Log integration tests specifically
+./gradlew :log:test --tests "dev.opendata.*"
 ```
 
 ## Code Conventions
@@ -145,52 +127,28 @@ public sealed interface StorageConfig {
 }
 ```
 
-The Rust JNI layer extracts these via reflection (`instanceof` checks and field access).
-
 ### Tests
 
-Use the **given/when/then** pattern with `should_` naming:
-
-**Rust:**
-```rust
-#[test]
-fn should_extract_config_fields() {
-    // given
-    let config = create_test_config();
-
-    // when
-    let result = extract_fields(&config);
+Use the **given/when/then** pattern with `should` naming:
 
-    // then
-    assert!(result.is_ok());
-}
-```
-
-**Java:**
 ```java
 @Test
 void shouldAppendAndScanEntries() {
-    // given
-    var config = new LogDbConfig(new StorageConfig.InMemory(), null);
+    try (LogDb log = LogDb.openInMemory()) {
+        log.tryAppend(key, value);
 
-    // when
-    try (var log = LogDb.create(config)) {
-        log.append("key".getBytes(), "value".getBytes());
+        try (LogScanIterator iter = log.scan(key, 0)) {
+            // assertions...
+        }
     }
-
-    // then
-    // assertions...
 }
 ```
 
 ### Error Handling
 
-- Rust JNI code should return errors via exceptions, not panics
 - Use `OpenDataNativeException` for all native errors
-- Document any overhead in `lib.rs` header comments
+- Specific subclasses for recoverable errors (QueueFullException, AppendTimeoutException)
 
 ### Imports
 
-Rust: Place all `use` statements at module level, not inside functions.
-
 Java: Standard import ordering (java.*, external, project).

CONTRIBUTING.md

@@ -15,8 +15,8 @@ Thank you for your interest in contributing! This project provides Java bindings
 
 This project contains:
 
-- `log/native` - Rust JNI bindings to the OpenData Log
-- `log/src` - Java interface classes (`dev.opendata` package)
+- `log/src` - Java interface classes and Panama FFM bindings (`dev.opendata` package)
+- `common/src` - Shared utilities, configuration records, and exceptions
 
 Before contributing, read the [README](README.md).
 
@@ -59,38 +59,23 @@ Open an issue describing:
 ```
 your-workspace/
 ├── opendata/          # Clone of opendata repo
-└── opendata-java/     # This project (opendata-omb)
+└── opendata-java/     # This project
 ```
 
 ### Building
 
 ```bash
-# Build native JNI library
-cd log/native
+# Build the native C library
+cd ../opendata/log/c
 cargo build --release
 
-# Build Java modules
-cd ../..
+# Build Java modules (generates FFM bindings via jextract)
+cd ../../../opendata-java
 ./gradlew build
 ```
 
 ## Code Style
 
-### Rust
-
-Run `cargo fmt` before committing:
-
-```bash
-cd log/native
-cargo fmt
-```
-
-All code must pass clippy with no warnings:
-
-```bash
-cargo clippy --all-targets -- -D warnings
-```
-
 ### Java
 
 - Follow standard Java conventions
@@ -101,33 +86,13 @@ cargo clippy --all-targets -- -D warnings
 ### Guidelines
 
 - Write clear, self-documenting code
-- Add comments for complex logic, especially JNI boundary handling
-- Document any overhead introduced (see `lib.rs` header)
-- Prefer returning errors over panicking in JNI code
+- Add comments for complex logic, especially FFM boundary handling
+- Prefer returning errors over panicking in native code
 
 ## Testing
 
 Include tests for new functionality.
 
-### Rust Test Style
-
-Use the `should_` prefix and given/when/then pattern:
-
-```rust
-#[test]
-fn should_extract_timestamp_from_header() {
-    // given
-    let value = create_timestamped_value(12345, b"payload");
-
-    // when
-    let (timestamp, payload) = extract_timestamp_and_payload(&value);
-
-    // then
-    assert_eq!(timestamp, 12345);
-    assert_eq!(payload, b"payload");
-}
-```
-
 ### Java Test Style
 
 ```java
@@ -138,7 +103,7 @@ void shouldAppendAndReadEntry() {
     byte[] value = "test-value".getBytes();
 
     // when
-    AppendResult result = log.append(key, value);
+    AppendResult result = log.tryAppend(key, value);
 
     // then
     assertThat(result.sequence()).isGreaterThan(0);
@@ -148,21 +113,20 @@ void shouldAppendAndReadEntry() {
 ### Running Tests
 
 ```bash
-# Rust tests
-cd log/native
-cargo test
+# Build C library first
+cd ../opendata/log/c && cargo build --release
 
-# Java tests (requires native library built)
+# Java tests (requires C library built)
+cd ../../../opendata-java
 ./gradlew test
 ```
 
 ## Pull Request Process
 
 1. Ensure your code builds and tests pass
-2. Run `cargo fmt` and `cargo clippy`
-3. Update documentation if needed
-4. Add a clear description of changes
-5. Reference any related issues
+2. Update documentation if needed
+3. Add a clear description of changes
+4. Reference any related issues
 
 ## License