[FEAT] Introduce RESTHeart Repository Connector (`oc-restheart-repository-connector`) for MongoDB & Document REST APIs

[OpenPj]

📄 Summary

We propose introducing a dedicated RESTHeart Repository Connector (oc-restheart-repository-connector) module to support RESTHeart the open-source Web API server for MongoDB and HTTP microservices as a native enterprise data source for OpenCrawling.

Adding native support for RESTHeart allows organizations using RESTHeart to expose MongoDB collections, document databases, GridFS binary files, and microservice APIs to seamlessly crawl, index, vector-embed, and secure JSON documents and attachments into downstream Vector Search, RAG pipelines, and Model Context Protocol (MCP) AI Agents.

---

🎯 Motivation & Enterprise Use Case

RESTHeart by SoftInstigate is a widely adopted open-source framework that instantly turns MongoDB databases and microservices into secure REST, GraphQL, and WebSocket APIs.

Enterprises rely on RESTHeart for:

1. JSON Document Storage & Microservices: Storing multi-structured domain entities, customer records, audit logs, and catalog items in MongoDB collections exposed over HTTP REST APIs.
2. GridFS Binary Attachments: Serving large files, PDF reports, images, and media blobs stored in MongoDB GridFS buckets via RESTHeart's RESTful file endpoints.
3. Built-in Security & RBAC: Enforcing fine-grained Access Control Lists (ACLs), Role-Based Access Control (RBAC), and JWT / Basic / API-Key authentication at the database and collection levels.
4. Change Streams & Real-time WebSockets: Emitting real-time document mutation events whenever documents are inserted, updated, or deleted.

However, when enterprise AI applications deploy Retrieval-Augmented Generation (RAG), documents stored within RESTHeart/MongoDB collections often remain isolated. By introducing oc-restheart-repository-connector, OpenCrawling can incrementally scan RESTHeart collections, extract raw JSON payloads and GridFS attachments, map native RESTHeart ACL security rules into Open Ingestion Standard (OIS) headers, and stream vector embeddings to vector databases with zero-trust identity security.

---

💡 Proposed Architecture & Data Pipeline

graph TD
    subgraph RESTHeart API & MongoDB Store
        RH[RESTHeart Web API Server]
        MongoDB[(MongoDB Collections / GridFS Buckets)]
        Sec[RESTHeart Security Engine - RBAC / ACLs]
        RH --> MongoDB
        RH --> Sec
    end

    subgraph OpenCrawling Ingestion Pipeline
        Connector[oc-restheart-repository-connector]
        Connector -->|1. Authenticate & Page REST Collections / GridFS| RH
        Connector -->|2. Fetch Document JSON & Binary Streams| RH
        Connector -->|3. Extract Roles & User ACLs| Sec
        
        Core[oc-core Pipeline - Tika & TokenTextSplitter]
        Kafka[Apache Kafka - Topic: raw-documents]
        
        Connector -->|4. Emit OIS RepositoryDocuments + ACL Metadata| Core
        Core -->|5. Publish Document Chunks| Kafka
    end

    subgraph Embedding & Vector Target
        Embed[Spring AI / EmbeddingModel - Ollama / OpenAI]
        VectorStore[(Vector Store - pgvector / Qdrant / ES)]
        Kafka --> Embed
        Embed -->|6. Store Embeddings + ACL Metadata| VectorStore
    end

    subgraph AI Retrieval / RAG
        MCP[OpenCrawling Secure MCP Server]
        LLM[LLM Agent / RAG Pipeline]
        LLM -->|7. Query MCP with User Identity| MCP
        MCP -->|8. Filtered Vector Search via OIS ACLs| VectorStore
    end

Loading

---

🏛️ Implementation Details & Specifications

1. Maven Module (oc-restheart-repository-connector)

Create a new Maven sub-module oc-restheart-repository-connector within the OpenCrawling project structure:

<dependency>
    <groupId>org.opencrawling</groupId>
    <artifactId>oc-core</artifactId>
    <version>${project.version}</version>
</dependency>

<!-- Reactive WebClient / HTTP Client for RESTHeart REST APIs -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
</dependency>

2. Core Repository Connector Implementation (RestheartRepositoryConnector.java)

• Implement RepositoryConnector interface from oc-core.
• Annotate with @Component and @ConditionalOnProperty(name = "opencrawling.connector.restheart.enabled", havingValue = "true").
• Execute collection scans leveraging Java 25 Virtual Threads and StructuredTaskScope for concurrent batch fetching across databases, collections, and GridFS buckets.

Key Crawling Strategies:

1. Full Collection Scan: Pages through collections using RESTHeart pagination headers and query parameters (GET /{db}/{coll}?page=1&pagesize=100&sort={"_etag":1}).
2. Incremental Crawling via _etag / _lastModified: Filters updated documents using RESTHeart filter syntax (GET /{db}/{coll}?filter={"_etag":{"$gt":{"$oid":"..."}}}).
3. GridFS File Ingestion: Detects GridFS file collections (*.files) and fetches binary file contents via RESTHeart binary endpoints (GET /{db}/{bucket}/{file_id}) for Apache Tika text & metadata extraction.
4. Change Stream Ingestion: Connects to RESTHeart WebSocket / Change Stream endpoints (GET /{db}/{coll}/_changes) for real-time document mutation streaming.

3. Security & ACL Mapping (RestheartAclExtractor.java)

RESTHeart provides role-based and permission-based security. The connector will automatically extract document and collection-level permission rules and translate them into Open Ingestion Standard (OIS) ACL headers:

// Maps RESTHeart security roles and allowed read principals to OIS ACL Headers
OisDocumentHeader oisHeader = OisDocumentHeader.builder()
    .uri("restheart://" + host + "/" + database + "/" + collection + "/" + documentId)
    .title(document.path("title").asText(documentId))
    .mimeType("application/json")
    .lastModified(Instant.ofEpochMilli(document.path("_etag").path("$timestamp").path("t").asLong()))
    .securityInheritanceEnabled(true)
    .allowedReadPrincipals(extractRestheartRoles(document)) // e.g. ["role:finance", "role:admin", "user:john.doe"]
    .deniedReadPrincipals(List.of())
    .build();

4. Configuration Schema (application.yml)

opencrawling:
  connector:
    restheart:
      enabled: true
      endpoint: "http://restheart-server:8080"
      authentication:
        type: "BASIC" # BASIC, BEARER_JWT, or API_KEY
        username: "${RESTHEART_ADMIN_USER}"
        password: "${RESTHEART_ADMIN_PASSWORD}"
        api-key: "${RESTHEART_API_KEY}"
      target-databases:
        - name: "enterprise_db"
          collections: ["contracts", "reports", "customer_files.files"]
          exclude-collections: ["system.indexes", "_users"]
      scan-gridfs: true
      batch-size: 100
      cron: "0 */15 * * * *" # Sync every 15 minutes

---

🧪 Acceptance Criteria

• New Maven module oc-restheart-repository-connector added to parent pom.xml.
• Implement RestheartRepositoryConnector supporting RESTHeart REST APIs (documents, collections, GridFS files).
• Implement incremental scan mechanism based on _etag / _lastModified timestamps.
• Implement RestheartAclExtractor to map RESTHeart roles and permissions to OIS ACL headers.
• Add integration test suite using Testcontainers (softinstigate/restheart and mongo container images).
• Add full documentation and usage guide to opencrawling.github.io.

---

📌 References

• RESTHeart GitHub Repository: SoftInstigate/restheart
• RESTHeart Documentation: restheart.org/docs
• OpenCrawling Ingestion Architecture: opencrawling.org

[OpenPj]

Prepaid connector foundation milestone (not the full connector): 125 Circle USDC, delivered within 24 hours after settlement-desk confirms payment and the maintainer pins the revision. Scope is limited to adding the Maven module, typed RESTHeart configuration, a disabled-by-default conditional connector skeleton, and mocked pagination for JSON collection documents mapped into the existing RepositoryConnector/OIS document contract. Acceptance: parent reactor includes the module; disabled config instantiates nothing; mocked two-page scan emits each document once with stable URI/title/mime/last-modified fields; auth material is never logged; module tests and repository build for the affected reactor pass. GridFS, change streams, ACL extraction, Testcontainers, deployment, and docs-site work require separate funded milestones. Payment before implementation via verified USDC on Base or Solana; address is sent only after an authorized maintainer chooses network and confirms license/IP, exact target revision, acceptance, and refund/dispute terms. Reply Base or Solana if this bounded foundation is useful.

Thank you for the interest and for proposing this milestone!
At this stage, we are accepting direct community Pull Requests under our standard Apache 2.0 open source contribution guidelines. Feel free to open a PR for oc-restheart-repository-connector if you'd like to contribute to the project!

[mkjsix]

Routing this through RESTHeart instead of connecting to MongoDB directly is the right call. It decouples OpenCrawling from the database, and the connector talks to a stable HTTP surface instead of a moving schema. GridFS fits the same reasoning, RESTHeart already exposes binary files over HTTP, ready for Tika to pick up. Change Streams as the mechanism for incremental ingestion is also the right instinct, event-driven ingestion beats polling for this kind of source.

A few implementation details need reconsidering before this gets built, and the ACL mapping is the one worth resolving first, since it's the part this connector exists to solve.

Change stream endpoint doesn't exist as described. There's no GET /{db}/{coll}/_changes. Change streams are declared through the streams collection metadata (aggregation stages plus a named uri), and a client connects over WebSocket or SSE at /{db}/{coll}/_streams/{streamName}. This also requires MongoDB running as a Replica Set. The Testcontainers acceptance criterion needs the mongo image started with --replSet and an rs.initiate() step. Without it, those tests fail regardless of the connector code.
Refs: https://restheart.org/docs/websocket/change-streams, https://restheart.org/docs/sse/tutorial

GridFS URL pattern is off. Bucket names must end in .files in the path itself, for example mybucket.files, and the binary payload sits behind a /binary suffix: GET /{db}/{bucket}.files/{id}/binary. The pseudo-code in the issue (/{db}/{bucket}/{file_id}) doesn't match this and needs updating.
Ref: https://restheart.org/docs/mongodb-rest/files

Crawl the domain model, not the collections, where a domain view already exists or is worth building. A connector that scans raw collections one by one indexes the physical schema, not the business objects it represents. RESTHeart's aggregation and GraphQL layers can produce that domain view (aggrs collection metadata with $match/$group/$lookup stages, or the restheart-graphql plugin resolving $fk joins across collections), but neither comes predefined. Someone has to write, test and maintain the pipeline or the GraphQL schema for the specific domain, the same as any other API contract, and that's real work, not a shortcut. The case for doing it there instead of inside an OpenCrawling transformation connector is that it stays declarative, testable on its own, and reusable by every other client hitting the same RESTHeart deployment, instead of join logic duplicated inside a Java transformation step that only the crawler benefits from. Where a domain-shaped endpoint already exists for other consumers, pointing the connector at it is close to free. Where it doesn't, building one is still likely less brittle than reconstructing the join inside the connector, but it should be scoped as its own task, not assumed as given.
Refs: https://restheart.org/docs/mongodb-rest/aggregations, https://restheart.org/docs/graphql/graphql-apps, https://restheart.org/docs/graphql/mappings

The ACL mapping needs a decision before implementation, not after. RESTHeart resolves permissions at request time. A permission document (predicate + roles + optional mongo.readFilter/writeFilter) gets evaluated against the incoming request, and readFilter is what implements row-level security, by injecting a query condition. Nothing on the document itself records who is allowed to read it, unless the filter happens to reference a literal field such as owner. RestheartAclExtractor.extractRestheartRoles(document) assumes the opposite, that each document carries its own list of readers.

That gap surfaces in one of two ways once implemented. A crawler running with admin credentials indexes everything and ignores the filters, which over-exposes data through the vector index. A crawler that tries to derive roles from document fields silently drops any permission based on a predicate or a filter not tied to a literal field, which under-serves the roles that depend on it.

Two paths avoid both. Run one crawl per role or service account and let RESTHeart's own readFilter do the partitioning, producing role-scoped indices tagged with that role in the OIS ACL header. Or read the acl collection directly, RESTHeart exposes it like any other collection, and store each role's readFilter as the OIS ACL metadata as is, since it's already plain MongoDB query JSON and needs no translation into a new expression format. Verifying it later means combining {"_id": <document id>} with that role's readFilter content via $and in a normal filter query parameter, for example filter={"$and": [{"_id": {"$oid": "648f..."}}, {"department": "finance"}]} if the role's readFilter is {"department": "finance"}. RESTHeart's own REST API already accepts arbitrary filter JSON, so no new syntax is involved. Recording which of the two the connector takes, before RestheartAclExtractor gets written, avoids a redesign once the first predicate-based permission shows up in a real deployment. The section below reuses the same existence-check idea, moved server-side into two optional RESTHeart plugins instead of round-tripping through the connector.
Refs: https://restheart.org/docs/security/permissions, https://restheart.org/docs/security/authorization, https://restheart.org/docs/mongodb-rest/read-docs

_etag works for incremental sync, but only as a version marker, not a timestamp. RESTHeart doesn't maintain a _lastModified field automatically. _etag is the only field it manages on every write, an ObjectId that gets reassigned each time, and it carries a timestamp component because ObjectIds are time-ordered, which is why $gt comparisons on it work for catching what changed. Treating it as a business "last modified" value is the wrong mental model, treating it as an opaque, monotonic version marker is the right one. Change Streams remain the better mechanism overall, since they push changes instead of requiring a poll-and-compare cycle, and the connector already plans to support them for other reasons.
Ref: https://restheart.org/docs/mongodb-rest/etag

Pagination will not scale on the Full Collection Scan strategy. page/pagesize is implemented with MongoDB's cursor.skip(), documented by RESTHeart itself as slowing down linearly and becoming IO-bound on large collections. Sorting and filtering on _id or _etag with a $gt cursor, the same approach already planned for incremental sync, avoids the skip cost entirely. Server-side pagesize is also capped at 1000, so the batch-size: 100 default in the config example is conservative for a full scan.
Ref: https://restheart.org/docs/mongodb-rest/caching

Two smaller points. RESTHeart supports field projection through the keys parameter, which fits a lightweight metadata-only pass that checks _etag before pulling the full document. On the authentication side, API_KEY isn't a named mechanism in RESTHeart the way Basic or JWT are. The closest existing building block is the token auth mechanism (TokenManager, tokenBasicAuthMechanism), and the config schema should point at whichever plugin backs this in practice.

Making the ACL side easier to build

None of this needs a change inside RESTHeart's own repository. RESTHeart ships a public, documented plugin SPI (Service, Interceptor, Authorizer, ACLRegistry, and a @Inject/Provider mechanism for one plugin to depend on another), packaged as a jar dropped into the plugins directory. That's enough to build this as a companion artifact this project owns and maintains, deployed alongside oc-restheart-repository-connector, not a request directed at RESTHeart's maintainers.

Two clarifications on how, since they matter for correctness. Authorizer.isAllowed() takes a live Request<?> tied to a real HTTP exchange, and there's no supported way to construct one synthetically outside the request pipeline, so nothing here can literally call into RESTHeart's own authorization chain for an arbitrary identity after the fact. ACLRegistry is for a plugin to register rules for its own endpoints, not for querying the existing mongoAclAuthorizer configuration, its right use here is locking down the two new endpoints below to admin or service-account roles, not implementing the read-side logic.

What both new pieces can do instead is exactly what the OpenCrawling-side option above does, read the acl collection through an injected MongoClient, and confirm a match by combining {"_id": doc._id} with that permission's readFilter content via $and in the same way, instead of reimplementing MongoDB's query semantics. Moving it into RESTHeart just means the check happens server-side instead of round-tripping through the connector. The scope that has to stay explicit either way is the general Undertow predicate field: this only handles permissions expressed through roles and mongo.readFilter, and permissions using conditions the plugin doesn't recognize should be treated as out of scope rather than silently ignored.

• A document-level ACL tag, computed on read. A MongoInterceptor that, for each document a GET request returns, runs the existence check above for each role-scoped permission and attaches a computed field, for example _acl: ["role:finance", "user:jdoe"], to the response only. Nothing gets persisted. This is what RestheartAclExtractor already assumes exists, and it turns the connector back into a single crawl instead of one per role.
• An effective-filter introspection endpoint. A Service plugin exposing GET /{db}/{coll}/_acl/_effective?role=X, reading the acl collection and composing the mongo.readFilter clauses that apply to that role into a single JSON filter. This one is the cleanest of the two, it only reads and merges declarative data, with no per-document evaluation involved.
• A live batch check for query time. POST /_acl/_check, taking a list of resource URIs and an identity, resolving that identity's roles and running the same existence check per URI. Same mechanism as the interceptor, applied to arbitrary identities instead of the current request's document, so it needs nothing the interceptor doesn't already need.
• Change streams on the acl collection, documented as the invalidation signal. This works today, acl is a normal collection, and needs no plugin at all, just a line in this project's own docs recommending it as the reindex trigger.

The interceptor is the one worth prototyping first, since it's what this connector's ACL story depends on most directly.
Refs: https://restheart.org/docs/framework/interceptors, https://restheart.org/docs/framework/providers, https://restheart.org/docs/mongodb-rest/read-docs

Note that the above requires a careful additional analysis step, don't take for granted that everything is feasible exactly as described, but it is an investigation direction.

[OpenPj]

Thank you so much @mkjsix ! Receiving contributions from the people behind RESTHeart for us is priceless!!!! 🤩

This is absolutely fantastic news! 🎉

I am incredibly thrilled to welcome you to the OpenCrawling community. Having someone with your deep architectural expertise (especially coming from the RESTHeart ecosystem) actively contributing to this framework is a huge win for the project and the Open Ingestion Standard (OIS) vision.

A native connector for RESTHeart will be a massive capability upgrade for the engine.

I’ll gladly assign this issue to you. Please feel free to open a draft PR whenever you have the initial scaffolding ready, or drop any questions here if you want to bounce architectural ideas around regarding the Connector interfaces, Virtual Threads mapping, or Testcontainers setup.

Thank you so much for stepping up and supporting the initiative. Let's build something great together! 🚀