Skip to content

feat(connectors): generic IMAP email connector for any provider (#518)#586

Open
alarcritty wants to merge 4 commits into
open-jarvis:mainfrom
alarcritty:feat/518-dynamic-imap-connector
Open

feat(connectors): generic IMAP email connector for any provider (#518)#586
alarcritty wants to merge 4 commits into
open-jarvis:mainfrom
alarcritty:feat/518-dynamic-imap-connector

Conversation

@alarcritty

Copy link
Copy Markdown
Contributor

Problem

Closes #518.

OpenJarvis only had provider-specific email connectors (Gmail OAuth, Gmail IMAP, Outlook), each hardcoded to one
provider's IMAP host. A user without a Gmail or Outlook account had no way to index their mailbox. The issue
asks to rely on SMTP/IMAP to connect to any inbox.

Testing this end to end against a real inbox surfaced two adjacent problems that broke the flow: a
header-encoding crash during indexing, and Deep Research being pinned to a model that was not pulled.

Solution

Generic IMAP connector (the core of #518)

  • New connectors/imap.py: one connector that works with any provider instead of a subclass per provider. The
    IMAP host is resolved from the email domain (a small table of common providers, falling back to imap.),
    so a user supplies only an email address and an app password.
  • Registered in connectors/init.py so it appears in the Data Sources API and UI.
  • New "Email (IMAP)" card in the frontend catalog (connectors.ts) with email and app-password inputs, alongside
    Gmail and Outlook.

Reliability fixes in the shared IMAP sync

  • Decode every header (From, To, Subject, In-Reply-To, Message-ID) to a plain string. Real mail with raw 8-bit
    header bytes made imaplib return an email.header.Header object, which crashed json.dumps in the knowledge store
    ("Object of type Header is not JSON serializable") and aborted the sync. The decoder also tolerates bogus
    charsets such as unknown-8bit. This also benefits the Gmail and Outlook IMAP connectors.
  • Reconnect and retry on a dropped TLS connection mid-backfill (for example [SSL: BAD_LENGTH]) instead of
    aborting the whole run, so a large inbox finishes without manual retries.

Deep Research model resolution

  • server/research_router.py no longer hardcodes the planner model (which returned a 404 when that model was not
    pulled). It resolves from the server's running model, then the configured default, then the built-in fallback,
    so research runs on a model that is actually available.

Chat tool wiring

  • cli/serve.py adds knowledge_search to the default chat agent tools and injects a KnowledgeStore, so
    non-streaming agent requests can search ingested data. Note: the streaming web chat path bypasses the agent, so
    this does not yet surface in the streaming UI.

Verification

  • Tested against a real Gmail inbox: connected, backfilled, and indexed hundreds of emails into the knowledge
    store; Deep Research returns answers over the indexed mail.
  • New tests in tests/connectors/test_imap.py cover registration, domain host resolution, credential round-trip,
    document output, raw 8-bit headers, and reconnect-on-drop.
  • ruff check and ruff format clean; connector tests pass.

Notes and limitations

  • Connects over implicit TLS on port 993 only. Providers that require STARTTLS on port 143 or a non-standard
    port, or whose IMAP host does not follow imap., are not supported yet (there is no manual host/port
    field). Outlook or Microsoft 365 accounts with app-password IMAP disabled will not connect.
  • This branch bundles three concerns (IMAP connector for Custom Email connector #518, a chat knowledge_search wiring change, and a Deep
    Research model fix), kept as separate commits. They can be split into separate PRs if preferred.

Two reliability fixes in the shared IMAP sync used by the Gmail, Outlook,
and generic IMAP connectors:

- Decode every header (From, To, Subject, In-Reply-To, Message-ID) to a
  plain string. Real mail with raw 8-bit header bytes makes imaplib return
  an email.header.Header object, which crashed json.dumps in the knowledge
  store ("Object of type Header is not JSON serializable") and aborted the
  sync. The decoder also tolerates bogus charsets like unknown-8bit.
- Reconnect and retry on a dropped TLS connection mid-backfill (for example
  "[SSL: BAD_LENGTH]") instead of aborting the whole run, so a large inbox
  finishes without manual retries.
…arvis#518)

Add a single configuration-light IMAP connector that works with any email
provider instead of a hardcoded subclass per provider. The IMAP host is
resolved from the email domain (a small table of common providers, falling
back to imap.<domain>), so most users only supply an email address and an
app password.

Registered in connectors/__init__.py so it appears in the Data Sources API
and UI. Tests cover registration, host resolution, credential round-trip,
document output, raw 8-bit headers, and reconnect-on-drop.

Refs open-jarvis#518.
Add a catalog entry for the generic IMAP connector so the Data Sources
page renders an "Email (IMAP)" card with email and app-password inputs,
alongside the existing Gmail and Outlook cards.

Refs open-jarvis#518.
Add knowledge_search to the default tool set and inject a KnowledgeStore
when building the chat and channel agents, so non-streaming agent requests
can search ingested data. The streaming web chat path bypasses the agent,
so this does not yet surface in the streaming UI.
@alarcritty
alarcritty force-pushed the feat/518-dynamic-imap-connector branch from 3661cf6 to 96ad5fd Compare July 9, 2026 10:54
@alarcritty

Copy link
Copy Markdown
Contributor Author

@jonsaadfalcon please review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Custom Email connector

2 participants