Camera/phone as a read-only OCR sensor for screens with no digital access — does it belong in OculiX? (working PoC)

[julienmerconsulting]

TL;DR — I built a working PoC that turns a phone (or a fixed USB/IP camera) into a read-only network "eye": it runs OCR on-device and hands OculiX the text over HTTP. I'm genuinely unsure whether this belongs in the OculiX universe or should stay a personal experiment — so I'd rather ask than assume. PoC repo: https://github.com/julienmerconsulting/oculix-eye

The gap

Some systems under test expose no digital access at all — no API, no DOM, no VNC, sometimes no network — yet they have a screen a human reads. OculiX covers local / ADB / VNC screens, but not "a screen we can only see, not connect to." It's a structural property of some regulated markets, not a niche:

• Medical — a certified device (infusion pump, vital-signs monitor) whose firmware can't be modified or instrumented without voiding its FDA/CE certification. A camera reads the screen to verify the displayed rate / alarm — touching nothing.
• Banking — an ATM/kiosk on a locked OS with no test API, or a mobile app that blocks screen capture (FLAG_SECURE). An external camera is the only non-intrusive way to assert what the real screen shows.

In each case the camera is the only bridge that touches nothing.

The idea

A read-only eye OculiX can query — deliberately not an IScreen (you can't click the physical world, so exposing a robot would be dishonest):

OculixScanner eye = OculixScanner.connect(host, port);
String text = eye.readText();   // OCR done on the eye — only text returns, the image never leaves it
eye.disconnect();

It's the remote, camera-side counterpart of OCR.readText(localImage). Two flavours, one API: a phone (CameraX + Tesseract) and a fixed USB/IP camera (OpenCV videoio + Tesseract, host-side). Sovereign by design — OpenCV + Tesseract only, no Baidu, no Play Services. It would live as a separate, read-only extension and not touch the MIT core.

Proof it works

Live test on a Samsung S20 FE: the phone read a printed page and returned the text to the PC over HTTP. Honest scope: raw OCR, good on clean print, deskew/threshold is the next step. I built the PoC before proposing anything — to check feasibility rather than waste your time on a wish.

What I'm asking — @RaiMan @adriancostin6

I really don't know if this fits OculiX, and I'd value your take on:

1. Is the direction relevant to OculiX at all, or better kept as a personal experiment?
2. If relevant — core, or separate extension?

No expectation on my side. It's read-only, extension-shaped, sovereign. Perfectly happy to keep it personal if it doesn't fit the project. 🦎

[RaiMan]

@julienmerconsulting
You are really very productive.

But it does not belong under the umbrella of Oculix, since „seeing“ is one aspect, but the main point is automation.
SikuliX: WYSIWYS - what you see is what you script
Oculix: If you can see it, you can automate it

On the other hand: on my iPhone already the Fotos app can read text from an image and I can send it elsewhere.

[julienmerconsulting]

@RaiMan — fair, and up front: I'm not arguing for the core. It's read-only, it doesn't act, and a separate read-only extension (or just my experiment) is the right home. We agree on placement.

But "Photos already does OCR" hides the why. It isn't OCR-a-photo — it's a read-only OculiX vision node: you hand it a template and it runs find() / exists() (image recognition) and readText() (OCR), via OpenCV + Tesseract. Literally OculiX's find(), but on a camera feed of a screen you can only see.

Where it matters: a certified device (avionics, an infusion pump on a bench) you may not instrument. You can drive its inputs through its real I/O (service bus — ARINC/1553/CAN/serial), firmware untouched; you can't read its display digitally (no API/DOM/VNC). So OculiX is the eyes, the bench is the hands, loop closed:

flowchart TB
    subgraph HOST["🧠 OculiX host — deterministic BDD, no AI"]
        ORCH["Scenario orchestrator"]
        ENG["Engine · find() + OCR + similarity"]
    end
    subgraph BENCH["✋ Hands — controlled test rig"]
        BUS["Real I/O bus<br/>ARINC · 1553 · CAN · serial"]
    end
    subgraph SUT["🔒 System under test — certified, no digital screen access"]
        DEV["Firmware<br/>non-instrumentable (FDA/CE/DO-178C)"]
        DISP["Physical display"]
    end
    subgraph EYE["👁️ The Eye — read-only (OpenCV + Tesseract, on-device)"]
        CAM["Camera<br/>fixed mount · controlled light"]
        PRE["OpenCV front-stage<br/>de-warp → threshold/CLAHE → normalize"]
        REC["find() + OCR<br/>on the rectified frame"]
    end
    ORCH -->|"① command 125 mL/h"| BUS
    BUS -->|"② drive real input"| DEV
    DEV -->|"③ render"| DISP
    DISP -.->|"④ photons · no contact"| CAM
    CAM --> PRE --> REC
    REC -->|"⑤ match / text only · image never leaves the eye"| ENG
    ENG -->|"⑥ assert: '125 mL/h' + alarm icon absent"| ORCH
    classDef host fill:#dbeafe,stroke:#2563eb,color:#1e3a8a,stroke-width:2px
    classDef hands fill:#fef3c7,stroke:#d97706,color:#7c2d12,stroke-width:2px
    classDef sut fill:#fee2e2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px
    classDef eye fill:#dcfce7,stroke:#16a34a,color:#14532d,stroke-width:2px
    class ORCH,ENG host
    class BUS hands
    class DEV,DISP sut
    class CAM,PRE,REC eye

Loading

Concretely: command 125 mL/h over the bus, then the eye asserts the certified display shows 125 mL/h (OCR) and the alarm icon is absent (exists() match) — the full chain down to the glass, zero firmware instrumentation. Apple Photos can't be a deterministic CI assertion in that loop; this can.

Honest limit: a camera isn't a pixel-perfect screenshot — lighting, glare, angle all bite. So the eye runs an OpenCV front-stage (perspective de-warp → adaptive threshold/CLAHE → normalize) before find/OCR, and it needs a controlled rig (fixed mount, controlled light). In regulated bench V&V that rig is the norm, not an exotic ask — and OpenCV is precisely the right tool for that normalization (the same OpenCV OculiX already ships).

So it doesn't replace automation — it's the verify leg where the digital path is sealed (medical, avionics, defense). Whether it ever earns a separate extension or stays mine is entirely your call; I just wanted the why clear. Thanks for engaging. 🦎

[RaiMan]

@julienmerconsulting
Ok, sorry, the iPhone hint was not something to be a replacement for your idea. It was only at the fact, that you do not need Oculix to read text from a picture.

I understood your approach fully, but since in those special environments Oculix would only be a helper to watch camera content about special facts and hence should not be a feature of Oculix, but only an example how one could use it in other scenarios than screen/mouse/touch/keybord based automation.

So it is totally up to you how to handle it.

[adriancostin6]

@julienmerconsulting would have never thought of such a use-case. seems like an exciting idea/use-case. definitely relevant in my eyes 😎