[Bug]: find_text returns false on low-contrast UI text that read_text_in_region transcribes cleanly — confidence threshold vs source profile

[julienmerconsulting]

Summary

Region.findText() returns null on low-contrast UI text (e.g. grey-on-white inactive tab labels) that Region.text() transcribes correctly. Both paths share the same TextRecognizer.optimize() pre-processing, but they diverge at the Tess4J call: getWords() filters by confidence to avoid false positives — and the legitimately-low confidence on faint glyphs gets rejected — while doOCR() keeps everything in the transcript. It is not, strictly, a bug. It is a precision-vs-recall tradeoff that calls for an architectural refinement, not a patch.

Steps to reproduce

1. Open https://coinmarketcap.com/ (or any modern UI with grey-on-white inactive tabs)
2. From a Jython script, run:
   region = Region(960, 0, 960, 600)
   m  = region.findText("Trending")    # → null
   tx = region.text()                  # → contains "Trending" in the transcript
3. Observe: the tab is clearly visible to a human and to read_text_in_region's transcription, but find_text refuses to localise it.

Expected behavior

Region.findText("Trending") should return a Match with bounding box coordinates for any text a human can read in the region — including low-contrast glyphs that Region.text() already transcribes correctly.

Actual behavior

findText returns null. The mismatch lives strictly between two Tess4J entry points called downstream of the identical optimize():

	Region.text()	Region.findText()
Inner path	OCR.readText()	Finder.doFindText() → OCR.readLines()
Tess4J call	doOCR() — full transcript	getWords() — bbox + per-word confidence
Confidence filter	none, keeps everything	rejects below internal threshold
PSM	SINGLE_BLOCK (6) forced	AUTO (3) default

The grey-on-white "Trending" glyph clears Tesseract LSTM's character recognition (it appears in the transcript), but its computed per-word confidence is in the 40–50 % range. getWords() discards it as a precaution against returning bounding boxes you would not want to click on — which is the right default for visual automation, where a false positive at the wrong coordinates costs more than a false negative.

Minimal reproducer (script)

from sikuli import *

# Open any web page with low-contrast tab labels (CoinMarketCap, Yahoo Finance, …).
# Pick a region containing such a tab. "Trending" on coinmarketcap.com is a perfect specimen.
region = Region(960, 0, 960, 600)

# Path 1 — findText (Finder.doFindText → OCR.readLines → Tess4J getWords)
match = region.findText("Trending")
print "findText:", match  # → null

# Path 2 — text (OCR.readText → Tess4J doOCR)
full = region.text()
print "text():", full     # → "Top  Trending  Watchlist  Stocks  Most Visited  ..."

# The word IS read by Tesseract. The word is NOT returned by findText.
# Both paths run through the same TextRecognizer.optimize() preprocessing.
# The split is at the Tess4J entry: getWords (confidence-filtered) vs doOCR (transcript).

Operating system

Windows 10 (reproduced on my side — but the bug lives in Finder.doFindText() / getWords() confidence filter, which is platform-independent Java + Tesseract code, so reproduction on SUSE, Ubuntu, macOS is expected; field confirmation welcome)

Java version

openjdk 25.0.2 2026-04-15

OculiX version / artifact

oculixide-3.0.4-complete-win.jar (built from claude/i18n-phase3)

Where does the bug happen?

API (Screen / Region / Pattern / Match), PaddleOCR / Tesseract

Logs / console output

# MCP find_text on the same region
> oculix_find_text("Trending", region={x:960,y:0,width:960,height:600})
{"found":false,"engine":"sikulix-region"}

# MCP read_text_in_region on a serrer scope
> oculix_read_text_in_region(region={x:960,y:240,width:960,height:60})
{"engine":"sikulix-region","text":"Top  Trending Watchlist Stocks Most Visited New  Gainers @ Rehypo >","lines":["Top  Trending Watchlist Stocks Most Visited New  Gainers @ Rehypo >"]}

# Cross-checked with Opus 4.8 on the same screen, same region, same workspace:
# identical "found:false" from find_text, identical successful transcript from read_text_in_region.
# Reproducibility is independent of the model wrapping the calls.

Additional context

The steppe analogy first, the code after

Asking find_text to localise "Trending" rendered in light grey on a white background is like asking the shepherd of the OculiX Pastoral Computing Suite™ to send his dog Aikash after Yak #42 in the steppe fog, three days of horseback from the WiFi router. The YAK MOOD PREDICTOR is 47 % confident "something bovine is grazing over there". The CHAMOIS DETECTION DRONE returns "biological presence detected, ungulate profile likely". But the shepherd refuses to release Aikash until SUSPICIOUS YAK ACTIVITY confirms the identity above 95 %. And he is right: sending Aikash to bite a parasite yak — possibly Yak #41 from neighbour Erlan, or worse, a dairy cow that wandered into the pasture by mistake — is a casus belli on the steppe, not a Ctrl+Z away. read_text_in_region, in this analogy, is only the observation drone: it produces the raw transcript "three ruminants down there, one likely yak, maybe two". Priceless for writing the daily report. Unusable for engaging.

Why this is not strictly a bug — and why an architectural refinement is the right answer anyway

The current behaviour is the correct default for the original SikuliX design assumption: a single OCR engine serving the visual-automation use case where the cost of a false positive (clicking at the wrong coordinates of a phantom word) is higher than the cost of a false negative (returning null and letting the caller retry or fall back). getWords()'s confidence filter is the implementation of that principle.

The opportunity surfaces when you observe that OculiX already knows the source of every image at the entry point — ScreenCapture, ADBScreen, VNCScreen are first-class types — and that the right preprocessing chain depends entirely on the source profile, not on a single one-size-fits-all optimize().

A draft chain dispatched by source profile

ScreenCapture                  (lossless, uniform illumination by construction)
  → Core.normalize(NORM_MINMAX)       (global level stretch → "Trending" grey 150 becomes a dark enough grey)
  → light unsharp
  → Tesseract
  // No CLAHE. No denoise. CLAHE corrects spatial illumination variation,
  // which a rendered UI does not have — it would solve a problem that
  // doesn't exist and pay in tile artefacts.

ADBScreen / VNCScreen          (lossy compression by construction)
  → Imgproc.medianBlur(3)               (kills JPEG ringing/blocking cheaply, edge-preserving)
  → CLAHE(clipLimit, tilesX/tilesY derived from region.h and expectedTextHeight)
  → unsharp
  → Tesseract
  // CLAHE earns its keep here: real local variance from compression artefacts.
  // The denoise comes FIRST, before any amplification step — you never
  // amplify then clean.

Future camera / mobile capture (real spatial illumination variation)
  → bilateralFilter (or fastNlMeansDenoising)
  → CLAHE
  → unsharp
  → Tesseract

Raw Mat with no provenance     (marginal case: Image.load(file))
  → blockiness metric on 8-pixel DCT grid  (cheaper than a full FFT,
                                             targeted at JPEG signature)
  → fallback on the ADB chain or the Screen chain depending on signature

The point that goes beyond a per-filter improvement: source-profile dispatch picks the entire chain, not a filter within a single chain. On the most common case — ScreenCapture of a rendered UI — CLAHE drops out of the path entirely, all its tuning evaporates with it, and the "Trending" issue is resolved by a one-line Core.normalize() with no tile artefacts and no parameters.

Public API: zero change

Region.findText(), Region.text(), OCR.readText(), OCR.readLines() keep their signatures. The dispatch happens in the TextRecognizer constructor, deciding the chain based on the concrete type of the source passed in. No migration for existing user scripts.

Three points open to discussion before I start

1. Where does SourceProfile live? Internal enum dispatched implicitly from the source's concrete type is the cleanest. But it makes it impossible for a user to override the chain on a stubborn case (someone who loads an Android screenshot as a generic Image and would benefit from the ADB chain). An optional explicit OCR.Options.sourceProfile(SourceProfile) opt-out preserves the override ability without poisoning the default. Worth the surface ?

2. Does ADBScreen really need its own chain? Argument for: the ADB pipeline is lossy by default (JPEG mode for performance). Argument against: a user can configure ADB in lossless PNG mode, in which case the ADB chain over-processes. Detection per-frame is expensive (see point 5 of the perf discussion below). Explicit flag on ADBScreen constructor, or autodetection from the first frame ?

3. Is medianBlur(3) always the right pre-CLAHE step on a lossy source? It excels on JPEG ringing/blocking. But certain VNC encoders (Tight ZRLE, Hextile RRE) produce different artefacts. I'd rather a fast median than a costly bilateral by default, but a cross-encoder mini-bench feels warranted before locking the chain.

A few opinions I've already cycled through (and discarded), for context

• A global confidence threshold knob exposed on OCR.Options — solves the symptom by lowering the bar globally, lets false positives back in. Patches the threshold, not the cause.
• Auto-retry with stepped confidence (70 → 50 → 30) — three Tesseract calls instead of one, masks the cause in latency. Wrong layer for the fix.
• Injecting the searched word into Tesseract's user_words dictionary — elegant in theory ("boost the confidence on the word I know I'm looking for"), but rebuilds the Tesseract config per call and is incompatible with findAllText use cases where the caller doesn't have a specific target.
• A CLAHE pre-processing added unconditionally in optimize() — my first proposal. Solves a problem that doesn't exist on the most frequent case, pays in tile artefacts, over-processes screenshots. Wrong tool by default.

The source-profile dispatched chain is the one shape that survives every counter-argument I've thrown at it.

Pinging the people whose hand on this matters most

@RaiMan — this is the kind of refinement that lives in direct lineage of the optimize() you originally designed. The "single engine, one preprocessing path" was the right call for the design pressures of the time. The question of whether to dispatch by source profile touches on your original architectural call about OCR being source-agnostic. I will not push anything without your read on it.

@adriancostin6 — if you want to benchmark in real conditions, I can rough out the four chains in parallel inside a measurement harness (Trending on web, ADB device capture, VNC remote desktop) and you supply the numbers. Cross-OS measurement with confidence intervals on Trending recall + clickable-coordinate accuracy is exactly your terrain, and would settle the open questions above more reliably than any opinion I have.

The gecko likes it when a bug, on careful look, turns out to be a refactor opportunity — which is rare, and which is what makes the craft interesting.

🦎

[adriancostin6]

@julienmerconsulting this really hits the nail on it's head.

story from back in the sikuli 2.0.5 days. findText was too unreliable in terms of matching so it was substituted with OCR.readText() in the upper layer to get more reliable matches.

happy to run the repro. just let me know what is needed.

[julienmerconsulting]

@adriancostin6 — sharp callout, you pinpoint the architectural hole exactly. readText reads everything but yields no coordinates, so the word is known to be there but cannot be engaged. That gap is precisely what the fix has to close.

The goal stated simply

Get findText to detect at the same level as readText, while keeping the bounding box it already returns. Not by lowering the confidence threshold (false-positive risk), not by adding a new tolerant method (API surface bloat), but by giving Tesseract a cleaner image before it computes its confidence — so the score on light-grey "Trending" climbs above the rejection threshold naturally.

In the analogy from the original post : Aikash finds Yak #42 in the steppe fog because we hand him better binoculars — not because we lower his identification standards. SUSPICIOUS YAK ACTIVITY stays at its 95 % bar, untouched. The dog just sees more clearly through the haze now. Same shepherd, same caution, sharper optics.

How

Source-profile dispatched preprocessing inside TextRecognizer.optimize() — the source type is already known at the entry point of every OCR call :

ScreenCapture        → Core.normalize() + light unsharp     → Tesseract
                       // global contrast stretch lifts the grey "Trending" without tile artefacts.
                       // A rendered UI has no spatial illumination variation, so CLAHE is the
                       // wrong tool here — it would solve a problem that doesn't exist and pay
                       // in tile-boundary artefacts.

ADBScreen/VNCScreen  → medianBlur(3) + CLAHE + unsharp      → Tesseract
                       // Denoise BEFORE amplification : you never amplify then clean.
                       // CLAHE earns its place here — real local variance from JPEG/RFB
                       // compression artefacts is exactly the use case it was designed for.

Future camera        → bilateralFilter + CLAHE + unsharp    → Tesseract
                       // Real spatial illumination variation — CLAHE is now mandatory.

Raw Image            → blockiness on 8-px DCT grid → fall back on the right chain
                       // For Image.load(file) cases where we lost the provenance.

Zero public API change. findText, text, readLines keep their signatures, their semantics, their precision-vs-recall guarantees. The dispatch happens in the TextRecognizer constructor based on the concrete source type passed in. The fix lives below the API, not next to it.

What I need from you

• Reproduce on your environments — first priority. SUSE Tumbleweed (you already have the VirtualBox + Cairo/FreeType setup that shifted the picture vs Windows on 🪟 [Bug] Modern Recorder: the "Choose image source" popup ghosts into the capture overlay #387, same kind of shift might appear here) and an Android screenshot via ADB if you can swing it. The CoinMarketCap reproducer in the issue runs in 30 seconds. If findText returns null on both while read_text_in_region reads "Trending" cleanly, we've confirmed the bug lives in the Java + Tesseract layer, not in OS-specific rendering — and the source-profile chain is the right fix at the right layer.

• One bonus data point : the actual confidence value Tesseract assigns to "Trending" on each OS. We don't expose it today, but a one-off debug Debug.log in Finder.doFindText would tell us where the natural baseline sits — Windows GDI vs SUSE Cairo/FreeType vs ADB. That number drives where the preprocessing target lives on each profile chain.

@RaiMan — this proposal touches the optimize() you originally designed at the foundation. The single-engine, source-agnostic preprocessing was the right call for the design pressures of the time. Splitting it by source profile is a real architectural shift, and I will not push it without your read on the principle. If you see a reason to keep optimize() source-agnostic that I'm missing, push back — the gecko listens before he digs.

🦎

[RaiMan]

@julienmerconsulting

As always: great stuff.

I am rather sure, that it was not me, who made optimize().

So I can happily say: do what is needed and is making things better.

[adriancostin6]

Repro ran on SUSE. Spot on, bug as described above, so it is confirmed ✅:

[DEBUG STARTUP] CLI -l parsed values: null
[DEBUG STARTUP] CLI -e present: false
[DEBUG STARTUP] IDE ready: on Java 25 ( 5.9 sec)
[DEBUG STARTUP] Jython ready: version 2.7.4  ( 9.6 sec)
──────── Run started @ 03:40:23 ────────
[error] Process failed: Cannot run program "xdotool": Exec failed, error: 2 (No such file or directory) 
[error] xdotool not available.
While you can use purely process based functionallity of the App class (e.g. open(), close(), isRunning()), you need to install xdotool to use window based stuff.
Error message: xdotool failed with code 1: 
[debug] TRACE: Screen: selectRegion
[debug] TRACE: Screen: userCapture
[debug] CapturePrompt: started at (9,191) on S(0)
[debug] CapturePrompt: finished at (645,242) on S(0)
[debug] TRACE: OverlayCapturePrompt: notifyObserver: true
[debug] Region: wait: waiting 0.0 secs for  to appear in R[9,191 637x52]@S(0)
[debug] Region: doFind: Switching to TextSearch
[debug] OCR: start: Tess4J 5.18.0 using bundled Tesseract (Legerix)
[debug] OCR: ready — engine=bundled Tesseract 5.5.0 (Legerix), lang=eng, available=chi_sim,eng,fra,hin,spa
[debug] Finder2: doFindText: Trending found: 1 times (1976 msec) 
[debug] Region: wait:  appeared (M[92,15 84x22]@S(0) S:0.01 C:134,26 [2188 msec])
findText: M[101,206 84x22]@S(0) S:0.01 C:143,217 [2188 msec]
[debug] Image: BufferedImage: (637, 52)
text(): Top Trending Watchlist Stocks Prediction Markets

[adriancostin6]

Added some logs for match confidence as well.. Since I was unsure where to add them exactly, I placed them in all the places where we add a match in Finder.doFindText(). Posting a patch below, as I did not make a commit, just so you can see what I did:

From d594b2fdd5b2680d7b446d18f48c5ea31fd12b32 Mon Sep 17 00:00:00 2001
From: Adrian Costin <adriancostin6@gmail.com>
Date: Sat, 4 Jul 2026 15:59:04 +0300
Subject: [PATCH] Finder2.doFindText: add debug logs for match confidence

---
 .../main/java/org/sikuli/script/Finder.java   | 34 ++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/API/src/main/java/org/sikuli/script/Finder.java b/API/src/main/java/org/sikuli/script/Finder.java
index 3fc23d80..f1abaef6 100755
--- a/API/src/main/java/org/sikuli/script/Finder.java
+++ b/API/src/main/java/org/sikuli/script/Finder.java
@@ -1091,6 +1091,14 @@ public class Finder implements Iterator<Match> {
                 Rectangle rword = new Rectangle(wordInLine.getRect());
                 rword.x += wordOrLine.x;
                 rword.y += wordOrLine.y;
+                Debug.log("doFindText: Adding match %s found in region [%d, %d, %d, %d], with a confidence score of %f", 
+                              wordInLine.getText(),
+                              rword.x,
+                              rword.y,
+                              rword.width,
+                              rword.height,
+                              wordInLine.getScore()
+                );
                 wordsMatch.add(new Match(rword, wordInLine.getScore(), wordInLine.getText(), where));
               }
             } else {
@@ -1120,10 +1128,26 @@ public class Finder implements Iterator<Match> {
                 rword.y += wordOrLine.y;
                 double score = (wordsInLine.get(startText).getScore() + wordsInLine.get(endText).getScore()) / 2;
                 String foundText = wordsInLine.get(startText).getText() + " ... " + wordsInLine.get(endText);
+                Debug.log("doFindText: Adding match %s found in region [%d, %d, %d, %d], with a confidence score of %f", 
+                              foundText,
+                              rword.x,
+                              rword.y,
+                              rword.width,
+                              rword.height,
+                              score
+                );
                 wordsMatch.add(new Match(rword, score, foundText, where));
               }
             }
           } else {
+            Debug.log("doFindText: Adding match %s found in region [%d, %d, %d, %d], with a confidence score of %f", 
+                          match.getText(),
+                          match.getRect().x,
+                          match.getRect().y,
+                          match.getRect().width,
+                          match.getRect().height,
+                          match.getScore()
+            );
             wordsMatch.add(new Match(match.getRect(), match.getScore(), match.getText(), where));
           }
         }
@@ -1141,6 +1165,14 @@ public class Finder implements Iterator<Match> {
         }
         for (Match match : wordsFound) {
           Rectangle wordOrLine = match.getRect();
+          Debug.log("doFindText: Adding match %s found in region [%d, %d, %d, %d], with a confidence score of %f", 
+                        match.getText(),
+                        match.getRect().x,
+                        match.getRect().y,
+                        match.getRect().width,
+                        match.getRect().height,
+                        match.getScore()
+          );
           wordsMatch.add(new Match(match.getRect(), match.getScore(), match.getText(), where));
         }
         findResult = new FindResult2(wordsMatch, fInput);
@@ -1746,4 +1778,4 @@ public class Finder implements Iterator<Match> {
     public void remove() {
     }
   }
-}
\ No newline at end of file
+}
-- 
2.54.0

After running the IDE repro below on the coinmarketcap.com site:

from sikuli import *

Debug.on(3)

# Open any web page with low-contrast tab labels (CoinMarketCap, Yahoo Finance, …).
# Pick a region containing such a tab. "Trending" on coinmarketcap.com is a perfect specimen.
region = selectRegion("Select")

# Path 1 — findText (Finder.doFindText → OCR.readLines → Tess4J getWords)
match = region.findText("Trending")
print "findText:", match  # → null

# Path 2 — text (OCR.readText → Tess4J doOCR)
full = region.text()
print "text():", full     # → "Top  Trending  Watchlist  Stocks  Most Visited  ..."

# The word IS read by Tesseract. The word is NOT returned by findText.
# Both paths run through the same TextRecognizer.optimize() preprocessing.
# The split is at the Tess4J entry: getWords (confidence-filtered) vs doOCR (transcript).

I get the following output:

[DEBUG STARTUP] CLI -l parsed values: null
[DEBUG STARTUP] CLI -e present: false
[DEBUG STARTUP] IDE ready: on Java 25 (15.2 sec)
[DEBUG STARTUP] Jython ready: version 2.7.4  (24.3 sec)
[info] ──────── Run started @ 15:57:12 ──── repro-ocr ────
[error] Process failed: Cannot run program "xdotool": Exec failed, error: 2 (No such file or directory) 
[error] xdotool not available.
While you can use purely process based functionallity of the App class (e.g. open(), close(), isRunning()), you need to install xdotool to use window based stuff.
Error message: xdotool failed with code 1: 
[debug] TRACE: Screen: selectRegion
[debug] TRACE: Screen: userCapture
[debug] CapturePrompt: started at (7,124) on S(0)
[debug] CapturePrompt: finished at (421,173) on S(0)
[debug] TRACE: OverlayCapturePrompt: notifyObserver: true
[debug] Region: wait: waiting 0.0 secs for  to appear in R[7,124 415x50]@S(0)
[debug] Region: doFind: Switching to TextSearch
[debug] OCR: start: Tess4J 5.18.0 using bundled Tesseract (Legerix)
[debug] OCR: ready — engine=bundled Tesseract 5.5.0 (Legerix), lang=eng, available=chi_sim,eng,fra,hin,spa
[debug] doFindText: Adding match Trending found in region [94, 8, 84, 22], with a confidence score of 0.641527
[debug] Finder2: doFindText: Trending found: 1 times (2899 msec) 
[debug] Region: wait:  appeared (M[94,8 84x22]@S(0) S:0.01 C:136,19 [3887 msec])
findText: M[101,132 84x22]@S(0) S:0.01 C:143,143 [3887 msec]
[debug] Image: BufferedImage: (415, 50)
text(): Top Trending Watchlist Stocks

Seems to find a match in both cases, but perhaps something changed and it was fixed in the meantime.