HDR displays: icon/name scanning fails (capture is SDR-only via GDI CopyFromScreen)

[DysektAI]

Summary

On displays with HDR enabled, item scanning does not work reliably — the capture path is SDR-only, so icon scans (and name scans) read an SDR-tonemapped/clipped view of an HDR desktop and fail to match. The only existing "handling" is an upstream FAQ note telling users to disable HDR (FAQ.md, commit e98a276). There is no code-level HDR support today.

Impact

• Item icon scanning fails/misidentifies when Windows HDR is on.
• Forces HDR users to disable HDR to use RatScanner.
• One user also reported the screen flashing during a scan attempt, as if HDR was being disabled/re-enabled (unconfirmed; another user reports no such effect). Needs investigation during local testing.

Root cause (current capture path)

Both name scans and icon scans funnel through a single GDI+ capture in src/App/RatScannerMain.cs:

// Returns the ruff screenshot
private static Bitmap GetScreenshot(Vector2 vector2, Size size)
{
    Bitmap bmp = new(size.Width, size.Height, PixelFormat.Format24bppRgb);

    try
    {
        using Graphics gfx = Graphics.FromImage(bmp);
        gfx.CopyFromScreen(vector2.X, vector2.Y, 0, 0, size, CopyPixelOperation.SourceCopy);
    }
    catch (Exception e)
    {
        Logger.LogWarning("Unable to capture screenshot", e);
    }

    return bmp;
}

Graphics.CopyFromScreen is legacy GDI+ capture. When HDR is enabled, Windows composes the desktop in FP16 linear scRGB (1.0 = 80 nits, values go far above that); GDI-based capture reads an 8-bit clipped SDR view of that buffer, so highlights blow out and colors are washed out/dimmed. The resulting Format24bppRgb bitmap is exactly what the SDR pipeline would see — there is no HDR capture, no tone mapping, no HDR-mode detection anywhere.

Known solutions to investigate (ShareX and others)

ShareX (and most GDI-based tools) had the same problem. The fix approaches proven in the ShareX ecosystem:

• DXGI Desktop Duplication with R16G16B16A16_FLOAT — captures the full HDR signal, then tone maps to SDR with the BT.2390 EETF evaluated in the PQ domain, anchored to the monitor's SDR reference white ("SDR content brightness" slider) so UI/text keep exact brightness and only highlights compress.
• Windows.Graphics.Capture (WinRT) also supports HDR capture and is the modern API.
• Reference implementations: shuakami/ShareX-HDR (FP16 DXGI capture + BT.2390 tone mapping; per-display handling for mixed HDR/SDR multi-monitor setups; falls back to GDI capture if the path fails). Stock ShareX PRs/discussions on HDR capture are also worth mining.
• HDR-mode detection: DXGI IDXGIOutput6::GetDesc1 (ColorSpace, MaxLuminance) or DISPLAYCONFIG SDR white level — only engage the HDR capture path when a display is actually in HDR mode, so SDR users keep the current fast path untouched.

The scan pipeline (RatEye → OpenCvSharp/Tesseract) can stay unchanged if the HDR capture path tone-maps to an SDR 8-bit bitmap before handing off — the fix can be contained in the capture layer.

Plan

1. Test locally on an HDR setup: reproduce icon-scan failure, check the flashing symptom (HDR toggling) and whether it's capture-related.
2. Evaluate DXGI Desktop Duplication / Windows.Graphics.Capture + tone mapping as a drop-in replacement for GetScreenshot when HDR is detected.
3. Keep SDR behavior byte-for-byte identical (detection-gated, with GDI fallback).
4. Verify accuracy on icon + name scans in HDR and SDR, and no display-mode flapping.

Acceptance criteria

• Icon scanning works with HDR enabled at accuracy comparable to SDR.
• No screen flashing / display-mode switching during scans.
• SDR users see zero behavior change; mixed HDR/SDR multi-monitor setups handled per display.

[DysektAI]

Investigation + implementation status

What was verified on this machine (RTX 4090 + 3 displays, incl. an HDR-capable Samsung G7)

• Capture choke point confirmed: both name scans and icon scans funnel through RatScannerMain.GetScreenshot (Graphics.CopyFromScreen → Format24bppRgb).
• HDR detection works via IDXGIOutput6::GetDesc1 (ColorSpace, BitsPerColor, MaxLuminance): currently all outputs are SDR (RgbFullG22NoneP709), DISPLAY1 reports bpc=10 + maxLum=1499. The DISPLAYCONFIG advanced-color query crashes with a hard CLR error when its struct layouts are wrong (flattened DISPLAYCONFIG_PATH_TARGET_INFO/undersized DISPLAYCONFIG_MODE_INFO corrupt the heap) — corrected to exact winuser.h layouts, degrades gracefully when unavailable.
• Desktop Duplication is viable: AcquireNextFrame → R16G16B16A16_Float frame on the 10-bit output (even in SDR — the DWM always composites in FP16 scRGB), B8G8R8A8_UNorm on the 8-bit sinks; staging + CopySubresourceRegion + Map readback works.

Implemented (working tree, uncommitted)

• src/App/Display/HdrScreenCapture.cs — session-cached per-output DD capture: IsHdrCaptureRequired(Rectangle) (GetDesc1 color space + DISPLAYCONFIG advanced color + SDR white level, 2 s TTL), CaptureRectangle with region-only staging, first-frame guard (LastPresentTime==0), AccessLost/WaitTimeout handling, failure-soft (null → GDI fallback).
• src/App/Display/HdrToneMapper.cs — BT.2390-4 EETF (hermite roll-off in PQ/ST.2084 domain), luminance-preserving RGB rescale + soft gamut mapping, ordered dithering, sRGB encode LUT, content-peak histogram, SDR fast path.
• Wiring — GetScreenshot callers route through CaptureRegionHdrAware (detection-gated; SDR/errors keep GDI byte-for-byte). Config escape hatch RatConfig.HdrCapture.Enable.
• 17 new unit tests (PQ round-trip, sRGB encoding, EETF monotonicity/identity/bounds, full-region conversion, HDR color-space detection). Full suite: 385/385 green, build 0 warnings.
• Deps: Vortice.Direct3D11 + Vortice.DXGI 3.6.2 (App only).

Live end-to-end proof on this SDR desktop

DD capture through the production CaptureRectangle compared against GDI on a 900×500 UI region: 1,350,000/1,350,000 pixels identical (0 mean abs diff, 0 max diff) — the FP16 fast-path conversion is byte-exact with GDI on SDR content.

Remaining (needs hardware action)

The true HDR path (tone-map roll-off above SDR white) cannot be exercised until a display is actually in HDR mode. Next step: enable Windows HDR on the G7 (Settings → System → Display → Use HDR) and run icon/name scans + a capture comparison. If you can flip that on, I can validate and tune the curve (SDR-white anchor vs. highlight roll-off) against real HDR captures. The HdrCapture.Enable config flag is the fallback if tone mapping misbehaves on specific hardware.

[DysektAI]

Live HDR testing done (no game needed) — findings revise the plan

The user enabled HDR on DISPLAY1 (GetDesc1: RgbFullG2084NoneP2020, bpc=10). We validated with known test content on screen (solid-color borderless windows, an HDR swapchain patch):

What works

• Detection gate: IsHdrCaptureRequired correctly flips to true with HDR on.
• DD capture pipeline: AcquireNextFrame → staging → Map readback works in HDR mode; the surface alternates between R16G16B16A16_Float and B8G8R8A8_UNorm between sessions (DWM-dependent). The App handles both.

What the measurements show

Grey-ramp windows (0/64/128/192/255) captured simultaneously via GDI and DD:

Window	GDI	Duplication surface
0	0	0
64	64	150
128	128	255
192	192	255
255	255	255

1. GDI is correct for SDR content in HDR mode — the original premise that GDI mangles all SDR content does not hold on this setup. GDI's failure mode is only clipping content above SDR white (highlights), not mis-encoding SDR content.
2. The duplication surface delivers SDR content with a boosted, unestablished transfer (64-grey → 150, 128-grey → 255). My earlier "mean lum 71 vs 27" desktop reading was this boost, not a washout fix.

Consequence for the implementation

Shipping the DD path on-by-default would make SDR-content captures (exactly what icon/name scans consume) brighter than GDI — worse, not better. The DD + BT.2390 architecture is sound, but it needs the duplication's HDR-mode transfer pinned down (un-boost step) and validation against real in-game HDR content.

Action taken: RatConfig.HdrCapture.Enable now defaults to off (config-file opt-in), so HDR users get the proven GDI path by default. The DD code, tone mapper, detection, and 17 unit tests remain in place for the follow-up.

What would settle it (any order)

• A real EFT HDR session: does icon/name scanning work today with GDI + HDR on? (Very possibly yes — SDR UI content reads correctly.) If it works, the practical fix may be much smaller than a capture rewrite.
• The duplication's exact HDR transfer (FP16 scRGB semantics under HDR) — needs a machine where the surface delivers FP16 while HDR is on.