Skip to content

perception: keep OCR within the focused window - #44

Closed
charan-rathore wants to merge 1 commit into
awlevin:mainfrom
charan-rathore:fix/window-only-ocr
Closed

charan-rathore wants to merge 1 commit into
awlevin:mainfrom
charan-rathore:fix/window-only-ocr

Conversation

@charan-rathore

Copy link
Copy Markdown
Contributor

Summary

Fixes #35. ocr_region joined a low window to the menu bar at the top of the display, then OCRed the entire rectangle between them. On a window over another app, unrelated background text entered the classifier's screen read and made OCR expensive. The crop now stays within the frontmost window plus its existing margin, clamped to the display. With no usable window, it keeps the full-display fallback.

This deliberately uses one crop rather than changing the OCR cache to manage two regions. The reporter measured 46.4 seconds for the joined region and 0.33 seconds for the window-only region on their 5K Finder repro. Those are reporter measurements, not a live replay by this PR.

Checks

  • uv run --no-sync ruff check .: passed.
  • uv run --no-sync ruff format --check .: passed.
  • uv run --no-sync pytest -q: 612 passed, 3 skipped, 1 xfailed on Linux.
  • Fake-screen tests cover the low-window crop and OCR box offsets, a window at the top, offscreen edges, no window, and a degenerate window. Existing AX tests show that labelled menu-bar controls remain in the accessibility walk. PR macos: walk the focused window and menu bar #42 also preserves the menu-bar root when it scopes the macOS walk to the front window.

Caveat

Text visible only in the menu bar that is not available through accessibility will no longer be OCRed. There was no live Finder/OCR replay here; the focused tests use fake captures and a fake OCR reader.

Instinct assisted.

@charan-rathore

Copy link
Copy Markdown
Contributor Author

Closing this as superseded. Main now deliberately keeps the menu bar strip in the OCR region for a focused window (see the ocr_region tests in tests/test_ocr_cache.py on main), and this PR removes that strip, so it would work against the current design. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OCR region reads everything between the menu bar and a window that starts low on the screen

1 participant