Skip to content

Implement backend of search across all columns - #1463

Open
jmcphers wants to merge 7 commits into
mainfrom
feature/data-explorer-search
Open

jmcphers wants to merge 7 commits into
mainfrom
feature/data-explorer-search

Conversation

@jmcphers

@jmcphers jmcphers commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds R backend support for searching across all columns in the Data Explorer (posit-dev/positron#3302). The frontend, protocol, Python, and DuckDB parts are in posit-dev/positron#17120; this PR is the ark half.

Approach

The search is sent as a new row filter type, global_search, that has no column. Because it goes through the existing set_row_filters call, it gets these behaviors without new plumbing:

  • It stacks with column filters.
  • Summary stats reflect it, since profiles already use the filtered rows.
  • It is re-applied when the data changes.
  • Older frontends and backends are unaffected. The frontend only sends global_search to backends that list it in supported_types.

To support this, column_schema on RowFilter is now optional. Code that reads it now unwraps it. Filters with no column are skipped when matching columns after a schema change, and are dropped by Convert to Code (a search is never turned into dplyr code).

The search always runs after the column filters, so it only scans rows that passed them. This follows RStudio's data viewer, which also inspired portions of the implementation.

Matching rules

A row matches when any searched column matches:

  • Numeric columns match numeric terms by implied precision. A term matches values that truncate (toward zero) to it at the term's number of decimals. So 3 matches 3.7, 3.1 matches 3.14159 but not 3, and -3 matches -3.5. Grouped commas (1,234), .5, and signs are accepted. Non-numeric terms like 1e3 or Inf never match numeric columns. Numeric columns are compared as numbers, not converted to strings, which keeps them fast.
  • Character and factor columns match a case-insensitive substring.
  • Other classed columns (dates, times, difftimes, etc.) match a case-insensitive substring of their displayed text, using the same format() path that the grid uses. This means 3 days matches days, unlike RStudio, which searches as.character().
  • Missing values match when the term is a null marker (NA, null, NaN, None, NaT, <NA>).
  • Character row names are searched too.
  • Plain list columns and nested data frames are skipped.
  • A column that fails to search (for example, a format() method that errors) matches nothing instead of failing the whole search.

The term is marked UTF-8 and regex-escaped before matching.

Large tables

The filter has an optional max_cells limit. When the candidate rows times the searched columns go over it, only the leading rows are scanned and the backend reports global_search_rows_scanned in its state. We do this to limit the amount of time spent searching, which would otherwise be unbounded (and it blocks the kernel).

Formatting dates and other classed columns is the slowest part of a search, so the formatted text is cached per column while the table and scanned rows stay the same. The cache is cleared when the table changes, when the scanned rows change, or when the search is cleared. Nothing is cached for unlimited searches, so memory use stays bounded.

Cell matches

get_data_values now returns an optional search_matches array next to columns, with one flag per cell saying whether it matched. The frontend uses this to highlight matched cells. The frontend can't work this out from the displayed text alone, because a numeric match like 3.141 on a cell shown as 3.14, or a match past the truncation point, would be missed.

Other changes

  • data_explorer_comm.rs is regenerated from the updated protocol. It also picks up fields from earlier protocol changes that ark hadn't regenerated yet (available_sheets, total_num_rows, sheet_name). The clippy large_enum_variant expectation no longer fires, so it was removed.

Tests

New integration tests in tests/integration/data_explorer.rs cover column types, numeric precision, data frames/tibbles/data.tables/matrices, stacking with column filters, max_cells truncation, per-cell search_matches, re-applying on data updates, and Convert to Code ignoring the search since it isn't representable in R.

@jmcphers
jmcphers requested a review from lionel- November 4, 2026 18:20
@jmcphers jmcphers changed the title Feature/data explorer search Implement backend of search across all columns Nov 4, 2026
@jmcphers
jmcphers requested review from jennybc and removed request for lionel- November 4, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant