Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds R backend support for searching across all columns in the Data Explorer (posit-dev/positron#3302). The frontend, protocol, Python, and DuckDB parts are in posit-dev/positron#17120; this PR is the ark half.
Approach
The search is sent as a new row filter type,
global_search, that has no column. Because it goes through the existingset_row_filterscall, it gets these behaviors without new plumbing:global_searchto backends that list it insupported_types.To support this,
column_schemaonRowFilteris now optional. Code that reads it now unwraps it. Filters with no column are skipped when matching columns after a schema change, and are dropped by Convert to Code (a search is never turned into dplyr code).The search always runs after the column filters, so it only scans rows that passed them. This follows RStudio's data viewer, which also inspired portions of the implementation.
Matching rules
A row matches when any searched column matches:
3matches3.7,3.1matches3.14159but not3, and-3matches-3.5. Grouped commas (1,234),.5, and signs are accepted. Non-numeric terms like1e3orInfnever match numeric columns. Numeric columns are compared as numbers, not converted to strings, which keeps them fast.format()path that the grid uses. This means3 daysmatchesdays, unlike RStudio, which searchesas.character().NA,null,NaN,None,NaT,<NA>).format()method that errors) matches nothing instead of failing the whole search.The term is marked UTF-8 and regex-escaped before matching.
Large tables
The filter has an optional
max_cellslimit. When the candidate rows times the searched columns go over it, only the leading rows are scanned and the backend reportsglobal_search_rows_scannedin its state. We do this to limit the amount of time spent searching, which would otherwise be unbounded (and it blocks the kernel).Formatting dates and other classed columns is the slowest part of a search, so the formatted text is cached per column while the table and scanned rows stay the same. The cache is cleared when the table changes, when the scanned rows change, or when the search is cleared. Nothing is cached for unlimited searches, so memory use stays bounded.
Cell matches
get_data_valuesnow returns an optionalsearch_matchesarray next tocolumns, with one flag per cell saying whether it matched. The frontend uses this to highlight matched cells. The frontend can't work this out from the displayed text alone, because a numeric match like3.141on a cell shown as3.14, or a match past the truncation point, would be missed.Other changes
data_explorer_comm.rsis regenerated from the updated protocol. It also picks up fields from earlier protocol changes that ark hadn't regenerated yet (available_sheets,total_num_rows,sheet_name). The clippylarge_enum_variantexpectation no longer fires, so it was removed.Tests
New integration tests in
tests/integration/data_explorer.rscover column types, numeric precision, data frames/tibbles/data.tables/matrices, stacking with column filters,max_cellstruncation, per-cellsearch_matches, re-applying on data updates, and Convert to Code ignoring the search since it isn't representable in R.