Skip to content

Add element access, iteration, values, and JSON pointers to json_view - #5622

Merged
nlohmann merged 22 commits into
json-view/08-view-builderfrom
json-view/11-view-access
Oct 21, 2026
Merged

nlohmann merged 22 commits into
json-view/08-view-builderfrom
json-view/11-view-access

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Part of the stack for the zero-copy view (#5295). This adds element and member access to basic_json_view, and the value conversions built on it.

Summary

  • basic_json_view gains the read-only access functions of basic_json: operator[] and at() with keys, indices, and JSON pointers, front() and back(), find(), contains(), and count(), begin()/end() and cbegin()/cend(), items() with structured bindings from C++17 on, and type_name().
  • Index types. operator[] and at() accept an index of any integer type (int, unsigned, long, std::int64_t, ...) through a template; a single int overload would have been ambiguous for unsigned, long, and int64_t. Negative values are out of range.
  • Exceptions: the ids and messages of the const functions of basic_json. Where basic_json has undefined behavior, the view answers safely: operator[] with a missing key or an index out of range returns a discarded view; front()/back() of an empty container throw invalid_iterator.214.
  • Chained access is safe. operator[] (key, index, JSON pointer) on a discarded view returns a discarded view, so v["a"]["b"][0] can be tested once at the end. at() still throws.
  • Order: objects are iterated in document order, and all members are visited.
  • Duplicate keys (please review): lookups return the first member, with early exit, so a lookup can stop at the first match; yyjson and simdjson do the same. parse() keeps the last value, and so do materialize() and the conversions to maps. This is documented prominently in a "Duplicate keys" section of the feature page, with a recipe for getting the last member. Returning the last member would mean scanning every object to its end on each lookup, which made lookups in small objects 1.6 to 3.4 times slower.
  • Lookup cost: keys of up to 16 bytes are compared with two overlapping loads, and most keys are rejected by their length alone, which the index already holds.
  • Array access: operator[](index) is linear in the index, because elements are not fixed-size in the flat index. The docs say so.
  • get<T>(), get_to(): convert directly, without building a basic_json value: arithmetic types (booleans become 0/1, as in basic_json); bool and std::nullptr_t; strings (std::basic_string as a copy, string_view_t without a copy); BasicJsonType (the same as materialize()); views themselves; std::vector, and maps with string keys, e.g. std::map<std::string, json_view>. Everything else, including user types with from_json(), goes through materialize() of that subtree. Floats are bit-identical to parse().
  • get_string(): the string without a copy, pointing into the source text or, for strings with escapes, into the document's buffer of decoded strings.
  • number_token(): the number as it appears in the source ("1.50", "1E2", all digits of a long integer) -- basic_json cannot provide this.
  • value(): with keys and with JSON pointers; the pointer overload also accepts arrays, as since Extend value to arrays when using JSON pointers #5223.
  • JSON pointers: operator[], at(), and contains(). The exceptions are those of basic_json, message included; where const basic_json::operator[] has undefined behavior, the result is a discarded view; value() returns the default wherever basic_json::value() catches out_of_range; contains() never throws; array indices follow json_pointer's rules (parse_error.106/109, out_of_range.404/410).
  • Core files touched: json.hpp, json_pointer.hpp, and value_t.hpp gain two small internal helpers. detail::json_pointer_access gives code outside basic_json the reference tokens of a pointer, so they need not be parsed from to_string() again. detail::value_type_name(value_t) holds the names that basic_json::type_name() used to switch over; type_name() now calls it, so the view uses the same names (no behavior change).
  • Other: NLOHMANN_VIEW_ALWAYS_INLINE is plain inline on MSVC, because __forceinline triggers C4714 (an error under /WX) for templates MSVC cannot inline. IWYU pragmas are added. Docs: API pages and examples for every new function, the "Duplicate keys" and "Getting values out without copying" sections, and fixes of the operator[] and is_discarded pages.

Performance

Measured on Apple M1 Max (Apple clang, -O2) and an x86-64 KVM Haswell VPS pinned to one core (GCC 13 / Clang 18, -O2; the VPS is noisy, about ±5-10%). µs, best of 3 interleaved rounds of 15 runs; files from nativejson-benchmark. Traversing every value (numbers read as double, strings copied) at this PR:

Apple M1 x86-64 GCC x86-64 Clang
canada 2074 4296 4331
citm_catalog 128 261 265
twitter 107 149 171

get<T>() of arithmetic types is inlined down to the conversion, so its checks of the node kind merge with those of the caller, and reading an integer needs no call.

The core library is unaffected: the three core files only gain the helpers above, and basic_json::type_name() returns the same strings.

Tests

  • Every element and member of 2,000 generated documents, compared with ordered_json.
  • Keys of every length around the load sizes.
  • Duplicate keys: lookups find the first member, iteration visits all.
  • The exceptions, compared with const basic_json.
  • Chained access on discarded views, and every integer type as an array index.
  • Iterators, items, and structured bindings.
  • The conversions of 2,000 generated documents.
  • 20,000 float tokens, compared bit for bit with parse() for double and for float.
  • Every JSON pointer (leaves and their parents) of 1,000 documents.
  • 27 malformed pointers, with the exceptions of at(), operator[], contains(), and value() compared with basic_json, also with JSON_DIAGNOSTICS and JSON_DIAGNOSTIC_POSITIONS.

Public API

No breaking changes to the existing API: this PR adds members of basic_json_view (access functions and value conversions), plus the internal detail::json_pointer_access and detail::value_type_name(). basic_json::type_name() behaves as before. The view API is new in this stack, so its decisions are not breaking relative to a release, but they are visible to users:

  • operator[] and at() accept any integer type as an index.
  • operator[] on a missing key, an out-of-range index, an unresolvable pointer, or a discarded view returns a discarded view instead of invoking undefined behavior as const basic_json::operator[] does; at() throws.
  • Lookups with duplicate keys return the first member; parse(), materialize(), and map conversions keep the last.
  • Exceptions thrown by the view's access functions never carry the JSON_DIAGNOSTICS path.

Written by Claude Code.

🤖 Generated with Claude Code

@nlohmann
nlohmann added this pull request to stack #5636 September 29, 2026 14:20
@nlohmann nlohmann changed the title json view/11 view access Add element access, lookup, and iteration to json_view Sep 29, 2026
@nlohmann

Copy link
Copy Markdown
Owner Author

CI fixes for this PR (commit 958fbc3, plus the lower fixes merged in):

Verified with gcc:16 (linux/amd64, CI flags, C++11/C++20) and with a JSON_NOEXCEPTION build run as --no-throw.

— posted by Claude Code on behalf of @nlohmann

@github-actions

Copy link
Copy Markdown

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel is out of date.

@nlohmann
nlohmann removed this pull request from stack #5636 September 30, 2026 13:19
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch from 29c684f to 4d99fe9 Compare September 30, 2026 13:20
@nlohmann
nlohmann added this pull request to stack #5739 September 30, 2026 13:21
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch 3 times, most recently from 5dce18c to a8beec0 Compare September 30, 2026 18:06
@nlohmann nlohmann added the review needed It would be great if someone could review the proposed changes. label Sep 30, 2026
@nlohmann
nlohmann marked this pull request as ready for review September 30, 2026 18:17
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch from a8beec0 to e17e23a Compare September 30, 2026 19:15
Comment thread include/nlohmann/detail/view/lookup.hpp Dismissed
Comment thread single_include/nlohmann/json_view.hpp Dismissed
@nlohmann
nlohmann removed this pull request from stack #5739 October 6, 2026 09:26
@nlohmann
nlohmann changed the base branch from json-view/10-view-document to json-view/08-view-builder October 6, 2026 09:26
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch from 5286c28 to 0adb778 Compare October 6, 2026 09:28
@nlohmann nlohmann changed the title Add element access, lookup, and iteration to json_view Add element access, iteration, values, and JSON pointers to json_view Oct 6, 2026
@nlohmann
nlohmann added this pull request to stack #5768 October 6, 2026 09:29
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch from 0adb778 to 77acd45 Compare October 7, 2026 14:44
* Lookups (operator[], at, find, contains, count, value, JSON pointers)
  return the last member of a duplicate key, as materialize() and
  parse() keep it.
* operator[] on a discarded view returns a discarded view instead of
  throwing, so v["a"]["b"] is safe for a missing "a".
* operator[] and at() take any integer type (not only int and size_t),
  fixing ambiguous calls with unsigned, long, std::int64_t, ...
* Fix the operator[] documentation, which claimed a discarded view for
  a type mismatch where type_error.305 is thrown.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…d its tests

A view is tested with is_discarded(); the lookups in at(), value() and
contains(json_pointer) and the unit tests no longer rely on the explicit
conversion to bool.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…mporary

root() of an rvalue document is deleted, because the views would dangle.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…xamples

The explicit conversion to bool of basic_json_view is gone.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Looking up the last member of a duplicate key cannot stop at a match, so
every lookup scanned the whole object (1.6 to 3.4 times slower for small
objects). operator[](key), at, find, value, contains, count, and JSON
pointer resolution return the first member again, as yyjson and simdjson
do; materialize() and get<map>() keep the last value, as parse().

The documentation says so in the feature page and on each lookup page,
and explains how to get the value parse() would give. The integer index
templates and the discarded chaining of operator[] stay.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
__forceinline makes MSVC report C4714 for function templates it cannot
inline (get<std::string>), an error under /WX. The forced inlining is
only a performance hint, so use plain inline there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Mark the includes that are needed by design (json.hpp, the macro
unscope, <tuple> for the tuple_size specialization), take size_t from
<cstring>, and drop the forward declaration of basic_json_document that
the friend declaration makes redundant.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.

- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
  json_view.hpp only includes json.hpp, so including them here would duplicate
  their definitions

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The same changes as on the dump branch, where they were first made, so
that this branch passes clang-tidy on its own.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
… view

Develop's #5800 made basic_json report a JSON pointer array index such as
"/a" as parse_error.109 instead of out_of_range.404, as longer ones were;
the view's array_index() follows, with "-" and the empty token still
unresolved.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
From gregmarr's review of #5622:

- count() counts every member with the key, so count(key) > 1 detects
  duplicate keys; it was contains() as 0/1 before.
- get_to() passes types that get<T>() does not convert directly to
  from_json with the existing value, as basic_json::get_to() does, so that
  a from_json which sets only the fields present keeps the others.
- array_index() checks the empty token and "-" first; the results are
  unchanged.
- Docs: no "if an exception is thrown" where a view never changes
  anything; the borrowed source must also stay unmodified; items() on a
  temporary view is safe; back()/front() notes shortened; the basic_json
  see-also entries say how the view's results can differ.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The conditional added 1 or 0 as int to a std::size_t, which GCC rejects
with -Werror=sign-conversion (ci_test_standards_gcc).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
From gregmarr's review of #5622:

- begin()/end(): "a null value or a discarded view iterates as an empty
  range", singular like the primitive case before it.
- The class page's list of function groups has no trailing commas.
- document_data::key_equals() compares a key node with a string, for the
  key comparisons that were spelled out in each lookup.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With a ! or && in the condition, astyle spaces the whole template
header (template < ... >); detail::negation and a named trait keep it
compact, as the library writes such conditions elsewhere. From gregmarr's
review of #5630.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
include-what-you-use asks json_view.hpp to include
detail/meta/type_traits.hpp for it, which the single header must not do
(the amalgamation would duplicate json's internals). The two conditions of
value() go through a trait in value.hpp, like the others.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the json-view/11-view-access branch from 1c8c13a to bf5a42c Compare October 20, 2026 06:58
@nlohmann nlohmann removed the review needed It would be great if someone could review the proposed changes. label Oct 20, 2026
@nlohmann nlohmann added this to the Release 3.13.0 milestone Oct 20, 2026
@nlohmann
nlohmann merged commit 43e97b6 into develop Oct 21, 2026
169 of 331 checks passed
@nlohmann
nlohmann deleted the json-view/11-view-access branch October 21, 2026 08:21
nlohmann added a commit that referenced this pull request Oct 21, 2026
* Scan json_view strings with SIMD and index large objects

Speed up json_view's parser with SIMD scanning and a hash table
for large objects.

Long runs of string bytes are scanned 16 bytes at a time with NEON
(AArch64, GCC and Clang) and SSE2 (x86-64), both baseline
instruction sets. Keys keep 16 table checks before the vector
loop, because their lengths repeat from record to record; string
values get 8, because their lengths vary more. Non-ASCII text is
validated 16 bytes at a time with simdjson's "lookup4" check
(Keiser and Lemire, 2021), with NEON on AArch64 and, on x86-64,
with SSSE3. SSSE3 is not part of baseline x86-64, so the check is
compiled for SSSE3 with a function attribute and used only where
CPUID reports it, which all x86-64 CPUs since about 2011 do; the
answer is cached in a statically initialized atomic, so there is
no guard of a local static and no global constructor. The same
input is accepted either way. JSON_VIEW_NO_SIMD selects the
portable code.

On x86-64, string runs are now checked vector-first: one SSE2
compare from the first byte finds the end of most keys and short
values, instead of a branch per byte for the first 8-16 bytes.
AArch64 keeps the byte-wise steps, where a NEON mask costs more and
the branches predict well. Entering an object or array no longer
stalls: open() stores the parent's frame field by field instead of
building it on the stack and reading it back with wider loads,
which waited for the narrower stores to retire.

Objects with 128 members or more get an open-addressing hash table
built when the object closes, so operator[], at(), find(),
contains(), count(), value(), and JSON pointers take constant time
on average in such objects; of duplicate keys, the first is kept,
as for the linear search. The idea comes from Boost.JSON.

simdjson is credited in simd.hpp's SPDX block, the README, and
license.md.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC module build and MSVC C4127 in the json_view SIMD code

GCC ignores the target attribute in modules, so the SSSE3 dispatch is
disabled for the module interface (the check stays portable, SSE2 is kept).
Make the 8-vs-16 byte unrolling condition in scan_string_run a
preprocessor/template split to avoid a constant condition (C4127).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Remove an accidentally committed GCC module cache

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bound the probe length of the large-object hash index

The key hash is not seeded, so keys chosen to collide made building the table quadratic (20,000 colliding keys took 470 ms to parse). A key may now sit at most 64 slots from its home slot; if a key would sit further away, the table is dropped and the object is searched linearly. Lookups stop after the same distance.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Release the spare capacity of the hash index in shrink_to_fit

shrink_to_fit() now trims the tables of large objects like the node array and the decoded strings, and the list of large objects is released as soon as the tables are built.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use the portable string scan on non-x86 targets that define __SSE2__

WebAssembly with -msse2 defines __SSE2__, but has no <cpuid.h> or x86 intrinsics headers.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Document the hash index number in node::extra

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the last of duplicate keys in the object hash index

Lookups in objects with 128 members or more now return the last member of
a repeated key, like the linear search of smaller objects and like
materialize() and basic_json::parse(). build_object_index let the first
occurrence win, so the same text gave different results depending on the
size of the object.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the first of duplicate keys in the object hash index again

Lookups in objects with 128 members or more return the first member of a
repeated key again, like the linear search of smaller objects. This undoes
the code change of a695420; its test now expects the first member from
lookups (with and without a table) and the last value from materialize()
and basic_json::parse().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy 22 findings in the json_view headers

Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast
would trip -Wcast-align=strict), use auto for a cast initialiser, add
parentheses to a mixed expression, and name bugprone-std-namespace-modification
in the NOLINTs of the tuple_size/tuple_element specialisations.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy 22 findings in unit-json_view.cpp and unit-json_view_builder.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Avoid GCC useless-cast and strict-overflow warnings in unit-json_view.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Index strings with size_t in the colliding-keys test (MSVC C4244 on Win32)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make json_view.hpp pass include-what-you-use

ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.

- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
  json_view.hpp only includes json.hpp, so including them here would duplicate
  their definitions

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move the dump, comparison, and large-object tests to unit-json_view_dump.cpp

The dump and comparison tests moved to their own file in the dump pull
request; the hash index tests of this branch join them there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Follow the renames from the review of the view builder

The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename slots in the object index for Qt compatibility

Qt defines slots as a macro, so a Qt header included before
json_view.hpp broke the declarations of the local variable (see #5865).
The test variable is renamed as well.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address the review of the json_view object index and SIMD scan

From gregmarr's review of #5628:

- Docs and comments say that the hash index is one of the object's keys.
- object_index.hpp: conditional_static_cast for the hash, "slot" for the
  table position, no flag for a degenerate table, and a static_assert
  that the smallest indexed object is larger than the probe window.
- scan.hpp unrolls with NLOHMANN_VIEW_REPEAT_FIRST_8/SECOND_8.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Encode JSON_VIEW_NO_SIMD and JSON_VIEW_USE_SSSE3 in the ABI namespace

Both macros select between definitions of the same inline functions, so
translation units that disagree on them violated the ODR. Like the other
ABI-affecting macros, they now add a tag to the inline namespace (_vns and
_vssse3), and the docs replace the "Define consistently" warning with the
usual ABI compatibility note. From gregmarr's review of #5628.

nlohmann_json.natvis covers twice as many tags now, so it lists four times
as many namespaces (14.7 MB instead of 3.4 MB).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not include type_traits.hpp directly in the object index

json.hpp, which object_index.hpp already includes, provides it; included
directly, the json_view amalgamation (which only treats nlohmann/json.hpp
as external) copied the json internals into json_view.hpp a second time,
so the fuzzers, which include both single headers, failed with
redefinitions (make_void, integer_sequence, priority_tag, ...).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Compare keys with key_equals() in the object hash index

As the linear lookups do since the second review round of #5622.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
nlohmann added a commit that referenced this pull request Oct 21, 2026
* Add editable json_documents: set, push_back, insert, and erase

basic_json_document gets a second template parameter, Editable
(false by default), plus the aliases json_editable_document,
json_editable_view, ordered_json_editable_document and
ordered_json_editable_view.

Editable documents can change values and structure without
rewriting the source text: set()/push_back() on values, keys,
array indices and JSON pointers; insert() before an array
element; erase() of an object key, array index or JSON pointer.

New values and element sequences go into edit storage that the
document owns and never moves, so views keep referring to their
value across edits and a parsed node never moves. Read-only
documents walk the plain node array and are unaffected.

Strings are checked for UTF-8 on entry, so dump() of an editable
document never throws type_error.316. Binary values cannot be
stored (type_error.319).

A seeded differential test applies random edits to an editable
document and to the equivalent ordered_json and compares both
after every step.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix old clang and MSVC C4127 in editable json documents

Value-initialize the const std::less in find_parent (clang 3.4/3.6 do not
implement DR 253), and test the Editable template argument through a
function to avoid MSVC C4127.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Copy nested values into a document without recursion

Counting and copying the nodes of a view or a basic_json value into an
editable document recursed once per nesting level, so that a deeply
nested value overflowed the stack. The four functions now walk the value
with an explicit stack, as materialize() does.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not leave a half-assigned container when set_moved() throws

assign() rewrote the kind, length and extent of a slot before set_moved()
registered its new element sequence. set_moved() can throw
(std::bad_alloc from reserving the bookkeeping vectors) for a slot that
is not moved yet, which left a container without the moved flag that
showed its old children. The bookkeeping is now reserved first
(reserve_moved()), so that nothing after the first write to the slot can
throw. block_of() reserves before it marks anything for the same reason.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Check strings copied from another document for valid UTF-8

An editable document only holds valid UTF-8, but copy_scalar() copied the
strings and keys of a view of another document unchecked. A document of
a weaker check (or a borrowed text that changed after parsing) could
therefore bring ill-formed UTF-8 into it. The copy is checked now, with
the error that dump() reports for the string; copies within the same
document stay unchecked.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Allow the edit arena to grow up to 4 GiB - 1 bytes

append_text() doubled the capacity of the arena and rejected the result
if it exceeded 4 GiB, so that an arena of more than 2 GiB could not grow
even though the 32-bit offsets of nodes address 4 GiB - 1 bytes. The
capacity is now clamped to that limit (text_capacity()), and an append
is rejected only if the bytes themselves do not fit.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Set a null value below a pointer as basic_json does

set(json_pointer, value) turned a null parent into an object for every
last reference token, so "/a/0" and "/a/-" below {"a":null} made
{"a":{"0":1}}, where basic_json's operator[](json_pointer) makes an
array. A null parent now becomes an array for "-" and for digit tokens
(filled with nulls up to the index) and an object otherwise. An invalid
index is reported before the parent changes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Replace a container by a scalar in constant time once it is linked

Replacing an array or object that spans several nodes by a scalar looks
up its parent from the root (find_parent), which is linear in the size
of the document: setting every element of a 20,000 element array took
more than half a second. The parent only has to switch to links once;
afterward the extent of the replaced value no longer matters. Nodes that
an entry of a moved sequence links to are now marked (node_flags::linked),
and assign() skips the lookup for them.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Update the json_view banner and type_error.319 for editable documents

The banner still described a read-only view, and the documentation of
type_error.319 listed set and push_back only, although insert rejects
binary values as well.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Assign the member that lookups find when set() meets duplicate keys

An editable document reads the last member of a repeated key (as the
read-only document does), but set(object, key, value) assigned the first
one: a view taken from object[key] before the call did not show the new
value. It now assigns the last member's value, keeps the key at the
position of its first occurrence (where materialize() puts it), and drops
the other members.

The seeded differential test now also edits documents whose objects
repeat keys and compares every lookup (operator[], at, find, value,
contains, count, and JSON pointers) with the parsed basic_json value.
A targeted test covers duplicates before and after edits that move the
object, in large objects with an index, in moved arrays, and in values
copied from other documents.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Assign the first member when set() meets duplicate keys again

Lookups find the first member of a repeated key again, so set(object, key,
value) assigns that member (keeping the key where it is) and drops the
others, as documented before, and a view taken from object[key] shows the
new value. This undoes the code and documentation changes of 5cbb8e6.
Lookups in edited objects (navigation of editable documents) stop at the
first match, like those of parsed objects.

The tests that commit added expect the first member from lookups now. In the
seeded differential test, the documents with repeated keys repeat them after
the real members; the reference holds the first members, which the edits
address and the lookups are compared with, while materialize() is compared
with what parse() makes of the document's dump.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix MSVC C4127 in the lookups of editable json documents

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy 22 findings in the json_view headers

Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast
would trip -Wcast-align=strict), use auto for a cast initialiser, add
parentheses to a mixed expression, and name bugprone-std-namespace-modification
in the NOLINTs of the tuple_size/tuple_element specialisations.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
(cherry picked from commit 0effbaa)

* Fix clang-tidy 22 findings in unit-json_view_edit.cpp and edit.hpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep a raw string with a backslash out of a CHECK (MSVC C2017)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make json_view.hpp pass include-what-you-use

ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), and the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), plus <version> for
std::nullptr_t, and removing <cstddef> and edit_storage.hpp.

- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
  json_view.hpp only includes json.hpp, so including them here would duplicate
  their definitions
- drop edit_storage.hpp, which edit.hpp includes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the view's edit headers clean for clang-tidy in C++17 mode

clang-tidy 22 checks the headers with -std=gnu++17 as well, where it asks
for a transparent comparator and std::make_unique. The regions map uses its
default comparator, and the allocation of the edit state keeps new (the
library is C++11) with a NOLINT.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Follow the renames from the review of the view builder

The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Expect count() to count every member in the edit tests

Since #5622, count() counts every member with the key, so documents
with duplicate keys no longer give 1.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Count matching keys as unsigned in the json_view edit tests

Like in the access test, adding the int result of the conditional to a
std::size_t fails -Werror=sign-conversion with GCC.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address the review of erase() in editable json documents

From gregmarr's review of #5630:

- invalid_iterator.202 is thrown by overloads 1 and 2 only; the page said
  "every overload" and then excluded 3 in parentheses.
- The note on links says what moves (the links after the removed element,
  as insert() moves them the other way) and that every iterator over the
  container is invalidated, not only those after the removed element.
- edit.hpp compares keys with document_data::key_equals().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address the review of editable json documents

From gregmarr's review of #5630:

- set() with a JSON pointer below a null value checks for "-" before
  scanning the token for digits.
- node_flags::storage is defined as escaped | edited, and the comment on
  document_data::base says that the fourth slot is unused.
- source_extent() names its root check.
- The integer-index overloads of set(), insert(), and erase() use a named
  trait instead of a negated condition, which astyle formats like the
  library's other enable_ifs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address the rest of the review of editable json documents

From gregmarr's review of #5630:

- alloc_nodes() gives a request of half of the next chunk or more a chunk
  of its own, so that the rest of the current chunk stays available for
  smaller requests instead of being abandoned.
- The comments on edit_state, append_text(), and reserve_moved() say that
  views stay valid across every edit (not just the first), that n is 0 for
  empty strings, and why both capacities are checked.
- The architecture page names flag bits 3, 4, and 5 (linked) separately
  and says that a discarded view has no node; erase.md rewords overload 3
  and says why it never throws invalid_iterator.202.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Call alloc_nodes() before reading chunks.back() in the json_view chunk test

The operands of == are unsequenced, and MSVC evaluated
d->edits->chunks.back() before alloc_nodes() added the new chunk, so the
check compared the result with the previous chunk and failed on
msvc-arm64 (Debug and Release).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants