Repository navigation
Master - #953
Closed
anmoltrivedi2312-ux wants to merge 108 commits into
Closed
Master#953anmoltrivedi2312-ux wants to merge 108 commits into
anmoltrivedi2312-ux wants to merge 108 commits into
Conversation
* FreeBSD StackTrace * Update CMakeLists.txt * Update CMakeLists.txt
…l_load_from_file (metacall#893) * test(core): add failing test for path heap-buffer-overflow in metacall_load_from_file * fix(core): reject paths exceeding LOADER_PATH_SIZE * Update metacall.c * Update loader_impl.c --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com>
…single <-> array value casts (metacall#898) * test: add regression tests for UAF/double-free in value_type_cast array casts * fix(reflect): prevent heap-use-after-free in array value casts * Update reflect_value_type_cast.c --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com>
* adt: add vector_insert_empty test covering front and middle insertion Refs metacall#896 * adt: loop up to count in vector_insert_empty test, check last element outside Refs metacall#896 * adt: drop the nested expected ternary in vector_insert_empty test Refs metacall#896 * adt: split vector_insert_empty test into front, middle and end cases * adt: fix vector_insert_empty memmove source and destination The arguments were swapped, so elements were moved left instead of being shifted right to open the gap. Fixes metacall#896. * adt: simplify vector_insert_empty tests with hardcoded asserts Keep only insert_empty_create_sequence() as a helper and hardcode every expected value. The inserted slot is no longer overwritten, so the duplicated value left by memmove shows the elements were shifted right. Refs metacall#896
…etacall#901) Line 511 had two leftover tabs on an empty line, which makes the clang-format check fail on develop and on every open PR. Fixes metacall#900
metacall#903) * adt: read trie node before popping it off the stack trie_node_iterate, trie_node_clear and trie_node_find kept a pointer into node_stack and read it after vector_pop_back. It only worked because pop_back never reallocated on small vectors. Copy the node out first so it stays valid once the shrink check in metacall#902 is fixed. Refs metacall#902 * adt: fix inverted shrink check in vector pop_back, pop_front and erase The condition was backwards compared to vector_resize, so removing an element reallocated almost every time and grew the capacity on a full vector. Only shrink once size drops below capacity / 8. Fixes metacall#902
…and casting (metacall#899) * test(reflect): add failing regression tests for sign-extension in promotion * fix(reflect): sign-extend negative integers in promotion and decimal casts * fix(reflect): force signed char cast for cross-platform sign-extension
) New keys were written to s->data[s->size++] without checking the capacity, so inserting into a full set went past the allocation. Check it after the lookup so updating an existing key still works. Fixes metacall#904
…etacall#906) * test(metacall): add regression test for truncation in metacall_execution_path * fix(metacall): reject overlong paths in metacall_execution_path
The memcheck target always ran every test under valgrind and printed all the logs at the end, so there was no way to check a single test. Add a MEMCHECK_TEST env var that is passed to ctest as a regex, and remove old MemoryChecker logs before running so only the logs of the tests that just ran are printed.
* tests: make memcheck fail on valgrind errors and leaks Valgrind always exited with the test's own exit code, so tests with leaks or invalid accesses still passed under memcheck and the target only failed when a test failed on its own. Add --error-exitcode=1 so ctest marks those tests as failed, same as the python valgrind job. Still reachable memory doesn't count as an error, so it won't fail. * Update CompileOptions.cmake --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com>
…et_directory (metacall#908) * portability: keep null terminator inside buffer in get_fullname and get_directory Both functions could write the terminator at buffer[size] when the result filled the whole buffer. Leave one byte for it in get_fullname and clamp last in get_directory, like get_directory_inplace does. Fixes metacall#907 * portability: explain the exact buffer tests in comments Refs metacall#907 * portability: draw the buffer layout in the exact buffer test comments Refs metacall#907
…zed (metacall#911) loader_impl_destroy only calls the loader's destroy when it was initialized, and that destroy is what ends up calling loader_impl_destroy_objects. A loader that only got created (e.g. by calling metacall_execution_path without loading anything) skipped it, leaking handle_impl_path_map, handle_impl_map, handle_impl_init_order and type_info_map. Found by valgrind in metacall-path-overflow-test.
metacall#921) py_loader_impl_error_print never released type_str_obj and value_str_obj, and py_loader_impl_error_value_from_exception never released value_str_obj. Both come from PyObject_Str so they are new references, and they leaked every time a python error was handled. exception_create_const copies the strings, so it is safe to release them after creating the exception. Found with valgrind.
…ibutes (metacall#924) * py_loader: release python values after setting them in dicts and attributes PyDict_SetItem, PyObject_SetAttr and PyObject_GenericSetAttr do not steal the references, so the key and value created when converting a map, and the value converted in py_object_interface_set and py_class_interface_static_set, were never released. py_class_interface_static_invoke also never released the result of the call, unlike py_object_interface_invoke. py_loader_impl_value_to_capi returned the class without incrementing its reference for TYPE_CLASS, while every other type returns a new reference and all callers treat it as owned. Increment it so releasing the converted values is safe for classes too. Found with valgrind, metacall-python-dict-test and metacall-python-object-class-test go from 109 and 122 bytes definitely lost to 0. * py_loader: do not decrement dict references when wrapping exports py_loader_impl_finalizer_wrap_dict decremented every key and value of the dict after copying it into the wrapper. That only balanced the extra reference py_loader_impl_value_to_capi was leaking for each map pair. Now that the conversion releases them, the original dict and the wrapper copy each own one reference, so decrementing them again made the keys be freed while still in use (heap-use-after-free in the py_port tests on CI). * py_loader: drop class reference increment in value_to_capi The TYPE_CLASS conversion is going to be reimplemented separately, so leave it as it was.
…eaks (metacall#940) - Fix missing Py_DecRef(module) in py_port_initialize() inside py_loader_port.c - Fix unhandled PyObject_Call return value in py_loader_impl_initialize_asyncio_module()
* tests: rb: Add type conversion test for ruby loader Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Implement buffer, array, map serialization and extend type conversion matrix Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * tests: rb: Add assertions for bool, int, double, string, array, map and buffer round-trips Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Guard IO::Buffer usage behind RUBY_API_VERSION_MAJOR >= 3 Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Fix clang-format style for preprocessor include Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Detect ruby/io/buffer.h availability via CMake instead of Ruby API version Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Combine header detection with Ruby API version check for extra safety Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * rb: loader: Use Ruby API version minor check instead of CMake header detection Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> * Update rb_loader_impl.c * Update metacall_ruby_conversion_test.cpp * Update rb_loader_impl.c * Update metacall_ruby_conversion_test.cpp * Update metacall_ruby_conversion_test.cpp * Update metacall_ruby_conversion_test.cpp * Update metacall_ruby_conversion_test.cpp --------- Signed-off-by: Ashutosh Sharma <itsashutosh769@gmail.com> Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com>
…ize (metacall#945) * fix(rb_loader): remove duplicate TYPE_BUFFER branch in rb_type_serialize * Update rb_loader_impl.c * fix(rb_loader): allocate internal IO::Buffer with NULL base and memcpy * Update metacall_ruby_conversion_test.cpp --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com>
…in test (metacall#938) * tests: add metacall-node-python-await-chain-test Chain metacall_await calls from the node port with multiple .then(), alternating node and python functions, with one then returning a plain value and the others returning another metacall_await promise. It checks the value of each step and that all of them run, so the chaining of the promise returned by the port is covered after the await future is destroyed right after awaiting (metacall#935). * node_loader: resolve port await promise inside a callback scope node_loader_impl_handle_promise_safe resolves the deferred from a uv async callback with only a handle scope, so the .then handlers stay in the microtask queue while the async handle is closed. The destroy hook then sees no active handles and unloads the loaders, and the pending handler calls metacall_await on a freed loader (heap-use-after-free in metacall-node-python-await-chain-test). Open a callback scope around the resolution so closing it runs the microtasks before the handle is closed. * node_loader: initialize the port await async handle in the js thread The promise of metacall_await can be resolved from another thread (python resolves it from its asyncio thread), and the uv async handle used to send the result back was initialized at that moment from that thread. uv_async_init is not thread safe, and while the promise is pending there is no handle that keeps the event loop alive, so the destroy hook sees an empty loop and unloads the loaders before the result arrives. Initialize the handle in node_loader_impl_promise_await, so it is created in the js thread and keeps the loop alive until the promise is resolved. Close it if the await fails and the promise will not be resolved. This also closes the handle (and frees the context) when the promise is resolved in the js thread, before the context was leaked because the handle was never initialized. * tests: check py_double input and drop atomics in node/python await chain test Everything in the chain runs in the V8 thread, so a plain counter is enough to track the steps. Also assert the input inside the python function, so a wrong value passed from node rejects the chain. * tests: add node/python await stress test Launch 2000 metacall_await calls to python and 2000 node/py/node chains at once, so many async handles are created and resolved from both the node and the python threads. Check every result and that all of them finish before destroy. * tests: use PREPROCESSOR_STRINGIFY for the await stress test size * tests: fix await stress test formatting and drop asyncio debug Stringify the stress size into its own macro, so clang-format does not realign the rest of the script after the PREPROCESSOR_STRINGIFY call. Remove PYTHONASYNCIODEBUG from the test environment, it is not needed and slows down the debug runs. * benchmarks: move node/python await stress test to metacall_node_py_await_bench The stress test was too slow for the debug test runs, so turn it into a benchmark without any of the python debug environment variables. Each run launches the python awaits and the node/py/node chains at once from a node function, awaits it from C and checks that all of them resolved with the expected values. It runs with 2000 and 10000.
* add mingw msys2 ci * minor bug * minor bug * minor bug * minor bug * minor bug and fork test off * experimenting * experimenting * experimenting * Update windows-mingw-test.yml * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt * Update CMakeLists.txt * experiment * experiemnt * experimenting * experimenting * experimenting * experimenting * experiment * experiment * experiment --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com> Co-authored-by: Vicente Eduardo Ferrer Garcia <vic798@gmail.com>
…etacall#943) Co-authored-by: abhinav-phi <abhinav-phi@users.noreply.github.com>
…metacall#944) * fix(loader): enforce unique native type names in loader type registry * tests(loader_types): use add_loader_dependencies and metacall_loaders header Address review comments on PR metacall#944: - Include <metacall/metacall_loaders.h> in metacall_loader_types_test.cpp and remove redundant compile definitions from CMakeLists.txt. - Replace manual loader dependency checks with add_loader_dependencies() and remove redundant simd_json_serial dependency.
metacall_clear() frees a loaded module handle. Calling it from a host callback that a guest runtime invoked unloads the module while the interpreter is still executing it, which segfaults when the callback returns. Document the restriction next to the existing lifecycle warnings.
…acall#931) When metacall_await is called from node and the promise is resolved or rejected from the JavaScript thread, handle_promise_safe never initializes its uv async handle, so destroy() closed nothing and the context (and the copied result) leaked. Release it directly in that case.
* initial rust template generics and template test * option 2 poc * clang and realted changes * fix clang format issues * wiring up in rust loader * hardcode to dynamic instantiation * added runtime rust template instantiation * Format template sources with clang-format 21. Apply the develop include guard style (#define X_H) to reflect_template.h and format rs_loader_impl.h, reflect_function.h and reflect_template.h. Add the license header and final newline to reflect_template.h/.c. Rework reflect_template_test: single line calls and assertions, one element per line in struct array initializers, check the created types in NestedTemplate and derive argument counts from the arrays. Replace the designated initializer (.value_template), not valid in C++17 (MSVC), by explicit member assignment. * Solve Format.cmake error with files without extension. The pre-commit hook passes all the staged files, for files without extension (i.e: CONTRIBUTORS, githooks/pre-commit) LAST_EXT is empty and string(SUBSTRING "" 1 -1) fails with a fatal error. Remove the leading dot with a regex instead, so empty extensions are filtered out. * Add template interface and template value type. Redesign templates following the function and class pattern: a template holds a template_impl and a template_interface (create, instantiate_function, instantiate_class, destroy) provided by the loader. The loader creates the instantiated function or class with its proper name and impl, so reflect no longer formats names or builds signatures from template arguments. Templates are reference counted and tracked by the memory tracker. Instantiated functions and classes hold a reference to their template (function_set_template / class_set_template) and release it on destroy. Add TYPE_TEMPLATE / METACALL_TEMPLATE value type with create, to, from and cast functions, copy and destroy support, a default value in metacall_value_create, and placeholders in the serializers. Other changes in reflect_template: forward declaration in reflect_template_decl.h, plain enums, named argument union, goto error handling, no strdup, duplicated parameters are rejected and template types are validated. The rs_loader is not adapted to the new interface yet, the FFI declarations still use the previous template_create and template_instantiate_function signatures. * Solve include guard of generated preprocessor headers. The preprocessor headers are generated from preprocessor_template.h.in on each configure, the template still defined the guard as "#define X_H 1", so every configure reverted the guard style updated in e866151. * Implement template interface in rs_loader. Templates are created with a TemplateImpl (parsed generic function, loader and compiled library) and the template interface singleton, and registered as template values in the context, which own them. The instantiation goes through template_instantiate_function, which calls back the Rust interface: it looks up the wrapper generated at compile time and creates the function with the name of the instantiation in Rust syntax (i.e: identity::<f64>) and the signature of the template arguments and return with the generics substituted. Function creation with signature is split from its registration so it can be reused by the template interface. Remove the duplicated instantiation loop of the registrator. A function can only be associated once to a template, same as classes. The call to the generic identity function in metacall_rust_test is commented out, it must be instantiated before calling it, pending to be reviewed. * Add license header and final newline to template tests. * Solve methods registered as functions when loading rust packages. The associated functions of a struct were also added as free functions of the crate (or templates if generic), so the package wrapper referred to them as metacall_package::<method>, which does not exist, and the compilation of the wrapper failed (metacall-rust-load-from-package-class-test). Associated functions are only registered in their class now, generic methods are skipped until they are supported. * Add template placeholder to py_loader value conversion. TYPE_TEMPLATE had no branch in py_loader_impl_value_to_capi, so it returned NULL and the call left a NULL item in the arguments tuple, crashing inside Python (metacall-conversion-test). Pass None with a TODO log until templates are implemented in the Python loader, same as futures. * Add error logs to reflect template. * Solve wrong license headers in tests. * Add template placeholder to node_loader value conversion. TYPE_TEMPLATE had no branch in node_loader_impl_value_to_napi, so it fell into the default case and threw. Requiring a Rust file with a generic function (basic.rs exports identity<T>) failed in the node port tests. Return undefined with a TODO until templates are implemented in the NodeJS loader, same as the py_loader placeholder. * Solve reference counting of instances in reflect template test. The test destroyed the instantiated functions and classes without holding a reference, so their counter went below zero. Without thread sanitizer the decrement refused it, logged an invalid reference counter error and destroyed anyway. With thread sanitizer the counter wrapped around, the instance was never destroyed and its template reference never released, failing DefaultConstructor and InstanceKeepsTemplateAlive in the clang tsan jobs. Take a reference after each instantiation, like reflect_function_test. * Solve underflow and overflow of atomic reference counter with thread sanitizer. The thread sanitizer implementation of threading_atomic_ref_count_increment and decrement did not check the bounds, so decrementing at zero wrapped around instead of returning an error as the atomic implementation does. Check the bounds inside the mutex. The atomic implementation loaded and checked the counter before adding or subtracting in a separate operation, so two threads could pass the check and overflow or underflow it. Use a compare and swap loop so the check and the update are a single atomic operation, keeping the same memory orders. --------- Co-authored-by: Vicente Eduardo Ferrer Garcia <7854099+viferga@users.noreply.github.com> Co-authored-by: Vicente Eduardo Ferrer Garcia <vic798@gmail.com>
…ing Python exception values (metacall#946)
Prepare the Rust Loader for instantiating templates after a script has been loaded. Instance crates must link against the script crate and share its statics, which requires the script to be compiled as a dylib with a shared standard library (rustc refuses a dylib that links std statically). Scripts are compiled with a hybrid linkage: - Without templates: cdylib, std linked in, no libstd preload, loaded with LOCAL | LAZY. As before, and unloadable. - With templates: dylib with prefer_dynamic, libstd preloaded, loaded with LOCAL | LAZY | NODELETE. Rust std registers thread local destructors with its own __dso_handle, so the system pins libstd instead of the script library, unloading it would crash when a thread exits afterwards. Add DYNLINK_FLAGS_BIND_NODELETE to dynlink, implemented with the mechanism of each platform: RTLD_NODELETE on unix, GET_MODULE_HANDLE_EX_FLAG_PIN on Windows, NSUNLINKMODULE_OPTION_KEEP_MEMORY_MAPPED on legacy macOS, and an error on BeOS and HP-UX where the loaders have no equivalent. Generate unique crate names (sanitized base name, a tag per loader load and an atomic counter), because libraries are resolved by SONAME and libraries that cannot be unloaded survive metacall_destroy. The library name is generated with dynlink_impl_get_name. Ignore private structs and nameless classes when analyzing the source. Tests: dynlink-test covers NODELETE, and metacall-rust-reinitialize-test reinitializes metacall three times with and without templates, checking thread local destructors of threads exiting before and after destroy.
… cache Add a public API for templates and instantiate Rust function templates at run-time, not only the instantiations requested at load time. Reflect: - Templates store the type_id of what they produce instead of a separate template_id enum, so other kinds (i.e: containers) can be templates later. - A single template_instantiate returns a value of the type of the template, the template interface has a single instantiate callback returning a value which is checked against the type of the template. - Instances are cached in the template (arguments compared by kind and pointer), the template owns the values and releases them before the interface destroy. A caller holding its own reference keeps the instance alive after the template is destroyed. - Template arguments can be a type, a class or a template. - Remove the template back reference from function and class, and merge reflect_template_decl.h into reflect_template.h. MetaCall: - metacall_loader_type returns a single type of a loader, and metacall_loader_types exposes the type reference, so it can be used as a template argument. - metacall_template, metacall_handle_template, metacall_template_size, metacall_template_parameter and metacall_template_instantiate, which returns the value owned by the template. Rust Loader: - Instantiations not compiled at load time are compiled into a new crate (dylib with prefer_dynamic) linked against the script crate, reusing its metacall_class module, and loaded with LOCAL | LAZY | NODELETE. - Only function templates are implemented, class templates return NULL. - Add RS, JL and LLVM to loaders.h.in, the Rust template test was compiled out without OPTION_BUILD_LOADERS_RS. Tests: reflect-template-test covers the cache, instance lifetime, class and template arguments and wrong instance types. metacall-rust-template-test covers the API, run-time instantiation, trait bounds and private handles. metacall-loader-types-test covers metacall_loader_type with the Rust Loader. TODO: The Rust Loader registers types that are not Rust types (Array, Map, Ptr and Null), so they cannot be used as template arguments yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a summary of the change and which issue is fixed. List any dependencies that are required for this change.
Fixes #(issue_no)
Type of change
Checklist:
make testorctest -VV -R <test-name>)../docker-compose.sh test &> outputand attached the output.OPTION_BUILD_ADDRESS_SANITIZERor./docker-compose.sh test-address-sanitizer &> outputandOPTION_TEST_MEMORYCHECK.OPTION_BUILD_THREAD_SANITIZERor./docker-compose.sh test-thread-sanitizer &> output.Helgrindin case my code works with threading.cmake --build build --target clang-format(orcmake -DMODE=fix -P cmake/Format.cmake) in order to format my code and my code follows the style guidelines.AI-ASSISTED: yes/no, model, human reviewer) and I own each line.If you are unclear about any of the above checks, have a look at our documentation here.