Repository navigation
Fixes for 6.0.0-rc.3 - #449
Open
dimensionscape wants to merge 31 commits into
Open
dimensionscape wants to merge 31 commits into
dimensionscape wants to merge 31 commits into
Conversation
Release stamp wrote extraParams.hxml from scratch with the two version defines, so the release package lost its third line, the macro that defines HXCPP_GC_GENERATIONAL and HXCPP_GC_INCREMENTAL_BARRIERS for -D HXCPP_GC_INCREMENTAL. With libhxcpp installed from a release, an incremental build stopped at hxcpp.h's "Incremental GC requires regenerated C++" error; from a checkout it built. The release workflow stamps before it packages, so 6.0.0-rc.2's package had the same file. Stamp now replaces only the libhxcpp and hxcpp define lines and keeps every other line as it is. test/release_stamp/check.sh stamps a copy and checks the defines and that each other line of the checkout's file survives; it runs in the haxelib job.
A link was redone only when an object or a library file on disk was newer than the output, or the list of objects changed. A flag or library added or removed alone left the output as it was, and the build reported success. On Emscripten every HXCPP_LINK_ define is link-only, and none reaches the compile options, so building a program with -D HXCPP_LINK_NO_EMBIND and again without it kept the program without Embind; the same held for HXCPP_LINK_STACK_SIZE, the memory growth defines and HXCPP_LINK_NODERAWFS. With MSVC, a -map: flag added to a target produced no map. The hash kept beside the output now covers the linker's and the target's flags, the libraries and the library paths as well as the objects, so a change to any of them links again. Every program links once more after updating, as its hash changes. test/link_flags builds once, then with a define that adds a map flag, and checks the map was written.
The build tool leaves every HXCPP_LINK_ define out of the options it compares between builds, taking them all for link options. On Emscripten HXCPP_LINK_NO_EXCEPTIONS also decides whether objects are compiled with -fwasm-exceptions. Adding it to a project built before without it recompiled nothing, and the link, now without -fwasm-exceptions, failed on __wasm_lpad_context and _Unwind_CallPersonality from the old objects (before the previous commit it did not link at all, so the program kept its exceptions). With the compile cache it worked, as the cache keys on the arguments. That one define is now compared like the others, so a change recompiles the groups that read Options.txt. test/emscripten/link_no_exceptions.sh builds without the cache, then again with the define, and runs it.
`haxelib run hxcpp defines` lists the defines the build tool reads by scanning tools/hxcpp/*.hx, but opened each file by its bare name, which resolves against the caller's folder, as haxelib runs the tool there. From anywhere but tools/hxcpp it failed with "Could not read .../tools/hxcpp : [file_contents,BuildTool.hx]" and exit code 1. It opens them in that folder now. The build tool tests run it and look for HXCPP_COMPILE_CACHE in the list.
With HXCPP_MSVC_VER, msvc-arm64-setup.bat sets up the install it names through that install's vcvars<host>_arm64.bat, but it spelled the x64 host as msvc_host_arch has it, x64, where Visual Studio's script is vcvarsamd64_arm64.bat, so on an x64 machine it stopped with "does not have vcvarsx64_arm64.bat". On an ARM64 machine (host x86) it checked for vcvarsx86_arm64.bat and then called vcvarsx86_amd64.bat, setting up the x64 compiler for an ARM64 build. It now names the x64 host amd64 and calls the _arm64 script it checked for. test/msvc_setup/check.ps1 runs the script against a stand-in install whose vcvars scripts say which ran, for both hosts.
The linker writes its record of the link, the .hash beside the output, as soon as the link succeeds, and the strip, manifest and PGO steps run after it. When one of those failed the build stopped with its error, but the output and its record were in place, so the next build found nothing to do and succeeded with the output unstripped or without its manifest: with -DmanifestFile set to a manifest mt.exe cannot parse, the second build exited 0. A failure after the link now deletes the record on the way out, so the next build links and runs the steps again. test/link_flags builds twice with a bad manifest and expects both to fail.
The fast paths of Mutex, Condition, Semaphore, Lock and Deque stop at a pending collection through __hxcpp_gc_safe_point (be8debe, cf748b7, bdf84d8). Mutex and Condition may be used inside a zone the caller has already entered, since their guard leaves only a zone it entered (603fd03). There the stop paused the thread as if it were outside the zone: during a collection PauseForCollect ended the process with "Bad Allocation while collecting", and before one it signalled and waited for mCollectDone, which the collector only sends to a thread it found outside a zone. A thread in a zone is already safe, so the safe point now returns there. The allocation paths still call PauseForCollect directly, so allocating inside a zone keeps its diagnostic. Test: SafePointInGcFreeZone takes Mutex and Condition in a loop inside a zone while the main thread collects. It ended with the critical error in 3 of 3 runs before the change, and passes 20 of 20 after it, Windows x64, MSVC 19.44.
The zone state is a flag, not a depth. Lock.wait, Semaphore.acquire, Semaphore.tryAcquire with a timeout and Deque's blocking pop and contended lock entered a zone with EnterGCFreeZone and left it with ExitGCFreeZone, so called inside a zone the caller had entered, as with cpp.vm.Gc.enterGCFreeZone around a blocking call, they took the thread out of it, and the caller's own exit ended the process with "GCFree Zone mismatch". Mutex and Condition already kept the caller's zone through hx::AutoGCFreeZone, which leaves only a zone it entered (603fd03). They use the guard too, closing it before a throw. The POSIX and Apple semaphores and the POSIX Deque branch are not compiled on Windows; the change there is the same substitution. Test: GcFreeZoneThreadWaits waits on each inside a zone, with a second thread releasing the ones without a timeout. It ended with the critical error at the first wait before the change, and passes 10 of 10 after it, Windows x64, MSVC 19.44.
Sys.sleep converted the duration straight to the native call's type: on Windows to a DWORD of milliseconds, which from 2^32 ms, about 49.7 days, wrapped to a short sleep or none, and elsewhere to an int of seconds, which overflowed past 2^31 s. Infinity, a way to sleep for good, returned at once on both. Windows now sleeps in parts of 0xfffffffe ms, one short of INFINITE, and POSIX in parts of 1e9 s, so an infinite sleep never returns. The POSIX branch is not compiled on Windows. Test: SleepLongDuration checks that sleeps of 4294967.3 s, 3e9 s and infinity are still running after 0.3 s, and that short, zero, negative and NaN sleeps return. Before the change, 4294967.3 s and infinity returned at once on Windows x64; after it all pass, 5 of 5.
init_timeval converted the timeout to an int of seconds, which overflowed past 2^31 s: Windows' select then returned at once with nothing ready, and Math.POSITIVE_INFINITY did the same. NaN went through the same undefined conversion. Socket.setTimeout already caps its own at 0x7fffffff seconds. The timeout is capped there too, and NaN waits for nothing. Windows' poll builds its timeval with the same function. Test: SocketSelectLongTimeout selects on a listener with timeouts of 3e9 s and infinity while another thread connects after 0.3 s. Before the change both returned at once with nothing ready on Windows x64; after it both wait for the connection, 5 of 5.
makeArrays gave every thread object a 65,535-byte scratch array, which only hx::thread::Scratch reads, and Scratch.cpp is compiled only at API level 500. With Haxe 4 every Haxe thread kept one for as long as it ran. A native thread that calls sys.thread.Thread.current() is added to Haxe 4's thread list and never removed, so its 64 KB stayed until the process ended: 2,000 such threads held 128 MB. Below API level 500 the array is no longer made. Marking and visiting already accept a null array. Test: ThreadScratchApi4 measures what 64 running threads hold after a collection. Each held more than 16 KB before the change and less after it, 5 of 5, Windows x64, MSVC 19.44.
The fuzz job's cppia target stopped in CallMember::linkSuperCall with "member access within null pointer of type 'CppiaFunction'": a super constructor call named a script class with no constructor, in it or in a script class above it, and linking read the constructor's function through its null newFunc. Haxe only calls a constructor a class has or inherits, and linkTypes gives a script class without one its script super class's, so only a malformed file gets there. Such a call now fails the load with "Super constructor call on a class with no constructor". Tests: CppiaMalformedConstructors loads such files, interpreted and with the JIT, and runs scripts whose classes take a constructor from a script class and from a host class above them; the fuzz input is in test/fuzz/regressions/cppia. #257.
A function's funExpr is its body only when the body is a function, so a constructor whose body is a bare RETURN, which a malformed file can give, or an interface's "new", which has no body, has none. linkTypes took such a function as the class's constructor anyway. A super call into it linked a CallFunExpr that read the return type through the null function, crashing the loader as the super call into a class with no constructor did, and a new of the class loaded and then crashed when it ran, interpreted and under the JIT. Haxe gives a class only a constructor with a body and an interface none, so such a file now fails to load with "Constructor is not a function". Tests: CppiaMalformedConstructors loads the files, interpreted and with the JIT; the fuzz input with the super call is in test/fuzz/regressions/cppia. #257.
A new of a type with a script class linked without a check, but a script interface has no host base: running the new made the object through the interface's null haxeBase, interpreted and under the JIT. Under the generational collector the JIT also reads the base's data offset as it compiles the new. Haxe makes no instance of an interface, so only a malformed file asks for one. Such a new is linked as one of a type with nothing to construct, and the file fails to load with "Could not link new" and the interface's name. Tests: CppiaMalformedConstructors loads such a file, interpreted and with the JIT. #257.
The fuzz job's ereg target reported a 12 second unit on run 44100276248, past libFuzzer's 10 second slow-unit report and 8 seconds short of the 20 second timeout. Its first match succeeds at once, so the target's stop after a match that gives up did not apply: split, replace and map each match again after that match, and each reached PCRE2's match limit of 10 million steps from one start before throwing. A pattern of adjacent \S* runs, on which a scheduled run timed out before, takes most of that limit at a start and still matches, so it is slow without ever reaching it: it replays in 5 seconds on MSVC without sanitizers. The limit is per start, so the pattern now begins with (*LIMIT_MATCH=100000), which a pattern can only lower. A pattern that does not backtrack takes a few steps a character: "^(a|b)*c$" over a 4,095 character subject needs between 8,000 and 20,000, and the other linear patterns tried over such a subject needed less. A pattern that backtracks past the limit throws, which the target still covers. The runtime keeps PCRE2's default limit. The approach and the \S* input come from an earlier fix that never reached main. Tests: both inputs are in test/fuzz/regressions/ereg, which the fuzz run and run.sh --replay go through. Without sanitizers, their replays take 1.5 and 5.1 seconds before and 0.2 to 0.4 after. #257.
A scheduled fuzz run's serializer target reported a Date of 2.16e18 ms changing in its last digit between the second and third round: the std's cpp Date keeps seconds, and past 2^53 ms, where a millisecond is less than a step between doubles, whole milliseconds can drift in a conversion, as fractions of one already could. That is the std's Date, not the runtime. canon() leaves such a Date's milliseconds out; a NaN time still compares as before. The 18 Dates from 2^50 to 1.2e16 ms that were tried, either side of 2^53, all settled after one round, so the cut-off stays at 2^53. From an earlier fix that never reached main, with the comparison turned round so a NaN time is not taken for a far one. Tests: the input is in test/fuzz/regressions/serializer, which failed on main with "serialized v2.1606691160662124e+18 then v2.1606691160662126e+18" and replays now. #257.
A function's funExpr is its body only when the body is a function, and
a bare RETURN, an abstract function's body, was taken for any function.
A dynamic function's value is a closure of its funExpr, made as its
class boots for a static one and as an instance is made for a member,
so with a bare RETURN it was a closure of no function: the file loaded,
and the next collection that marked a static one crashed, as did a
call of a member one, interpreted and under the JIT. Haxe has no
abstract dynamic function ("Invalid modifier combination"), so only a
malformed file gives one.
Such a file now fails to load with "Dynamic function is not a
function". The check sits with the one for a body that is no function
and no bare RETURN, so an interface's functions, which have no body,
are left as they were.
Tests: CppiaDynamicFunctionBody loads such files, then boots, runs and
collects, interpreted and with the JIT. #257.
A weak map's toString allocates as it walks the buckets, so a collection can run in the middle of the walk. That collection unlinked the entries whose keys had died, and the walk went on from its current entry's next, which still led to them when the current entry had been unlinked too: it called toString on keys the collector had finalized and freed, and on whatever had since taken their memory. 200 walks with a full collection part way through reached 54 finalized keys, and crashed once their memory was reused. While a walk is on a weak map, marking it marks its keys and values as a strong map does, so no entry is unlinked under the walk. Starting a walk remembers the map, so an incremental cycle that has already scanned it, or a minor collection of an old map, marks it again. The keys go at the first collection after the walk.
GetMemType sorts the bases of the large allocations into an index on the first lookups of a collection (#318), on the assumption that only the collecting thread looks up while marking. The marking threads look up too: a CFFI abstract made with create_abstract and memory of its own marks that memory conservatively in its __Mark, which any of them runs. Two threads could resize and sort one index at once, and one could search an index another was still filling, so a large buffer the abstract's memory pointed to went unmarked and was freed while held. With 20,000 buffers of 4,000 bytes held only through the memory of 8,000 abstracts, 10 to 14 of 16 full collections lost buffers, or the process crashed. With the marking on one thread (HXCPP_GC_VERIFY) none did. The index is built under a mutex, its ready flag is published with release and read with acquire, and the lookup counters are atomic.
With HXCPP_GC_NURSERY a small allocation keeps its size in the low 16 bits of its header and a large one 0 there. ObjectSizeSafe took any bit of the low 24 as a small size, so a large allocation with a flag, as a string of 4,000 bytes or more has for its hash slot or its UTF-16 text, gave a size of 0. _hx_gc_freeze copied none of such a string, and the frozen string read past its copy. ObjectSizeSafe reads the 16 size bits, as MarkAllocUnchecked already did.
The JIT's compound assignment to an array element gave * and - a Float right-hand side for every element type, so on an Array<Int> it worked in Float and converted the result back to Int. A result past the Int range converted to -2147483648, where compiled code and the interpreter wrap: values[i] *= 3 with 2147483647 gave -2147483648 instead of 2147483645, and a hash kept in an array went wrong as soon as it overflowed. An Int element now takes an Int right-hand side and Int arithmetic. The new test runs under the interpreter and the JIT.
The interpreter's negation gave a Float and a Dynamic made from -x whatever x was, and the JIT negated in Float arithmetic for every destination but an Int. So wherever -x was not used as an Int, as Dynamic, in a String or as a Float, an Int was negated as a Float: -0 printed as "-0", -(-2147483648) gave 2147483648 instead of wrapping as compiled code does, and Type.typeof gave TFloat. An Int is now negated as an Int, then converted. The new test runs under the interpreter and the JIT.
Array<Any> has builtins of its own, and their runObject boxed the results of contains and remove with the Int results of push and indexOf, before the case meant to box remove as a Bool, which was never reached. So in the interpreter "" + a.contains(x) was "1" and Std.isOfType(a.remove(x), Bool) was false; the JIT already boxed both as Bool. CppiaBoolBuiltins checks them too, under the interpreter and the JIT.
concat with a null array throws Null Object Reference under HXCPP_CHECK_POINTER, and HXCPP_CATCH_SEGV, which the Cppia host is built with, turns the fault into an exception too. The JIT's native calls for concat, on typed arrays and on Array<Any>, had no exception bridge, so the exception went through jitted frames and ended the process, where the interpreter reaches the script's catch. They now store it as the script's pending exception, which the code after the call checks, as blit, setSize and resize do. The new test runs under the interpreter and the JIT, with HXCPP_CHECK_POINTER.
A loading module is listed with the others from the start, and its list of types is sized before each type is made. Marking it already skipped the entries not yet made, but visiting it, which updates its pointers after a collection moved objects, read them all, so with HXCPP_GC_MOVING a collection that moved objects while a module was making its types crashed the host. The visit now skips them too. The new test loads a module 20 times, interpreted and under the JIT, in a debug build with HXCPP_GC_DEBUG_ALWAYS_MOVE that collects every three allocations while each module loads. It crashed in each of six runs before this change and passed in each of twelve after it.
A collection started because no free block could take an allocation counts the rows of the free blocks left as in use, since the allocator could not use them. It counted from mNextFreeBlockOfSize[0], which only a request of a line or less moves on, so the blocks that bigger requests had taken and filled since were counted too, with all the free rows they had before. Their rows are counted by marking. With a churn of 200 to 4,000-byte objects and almost nothing live, the heap looked full at every other collection and its target doubled, and the groups the collection after gave back were faulted in again. Only the free blocks no allocator owns are counted now. Churning 2,000,000 Bytes of 2,000 bytes reserves 20 MB instead of 41 MB, the same churn of strings after it 20 MB instead of 83 MB (896 MB with HXCPP_GC_MOVING), and 200,000 of those Bytes take 16 ms instead of 91 ms. GcChurnHeapTarget checks the reserved memory stays under 32 MB.
For x *= y with x not a Float, the JIT computed y into a float scratch register and then converted x to a Float with a native call, which may change every float scratch register. sljit 0.95 maps the first one to xmm0 on x86-64, where 0.93 used xmm1, and the conversion leaves its result there: on Windows x64, var x:Dynamic = 3; x *= 5 gave 9, and the cppia suite's check for hxcpp #926 failed under the JIT. x is now converted first, as -= and /= already did, and evaluated before y, as Haxe orders it. The new regression test covers *= on Dynamic locals, fields and array elements, with Int, Float and call operands; it gives 9 for the first case without this change. Refs #419
Haxe 4's haxe.atomic.AtomicInt on C++ is a cpp.Pointer into a one-element array that nothing else holds. The collector ignored cpp.Pointer fields, so an AtomicInt kept in a field read freed memory once the array was collected (HaxeFoundation/hxcpp#1225). Haxe 5 changed AtomicInt instead (HaxeFoundation/haxe#12236), which no Haxe 4 release has. A cpp.Pointer field and a pointer boxed in a Dynamic are now marked as a pointer on the stack is: whatever in the GC heap they point into is kept, and pinned under HXCPP_GC_MOVING. Pointers to memory outside the heap are left alone. HaxeAtomicIntField makes 10,000 atomics and 10,000 boxed pointers in passes of their own, so their lines hold nothing else that lives, and collects while churning one-element arrays. Before, 9,999 atomics and 9,996 pointers read freed memory; now none do, at 32 and 64 bits and with HXCPP_GC_MOVING, GENERATIONAL, NURSERY, BIG_BLOCKS, DEBUG_ALWAYS_MOVE and INCREMENTAL.
The test read and wrote plain statics from two threads, which ThreadSanitizer reports as a race and fails the job with exit code 66.
macOS' select fails with EINVAL for a timeval past 1e8 seconds, so the 2^31-1 second cap threw "Select error" there for a timeout of 3e9 s or infinity. The cap is 1e8 seconds on every system, over three years. Test: SocketSelectLongTimeout, which failed on macOS arm64 CI, passes on Windows x64 3 of 3.
A thread calling a function the lazy JIT was compiling on another read compiled with a plain load, while the compiler set it under its lock after a release fence. ThreadSanitizer, which does not model the fence, reported the race in CppiaLazyJit. compiled is now read with an acquire load and set with a release store, through getCompiled and setCompiled, and each caller calls the code it read rather than reading it again. Test: the Cppia regressions 58 of 58, the cppia suite under the interpreter and the JIT, and CppiaLazyJit 8 of 8 on Windows x64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for 6.0.0, to release as 6.0.0-rc.3 from main once CI passes. Each commit has its own regression test where one could show the problem, and a Changes.md entry where users would notice the change. Toward #154.
Release blocker
Build tool
Threads
GC
cppia
*=and-=wrap under the JIT, a negated Int stays an Int, and Array contains and remove return Bools in the interpreter.x *= yon a Dynamic evaluates x first and keeps y across the conversion call under the JIT. It is latent with the current sljit, and shows with sljit 0.95 (Upgrade the cppia JIT's sljit to 0.95 #419).Fuzzing
Tested on Windows Server 2022 with MSVC 19.44 and Haxe 4.3.7, at 64 bits: test/haxe (2698 of 2698), the fuzz replay (10 targets), the cppia suite interpreted and with the JIT, every Cppia regression and the GC and thread regressions. The heap change also passed every regression, test/haxe and test/std at 32 and 64 bits. Linux, macOS, ARM and the sanitizers have not run yet; this PR is for that.