Skip to content

Commit 2d76874

Browse files
committed
System.Net.Primitives from the framework's IL; delete the IPAddress/IPNetwork shadows
IPAddress, IPEndPoint, EndPoint and IPNetwork now come from System.Net.Primitives' own IL (ADR 0023, Phase 1 of ADR 0025); corelib/Net/IPAddress.cs and IPNetwork.cs (633 lines) are deleted. SslClientHelloInfo, a System.Net.Security type that lived in IPNetwork.cs, moves to Net/SslStream.cs. The shadow's hand-written parser was silently wrong on IPv6 scope ids: "fe80::1%", "fe80::1%nosuchif0", "fe80::1%2%3", "fe80::1%4294967296" and "fe80::1%-1" are valid addresses with scope 0 in .NET and were rejected. It also lacked the span and UTF-8 Parse/TryParse, TryFormat and EndPoint.Serialize/Create. What stays csharpc's is the socket engine's contract (ADR 0024 boundary A): SocketException (native error and text from the OS layer) and the two internal classes of the assembly that P/Invoke libSystem.Native, SocketAddressPal (struct sockaddr, the same layout NetNative uses, per target OS) and InterfaceInfoPal (if_nametoindex), in the new corelib/Net/SocketAddressPal.cs. Every sockaddr access is bounded by the buffer length, as libSystem.Native's EFAULT checks are. General pieces the IL needed: - UTF-8 Parse/TryParse for short, sbyte, byte, ushort, uint and ulong; - a boxed Int128/UInt128's ISpanFormattable slot formats only that type instead of going through the switch over every primitive (decimal, DateTime and the calendars): tests/size/net.cs 393 536 to 255 976 B, json.cs 1 572 320 to 1 559 040 B; - a program that names any System.Net namespace takes System.Net.Primitives, since the corelib's Socket, Dns and TLS shadows take and return its types. (The branch's earlier commits brought MemoryExtensions.TryWriteInterpolatedStringHandler from the CoreLib slice, which IPNetwork.TryFormat uses, and Vector128 array/span forms.) Numbers (linux-x86_64, -O2 stripped): tests/size/net.cs 171 480 (shadow) to 255 976 B, NativeAOT 1 365 832; IPAddress.Parse/ToString loop (2 M, median of 7) shadow 336 ms, IL 237 ms, NativeAOT 295 ms; compile of net.cs 4.8 to 6.3 s user. New budget linux-x86_64 net 264192 (measured + max(4 KiB, 3 %)). Samples 1316 (parsing corner cases) and 1317 (IPEndPoint.Serialize/Create, SocketAddress) are recorded from real .NET; the 46 networking samples pass at -O0 and -O2.
1 parent f828fef commit 2d76874

20 files changed

Lines changed: 481 additions & 144 deletions

‎docs/16-interop-io.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -232,12 +232,13 @@ to a method expecting one) is the corelib object and dispatches virtually.
232232
## System.Net (sockets)
233233

234234
`src/Csharpc/corelib/Net/` implements the basic networking types in C# over the OS's BSD socket API
235-
(libc on Linux and macOS, Winsock/`ws2_32` on Windows): `IPAddress`/`IPEndPoint` (parsing and
236-
formatting follow .NET: inet_aton-style IPv4 input such as `127.1` or `0x7f.1`, RFC 5952 IPv6
237-
output with embedded IPv4 for mapped addresses), `Socket` (TCP and UDP; `new
235+
(libc on Linux and macOS, Winsock/`ws2_32` on Windows): `Socket` (TCP and UDP; `new
238236
Socket(SocketType, ProtocolType)` and `new TcpClient()` are dual-mode IPv6 sockets, so IPv4 peers
239237
appear as `::ffff:a.b.c.d`, as in .NET), `NetworkStream`, `TcpListener`, `TcpClient`, `UdpClient`
240-
and `Dns.GetHostAddresses`/`GetHostName` (getaddrinfo). Constants, `sockaddr` layouts (macOS has a
238+
and `Dns.GetHostAddresses`/`GetHostName` (getaddrinfo). `IPAddress`, `IPEndPoint`, `IPNetwork` and `SocketAddress`
239+
are System.Net.Primitives' own code, compiled from the framework's IL (ADR 0023, #326); its two internal classes that
240+
call libSystem.Native are served by the corelib over the same `sockaddr` layout (`corelib/Net/SocketAddressPal.cs`:
241+
`SocketAddressPal`, and `InterfaceInfoPal`, whose `if_nametoindex` resolves a scope such as `fe80::1%eth0`). Constants, `sockaddr` layouts (macOS has a
241242
length byte first) and errno values are per target OS. Errors are `SocketException`s with .NET's
242243
`SocketErrorCode` (Winsock numbering, mapped from errno), the native error as `ErrorCode`/
243244
`NativeErrorCode`, the OS's message text, and — for failed connects — the endpoint appended, as .NET

‎docs/18-compatibility.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -580,8 +580,9 @@ What it took, besides library members (`Socket.Poll`, `DnsEndPoint` connects, `S
580580

581581
- **Classes deriving from `TaskCompletionSource(<T>)`**: the connection waiters and write-queue entries of the
582582
pool. A derived class is laid out over csrt's task struct ([10-async.md](10-async.md#tasks-and-the-scheduler)).
583-
- **System.Net.Primitives from IL** next to the corelib's `IPAddress`/`IPEndPoint` shadows (a *mixed* assembly,
584-
ADR 0013): `CookieContainer`, `CredentialCache`, `NetworkCredential`, `DnsEndPoint`, `HttpVersion` are .NET's own code.
583+
- **System.Net.Primitives from IL** (a *mixed* assembly, ADR 0013): `CookieContainer`, `CredentialCache`, `NetworkCredential`,
584+
`DnsEndPoint`, `HttpVersion` are .NET's own code, and since #326 `IPAddress`, `IPEndPoint` and `IPNetwork` too; only
585+
`SocketException` and the sockaddr shims stay with the socket engine.
585586
- **Awaits on socket operations continue on the thread pool.** The epoll thread used to complete Task-based
586587
operations inline, so an `await` continuation (and the rest of `Main`) ran on it; one synchronous socket call there
587588
(`HttpClient.Send` after an `await`) blocked the only thread that delivers readiness, a deadlock (sample 1008).

‎docs/21-ownership.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@ System.Private.CoreLib. Only the code that is necessary stays.
99

1010
> The tables below are the snapshot taken before the Regex flip (#206) and the System.Text.Json flip (#286): the Regex and
1111
> Json shadows (the `Regex` and `Json` rows; `Json/ArrayBufferWriter.cs` went with System.Memory, #306) are deleted and `ownership.tsv` no longer lists them, nor the Linq ones (`Enumerable.cs`, `LinqIterators.cs`: #267;
12-
> `LinqSpans.cs` is now `RuntimeSpans.cs`); the totals are regenerated by `tools/il-first/ownership.sh` on the next refresh.
12+
> `LinqSpans.cs` is now `RuntimeSpans.cs`), nor `Net/IPAddress.cs` and `Net/IPNetwork.cs` (System.Net.Primitives from IL, #326; the
13+
> socket engine's sockaddr shims are the new `Net/SocketAddressPal.cs`, GLUE); the totals are regenerated by `tools/il-first/ownership.sh` on the next refresh.
1314
1415
This page is the map from today's `src/Csharpc/corelib` (354 files, 169,705 lines, 8.1 MB) to that principle. The
1516
per-file data is [`tools/il-first/ownership.tsv`](../tools/il-first/ownership.tsv); regenerate it, or print the tables

‎docs/adr/0013-framework-il.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,11 @@ everything the corelib covers — compile and link exactly as before.
105105
needed them, and writing them again as shadows would have been ~3 000 lines of ported code. The
106106
libSystem.Native calls of that assembly (socket-address shims) sit in the shadowed types only. A program that
107107
names System.Net.Sockets takes System.Net.Primitives too, since the corelib's sockets test for its DnsEndPoint.
108+
Since #326 (ADR 0023) IPAddress, IPEndPoint, EndPoint and IPNetwork come from IL as well; what stays shadowed is
109+
the socket engine's contract: `SocketException` (its native error and text are the OS layer's) and the two internal
110+
classes that P/Invoke libSystem.Native, `SocketAddressPal` (struct sockaddr) and `InterfaceInfoPal` (`if_nametoindex`),
111+
in `corelib/Net/SocketAddressPal.cs`. A program that names any System.Net namespace takes System.Net.Primitives,
112+
since the corelib's Socket, Dns and TLS shadows take and return its types.
108113
- System.Collections.NonGeneric was mixed for a while (`CollectionBase` a shadow, [ADR 0022](0022-tls.md)); the shadow is gone (ADR 0023,
109114
#271) and the whole assembly comes from IL: the corelib's certificate collections derive from the real `CollectionBase`.
110115
- *Assemblies that define types and forward others to CoreLib* (`TwinReference`, 2026-11-10: System.Collections, System.Collections.Concurrent).

‎docs/adr/0023-framework-il-first.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ samples are tiny and start-up dominated.
100100
| `System.Collections.Generic.Dictionary<TKey,TValue>`, `HashSet<T>` and `System.Runtime.InteropServices.CollectionsMarshal` (CoreLib types, #274) | the 116 samples that mention them (`Dictionary`, `HashSet`, `ToLookup`, `GroupBy`, `Distinct`, JSON, Linq, Roslyn library members...) pass at -O0 and -O2, 36 Int128/Vector128/BigInteger samples too, `tests/slicer.sh` (11 slice runs) passes; three new samples recorded from real .NET for what the flip exposed (1305: boxed `Index`/`Range` as keys and `object`, an `Int128` in a boxed struct and a `Dictionary<Int128,V>` enumerator boxed through `IEnumerable`, `Comparer<Int128>.Default` which threw `InvalidOperationException`; 1306: `Comparer` of string collections and `ReadOnlySpan<char>` alternate lookups; 1307: the missing-key and duplicate-key messages for 18 key types), found along the way: #277 (SortedDictionary duplicate-key message) | **+5 % to +14 % on programs that format a number**, because the corelib's culture caches are `Dictionary<string, ...>` and now carry the real `Resize`/`Remove`/enumerator code and tables: linux-x86_64 async 156 544 to 172 144 B (+10.0 %), collections 143 448 to 163 032 (+13.7 %), linq 174 600 to 189 864 (+8.7 %), numbers 129 096 to 143 912 (+11.5 %), regex 825 600 to 870 304 (+5.4 %), regexoptions 1 949 520 to 2 093 104 (+7.4 %); hello, exceptions, typeinfo unchanged. A 4-collection program (`Dictionary<string,int>`, `HashSet<int>`, `HashSet<string>`, `Dictionary<int,int>`) 145 056 to 155 744 B (+7.4 %). Without the two profile replacements below it was +46 % (212 368 B) | collections 3.92 to 4.08 s (+4 %), linq 4.59 to 4.95 s (+8 %) CPU | instructions, same program, work beyond start-up: `Dictionary<string,int>` set/get/remove 33.5 M to 33.9 M (**+1.4 %**), `HashSet<int>` add/contains 69.2 M to 67.4 M (-2.5 %), `HashSet<string>` 28.1 M to 31.0 M (**+10 %**), `Dictionary<int,int>` 93.7 M to 84.3 M (**-10 %**); the spike before the type-guarded default comparer (#269) was 21 to 31 % behind | 1 425 deleted and 71 added in `corelib` (`HashSet.cs` and `CollectionsMarshal.cs` deleted, `DictionaryImpl` out of `Dictionary.cs`; added: contract members in `UpstreamSupport.cs`), against 1 431 deleted and 218 added over `src` and `tools` (the compiler side: boxed `Index`/`Range` method tables, 8-aligned `Int128` aggregates, `Comparer<Int128>`, IL folding of `typeof(A) == typeof(B)`, generic forwarding in the slicer) | **Flipped**. Two things the flip decides: (1) the string comparer family stays redirected to the corelib, with a wrapper (`UpstreamWrappedDefaultStringComparer`) for the default comparer so alternate lookups and `Comparer` behave as .NET's; the randomized comparer upstream switches to after many collisions (hash-flooding mitigation) is not there, as it was not there for the shadows: not weaker than before, first thing to add when the string comparers are flipped. (2) The throw helpers that format the key with `string.Format` are replaced (`csharpc` profile) by direct text for `string`, `int`, `long`, `uint`, `ulong` and `char` keys and `ToString`/`IFormattable` for the rest, so a program reading a Dictionary does not carry composite formatting and every number formatter. The size cost is the owner's call under the standing authorisation; a follow-up could replace the culture caches' Dictionaries with something smaller. |
101101
| `System.Text.StringBuilder` and its chunk enumerator (CoreLib type, #304) | all 69 samples that mention StringBuilder, plus the record, interpolation, format, tuple and regex samples (244 at -O0 and -O2) pass; sample 1309 (recorded from real .NET) for the generic shapes the framework's handler uses on boxed values; found along the way: #305 (interface tests on boxed Nullable, enums and Half in generic code) | **neutral overall**, smaller where StringBuilder is used by the corelib: linux-x86_64 regexoptions -11.5 KB (-0.5 %), regex -3.5 KB, collections -3.8 KB, async/linq/numbers +0.3 KB; hello, exceptions, typeinfo unchanged. **A program that appends numbers and interpolations to a StringBuilder grows**: the benchmark program 140 000 to 209 280 B (+49 %, the real handler's generic formatting) | linq 6.84 to 7.11 s CPU (+4 %) | instructions per operation, 100 000 to 300 000 operations: `Append(char)` equal, `Insert` equal, `ToString` +8 %, `Append(string)` **+99 %** (27 to 53 per call: `Buffer.Memmove` is a call to libc for short strings, the shadow copied inline), `Append(int)` +46 %, `AppendLine` +88 %, `Append($"...")` **+75 %** (10.6 M to 18.8 M; was 10x slower until three IL folds: `box T; isinst I` and `castclass I` on a known T are constants, `box T; callvirt I.M` is T's own method, `typeof(T).IsEnum/IsValueType` prune branches) | 1 498 deleted and 44 added in `corelib` (`StringBuilder.cs` out; `StringBuilderRuntime.cs` and a `Rune.AsSpan` contract member in) | **Flipped**. Security: bounds, capacity and `ArgumentOutOfRangeException` checks of `Append`/`Insert`/`ToString` are now upstream's own code, not a port; `Buffer.Memmove<T>`'s element count is `checked` (no silent truncation). The speed and size costs of appending strings and numbers are the price; the compiler's own interpolation lowering (the shadow's fast path) cannot stand beside the real handler, closing the gap is a follow-up. |
102102
| System.Memory (`System.Memory`: `ArrayBufferWriter<T>`, `ReadOnlySequence<T>`, `SequenceReader<T>`, `SequencePosition`, `MemoryPool<T>`, `EncodingExtensions`) | 14 of 14 samples that use these types byte-identical to real .NET at -O0, -O2 and `--via-il` (1051, 1250 both, 1251, 1266, 1281, 310, 483, 505, 506, 834, 835, 846, 849); the ownership map's `Memory<T>` blocker (csharpc's `Memory<T>` does not record its array) did not show in any of them. System.Memory defines types and forwards others to CoreLib: it takes the renamed reference twin of #287 (`TwinReference.SupportedAssemblies`), nothing System.Memory-specific in the compiler | an `ArrayBufferWriter` program (1281) 136 376 to 267 264 B (1.96x), a `ReadOnlySequence`/`SequenceReader` program (505) 247 040 to 329 736 B (1.33x), the JSON writer sample (310) 649 736 to 723 184 B (1.11x), a `MemoryPool` program (483) 193 608 to 165 016 B (**0.85x**); `tests/size`: regex 875 664 to 894 096 B and regexoptions 2 097 888 to 2 130 616 B (both within their budgets), the other size programs unchanged | compile cpu 4.1 to 6.7 s (1281), 7.5 to 8.6 s (505), 4.4 to 4.0 s (483), 15.8 to 17.5 s (310) | not measured: no sample has a hot path in these types | 3 146 (`Buffers/ReadOnlySequence.cs` 2 931, `MemoryPool.cs` 87, `Json/ArrayBufferWriter.cs` 88, the `EncodingExtensions` members of `Text/Encoding.cs` 40); `git diff --shortstat`: 1 insertion, 3 146 deletions over 5 files in `src` | **Flipped (#306)**. Security: the framework's own bounds and overflow checks replace the ports' (`ArrayBufferWriter`'s growth, which the port once got wrong (#194), `SequenceReader`'s advance and rewind, `ReadOnlySequence`'s slicing across segments, `EncodingExtensions.GetString` decoding a character split across segments); no native code, no unsafe code of csharpc's. |
103+
| System.Net.Primitives (`IPAddress`, `IPEndPoint`, `EndPoint`, `IPNetwork`; the rest of the assembly was IL already, ADR 0013) | the 46 networking samples (IPAddress/IPEndPoint/IPNetwork, Dns, Socket, HttpClient, SslStream, cookies, credentials) pass at -O0 and -O2 (92 runs), nine of them also with `--via-il`; two new samples recorded from real .NET: 1316 (parsing of untrusted text: inet_aton limits, IPv6 compression and embedded IPv4, scope ids, brackets and ports, the span and UTF-8 overloads, `TryFormat`) and 1317 (`IPEndPoint.Serialize/Create`, `SocketAddress`). **The shadow was silently wrong** on scope ids: `fe80::1%`, `fe80::1%nosuchif0`, `fe80::1%2%3`, `fe80::1%4294967296` and `fe80::1%-1` are valid addresses with scope 0 in .NET and were rejected; it also lacked the span/UTF-8 `Parse`/`TryParse`, `TryFormat` and `Serialize`/`Create` (compile errors). `SocketException` stays a shadow (the socket engine's native error and text), and the two internal classes that P/Invoke libSystem.Native, `SocketAddressPal` and `InterfaceInfoPal`, are a 196-line contract over the socket engine's `sockaddr` layout and `if_nametoindex` (`corelib/Net/SocketAddressPal.cs`) | `tests/size/net.cs` 171 480 to 255 976 B (1.49x; **NativeAOT 1 365 832**), the speed program 148 872 to 246 104 (NativeAOT 1 357 608), sample 195 182 440 to 267 208, 615 175 608 to 259 640: the IL's generic parser and formatter over `char` and `byte`, `UInt128` in `IPNetwork` and the number formatting they reach. Before a general fix in this PR it was 393 536: a boxed `UInt128`'s `ISpanFormattable` slot went through the formatter switch of every primitive (decimal, DateTime with the calendars); now it formats only `UInt128` (`tests/size/json.cs` 1 572 320 to 1 559 040 from the same fix). The other `tests/size` programs are unchanged | `net.cs` 4.8 to 6.3 s user (wall 4.6 to 5.4; NativeAOT publish 9.9 s user, 13.4 wall) | `IPAddress.Parse` + `ToString` + `GetAddressBytes`, 2 M iterations over 8 addresses, median of 7: shadow 336 ms, IL **237 ms (0.71x)**, NativeAOT 295 ms; the checksum equals real .NET | 633 (`Net/IPAddress.cs` 509, `Net/IPNetwork.cs` 124; `SslClientHelloInfo`, a System.Net.Security type, moved to `Net/SslStream.cs`); 196 added for the sockaddr contract | **Flipped (#326)**. Security: the hand-written parsers of untrusted text (`IPAddress`, `IPEndPoint`, `IPNetwork`) are replaced by the framework's own `IPAddressParser`/`IPv4AddressHelper`/`IPv6AddressHelper`, which are fuzzed upstream and bound every component (sample 1316 pins the limits); cookie parsing was already the framework's IL. The new contract code bounds every `sockaddr` access by the buffer length (`ArgumentOutOfRangeException("socketAddressSize")`, libSystem.Native's EFAULT) and passes `if_nametoindex` a NUL-terminated copy. General pieces added on the way: `MemoryExtensions.TryWriteInterpolatedStringHandler` from the CoreLib slice (nested types of a type outside the slice are hoisted), UTF-8 `Parse`/`TryParse` for the remaining integer types, `Vector128` array and span `Create`/`CopyTo`, a program that names any System.Net namespace takes System.Net.Primitives (the corelib's Socket and Dns shadows return its types). |
103104

104105
**Assemblies that define types and also forward CoreLib types (System.Collections, System.Collections.Concurrent)** needed one more piece of the IL
105106
front-end (#275). The implementation forwards `List<T>`, `Queue<T>`, `ConcurrentQueue<T>`... to System.Private.CoreLib, and binding against it in place of

‎src/Csharpc/Frontend/FrameworkIl.cs‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,10 @@ private sealed class Assembly
5353
/// Assemblies with some types csharpc implements itself (corelib shadows) and the rest portable IL:
5454
/// eligible anyway. Every member of a shadowed type is redirected to its shadow (the IL of those types
5555
/// is never lowered, so the native calls in them are not either), and the other types are compiled
56-
/// from IL. System.Net.Primitives: IPAddress, IPEndPoint and EndPoint are shadows (their IL calls
57-
/// libSystem.Native's socket-address shims); Cookie, CookieContainer, CredentialCache,
58-
/// NetworkCredential, DnsEndPoint and HttpVersion are plain C#. System.Net.NetworkInformation: NetworkInterface and its
56+
/// from IL. System.Net.Primitives: SocketException is a shadow (its native error number and text come
57+
/// from the socket engine's OS layer, where the IL calls libSystem.Native); IPAddress, IPEndPoint, IPNetwork,
58+
/// Cookie, CredentialCache and the rest come from IL (SocketAddress's IL calls libSystem.Native's socket-address
59+
/// shims, which are not linked: DllNotFoundException at the call). System.Net.NetworkInformation: NetworkInterface and its
5960
/// property types are shadows (getifaddrs / GetAdaptersAddresses in the corelib); PhysicalAddress, NetworkInformationException
6061
/// and the rest the program may name (a List&lt;PhysicalAddress&gt; in ASP.NET Core) come from IL.
6162
/// </summary>
@@ -193,11 +194,11 @@ void Add(Assembly a, string reason)
193194
if (index.TryGetValue(dependency, out var needed) && needed.Ineligible is null && Unusable(needed) is null)
194195
Add(needed, "the corelib uses its types");
195196

196-
// The corelib's sockets (shadows) take and test types of System.Net.Primitives that are plain IL there
197-
// (DnsEndPoint, for Socket.Connect(EndPoint)): a program that names System.Net.Sockets needs them, even
198-
// when it never names System.Net.
199-
if (Named("System.Net.Sockets") && index.TryGetValue("System.Net.Primitives", out var primitives) && primitives.Ineligible is null)
200-
Add(primitives, "the corelib's sockets use its DnsEndPoint");
197+
// The corelib's networking shadows (Socket, Dns, NetworkInterface, SslStream) take and return types of
198+
// System.Net.Primitives, which come from its IL (IPAddress, IPEndPoint, DnsEndPoint): a program that names any
199+
// System.Net namespace needs them, even when it never names one of those types (Dns.GetHostAddresses).
200+
if (Named("System.Net") && index.TryGetValue("System.Net.Primitives", out var primitives) && primitives.Ineligible is null)
201+
Add(primitives, "the corelib's networking types use its IPAddress and EndPoint types");
201202

202203
// Closure: what the IL assemblies (the program's and the chosen framework ones) reference.
203204
var queue = new Queue<(string Name, string[] References)>();

0 commit comments

Comments
 (0)