Data Corruption in System.Buffers.Text.Base64.EncodeToUtf8

[jeffhandley]

We started observing intermittent CI failures on tests involving Base64 encoding in late June. This is recognized as a high-risk potential regression lurking in the .NET 10 release, and we need to find the root cause of this in time to resolve it before the .NET 10 RC1 release.

Here are the CI failures associated with this root cause:

1. Cryptography.Tests: No PEM encoded data found #117163
   2. WasmTestOnChrome-ST-System.Memory.Tests errors #117169
2. System.Xml.Tests failing #117614
3. System.Security.Cryptography.Rsa.Tests.RSAXml Test Suite Fails with Strings Differ/Containers Differ #117665

Both #117163 and #117169 were reported on June 30, which is the earliest recognized signal of this issue. As was noted in #117665:

That seems... bad?

-f6 ba 82 83 26 0c 39 91 1b 8b 5d b1 8a 0f f3 6a 78 d5 59 a8 0d 64 29 3d d0 0c 35 87 56 00 ...
+f4 0a 80 80 06 00 38 01 00 88 0d 80 88 0f c0 68 08 c0 58 08 00 64 09 00 d0 0c 00 84 06 00 ...
It's like something is clearing every other 6 bits?

• F4BA XOR F40A is 02B0: 0b0000_0010_1011_0000
• 8B5D XOR 880D is 0350: 0b0000_0011_0101_0000

Since base64 is a 6-bit map... I wonder if this is a base64 encoding problem.

This is happening across multiple architectures and operating systems, as well as across different physical machines.

• Machine a00DH2L running Windows-2016Server-10.0.14393-SP0
• Machine a009LPH running Linux-6.8.0-1030-azure-x86_64-with-glibc2.35
• Machine a009QWZ using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly

@eiriktsarpalis conducted a preliminary investigation, reporting:

• The failing tests seem to all share a call to System.Buffers.Text.Base64.EncodeToUtf8, with that method producing corrupted data
• Some of the tests exercise the API through the System.Convert APIs, which only use the Base64 class implementation if vectorization is available and applicable
  

  runtime/src/libraries/System.Private.CoreLib/src/System/Convert.cs

  Lines 2451 to 2454 in 9df2042

  	if (Vector128.IsHardwareAccelerated && !insertLineBreaks && bytes.Length >= Base64VectorizationLengthThreshold)
  	{
  	ToBase64CharsLargeNoLineBreaks(bytes, chars, charLengthRequired);
  	}

• The relevant implementation in System/Buffers/Text/Base64Helper/Base64EncoderHelper.cs hasn't been updated since October 2024

/cc @artl93 @jeffschwMSFT @marek-safar @lewing for visibility/awareness

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-buffers
See info in area-owners.md if you want to be subscribed.

[jeffhandley]

With the issue ostensibly narrowed down to System.Buffers.Text.Base64.EncodeToUtf8, a potential next step is to write a fuzz test against this method with instrumentation that would capture a memory dump when the test fails. Other instrumentation within the failing tests might also yield clues.

[jeffhandley]

@tannergooding also suggested that once we can repro this, we should run the repro with:

• DOTNET_EnableAVX512=0 to disable vectorization (edited to reflect this is case-sensitive on non-Windows platforms)

If we narrow it down to a vectorization issue, we can use the following to narrow it down further with AVX512 enabled:

• DOTNET_ EnableEmbeddedBroadcast=0
• DOTNET_ EnableEmbeddedMasking=0

[EgorBo]

• DOTNET_EnableAvx512=0 to disable vectorization

A note: it's DOTNET_EnableAVX512=0 since it's case sensitive on non-windows platforms

[jeffhandley]

I've reviewed all of the failures and have only found occurrences on Amd64 hardware. There was one that appeared to be Arm64, but it looks to be a false-positive match on the failure.

[bartonjs]

I don't know if it's useful for differential diagnosis, but in case it is, I've never seen a repro on this CPU:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 2
model name      : AMD FX(tm)-8350 Eight-Core Processor
stepping        : 0
microcode       : 0x6000852
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 16
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed ibpb_no_ret
bogomips        : 8035.70
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

[EgorBo]

Interesting, seems to be some old CPU without AVX2 (AVX only).

[bartonjs]

@EgorBo Might be old, but it does what I need it to 😄. Since there seems to be speculation that it has to do with JIT optimizations and/or intrinsics-enhanced codepaths, I figured I'd share a case where the corruption hasn't ever appeared.

[EgorBo]

I tried to optimistically repro it by comparing Base64.EncodeToUtf8 output with a naive version generated by GPT for random inputs - was not able to reproduce any mismatch (also tested various EnableAVX knobs). https://gist.github.com/EgorBo/1bf17a02c94d587917d81789d53af847

Probably also makes sense to try it on inputs created with VirtualAlloc to detect cases when it reads out of bounds (if it does).

[EgorBo]

I think we need a dump and/or more info about input size it failed on

[EgorBo]

Hm.. probably found something, this repro fails on Checked runtime built from source (main) with:

Unhandled exception. System.Exception: Mismatch in Base64 encoding!
   at Program.<>c.<Main>b__0_0(Object _) in C:\Users\egorb\source\repos\ConsoleApp19\ConsoleApp19\Program.cs:line 34
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

but only with AVX512