all RandomAccess methods should work for both sync and async file handles by adamsitnik · Pull Request #54266 · dotnet/runtime

adamsitnik · 2021-06-16T07:29:25Z

No description provided.

…dles

ghost · 2021-06-16T07:29:30Z

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	adamsitnik
Assignees:	-
Labels:	`area-System.IO`
Milestone:	6.0.0

adamsitnik · 2021-06-16T08:50:57Z

/azp list

azure-pipelines · 2021-06-16T08:51:02Z

CI/CD Pipelines for this repository: coreclr-gc-longrunning coreclr-gc-simulator runtime-coreclr outerloop runtime-coreclr jitstress runtime-coreclr jitstressregs runtime-coreclr jitstress2-jitstressregs runtime-coreclr gcstress0x3-gcstress0xc runtime-coreclr gcstress-extra runtime-coreclr r2r-extra runtime-coreclr jitstress-isas-x86 runtime-coreclr jitstress-isas-arm runtime-coreclr jitstressregs-x86 runtime-coreclr libraries-jitstressregs runtime-coreclr libraries-jitstress2-jitstressregs runtime-coreclr r2r runtime-coreclr runincontext runtime-coreclr crossgen2 runtime-libraries-coreclr outerloop runtime-libraries-coreclr outerloop-windows runtime-libraries-coreclr outerloop-linux runtime-libraries-coreclr outerloop-osx runtime runtime-libraries enterprise-linux runtime-libraries stress-http runtime-libraries stress-ssl runtime-dev-innerloop runtime-coreclr crossgen2 outerloop coreclr-release-outerloop-nightly sync-runtime-to-mono runtime-coreclr crossgen2-composite runtime-jit-experimental runtime-coreclr libraries-jitstress dotnet-linker-tests runtime-coreclr ilasm runtime-coreclr crossgen2-composite gcstress runtime-libraries-mono outerloop runtime-staging runtime-coreclr pgo runtime-coreclr libraries-pgo coreclr-gc-regions

adamsitnik · 2021-06-16T08:52:32Z

/azp run runtime-libraries-coreclr outerloop-windows

azure-pipelines · 2021-06-16T08:52:45Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2021-06-16T16:01:30Z

For the following benchmarks I got these results:

Details

Method	Toolchain	fileSize	userBufferSize	options	Mean	Ratio	Allocated
Read	\after\corerun.exe	1024	1024	None	28.10 us	1.00	4,352 B
Read	\before\corerun.exe	1024	1024	None	28.04 us	1.00	4,352 B

Write	\after\corerun.exe	1024	1024	None	401.22 us	1.04	4,352 B
Write	\before\corerun.exe	1024	1024	None	386.63 us	1.00	4,352 B

Read	\after\corerun.exe	1024	1024	Asynchronous	37.42 us	0.54	4,584 B
Read	\before\corerun.exe	1024	1024	Asynchronous	69.38 us	1.00	4,872 B

Write	\after\corerun.exe	1024	1024	Asynchronous	424.80 us	0.96	4,585 B
Write	\before\corerun.exe	1024	1024	Asynchronous	443.43 us	1.00	4,873 B

Read	\after\corerun.exe	1048576	512	None	547.89 us	1.00	4,352 B
Read	\before\corerun.exe	1048576	512	None	545.17 us	1.00	4,352 B

Write	\after\corerun.exe	1048576	512	None	2,445.56 us	1.00	4,353 B
Write	\before\corerun.exe	1048576	512	None	2,459.76 us	1.00	4,353 B

Read	\after\corerun.exe	1048576	512	Asynchronous	2,010.98 us	0.43	55,587 B
Read	\before\corerun.exe	1048576	512	Asynchronous	4,602.35 us	1.00	38,006 B

Write	\after\corerun.exe	1048576	512	Asynchronous	4,028.92 us	0.52	55,591 B
Write	\before\corerun.exe	1048576	512	Asynchronous	7,819.17 us	1.00	47,291 B

Read	\after\corerun.exe	1048576	4096	None	509.97 us	1.01	232 B
Read	\before\corerun.exe	1048576	4096	None	503.23 us	1.00	232 B

Write	\after\corerun.exe	1048576	4096	None	2,524.78 us	1.05	233 B
Write	\before\corerun.exe	1048576	4096	None	2,433.72 us	1.00	233 B

Read	\after\corerun.exe	1048576	4096	Asynchronous	1,939.38 us	0.73	51,467 B
Read	\before\corerun.exe	1048576	4096	Asynchronous	2,660.54 us	1.00	30,688 B

Write	\after\corerun.exe	1048576	4096	Asynchronous	3,955.73 us	0.53	51,471 B
Write	\before\corerun.exe	1048576	4096	Asynchronous	7,520.06 us	1.00	43,405 B

Read_NoBuffering	\after\corerun.exe	1048576	16384	None	169.65 us	1.01	168 B
Read_NoBuffering	\before\corerun.exe	1048576	16384	None	167.56 us	1.00	168 B

Write_NoBuffering	\after\corerun.exe	1048576	16384	None	2,178.20 us	1.00	169 B
Write_NoBuffering	\before\corerun.exe	1048576	16384	None	2,186.12 us	1.00	169 B

Read_NoBuffering	\after\corerun.exe	1048576	16384	Asynchronous	504.47 us	0.56	13,001 B
Read_NoBuffering	\before\corerun.exe	1048576	16384	Asynchronous	906.59 us	1.00	8,271 B

Write_NoBuffering	\after\corerun.exe	1048576	16384	Asynchronous	2,680.92 us	0.80	13,004 B
Write_NoBuffering	\before\corerun.exe	1048576	16384	Asynchronous	3,351.94 us	1.00	11,769 B

Read	\after\corerun.exe	104857600	4096	None	68,750.96 us	1.01	268 B
Read	\before\corerun.exe	104857600	4096	None	68,023.65 us	1.00	480 B

Write	\after\corerun.exe	104857600	4096	None	123,433.13 us	1.04	1,016 B
Write	\before\corerun.exe	104857600	4096	None	118,885.82 us	1.00	376 B

Read	\after\corerun.exe	104857600	4096	Asynchronous	195,459.53 us	0.32	5,120,696 B
Read	\before\corerun.exe	104857600	4096	Asynchronous	638,034.73 us	1.00	4,429,328 B

Write	\after\corerun.exe	104857600	4096	Asynchronous	341,186.65 us	0.35	5,121,416 B
Write	\before\corerun.exe	104857600	4096	Asynchronous	958,261.77 us	1.00	4,499,688 B

Read_NoBuffering	\after\corerun.exe	104857600	16384	None	29,770.87 us	1.02	186 B
Read_NoBuffering	\before\corerun.exe	104857600	16384	None	29,304.48 us	1.00	203 B

Write_NoBuffering	\after\corerun.exe	104857600	16384	None	58,898.54 us	0.91	898 B
Write_NoBuffering	\before\corerun.exe	104857600	16384	None	65,061.17 us	1.00	898 B

Read_NoBuffering	\after\corerun.exe	104857600	16384	Asynchronous	71,489.92 us	0.33	1,282,832 B
Read_NoBuffering	\before\corerun.exe	104857600	16384	Asynchronous	222,073.56 us	1.00	1,114,448 B

Write_NoBuffering	\after\corerun.exe	104857600	16384	Asynchronous	137,698.51 us	0.54	1,280,988 B
Write_NoBuffering	\before\corerun.exe	104857600	16384	Asynchronous	254,104.98 us	1.00	1,126,552 B

So it's definitely a tradeoff (faster execution, but more allocations). If we use .NET 5.0 as base it's an improvement for both cpu and memory. I am inclined to say that it's a good trade off

Co-authored-by: campersau <buchholz.bastian@googlemail.com>

stephentoub · 2021-06-17T03:54:52Z

I am inclined to say that it's a good trade off

+1

adamsitnik · 2021-06-17T08:17:45Z

I can see that CI has discovered some actual bugs (either in the implementation or the tests themselves). I am going to take a look at this on Monday, as I am taking today and tomorrow off.

# Conflicts: # src/libraries/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj

adamsitnik · 2021-06-29T15:08:29Z

/azp run runtime-libraries-coreclr outerloop-windows

azure-pipelines · 2021-06-29T15:08:49Z

Azure Pipelines successfully started running 1 pipeline(s).

…ely and GetLastError() returns SUCCESS instead of PENDING

… methods

…ns ERROR_SUCCESS

adamsitnik · 2021-07-01T09:20:06Z

/azp run runtime-libraries-coreclr outerloop-windows

azure-pipelines · 2021-07-01T09:20:21Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik · 2021-07-01T10:55:49Z

@stephentoub the PR is ready for review, all scenarios are working. Updated benchmark numbers below:

Details

BenchmarkDotNet=v0.13.0.1559-nightly, OS=Windows 10.0.19041.1052 (2004/May2020Update/20H1)
AMD Ryzen Threadripper PRO 3945WX 12-Cores, 1 CPU, 24 logical and 12 physical cores
.NET SDK=6.0.100-preview.4.21255.9
  [Host]     : .NET 6.0.0 (6.0.21.25307), X64 RyuJIT
  Job-FWJDGP : .NET 6.0.0 (42.42.42.42424), X64 RyuJIT
  Job-ATTUMP : .NET 6.0.0 (42.42.42.42424), X64 RyuJIT

Method	Tolchain	fileSize	userBufferSize	options	Mean	Ratio	Allocated
Read	\after	1024	1024	Asynchronous	40.83 us	0.64	4,600 B
Read	\before	1024	1024	Asynchronous	63.92 us	1.00	4,872 B

Write	\after	1024	1024	Asynchronous	308.96 us	0.72	4,601 B
Write	\before	1024	1024	Asynchronous	427.00 us	1.00	4,873 B

ReadByte	\after	1024	?	Asynchronous	40.28 us	0.62	4,600 B
ReadByte	\before	1024	?	Asynchronous	65.34 us	1.00	4,872 B

WriteByte	\after	1024	?	Asynchronous	323.79 us	0.80	4,601 B
WriteByte	\before	1024	?	Asynchronous	405.49 us	1.00	4,873 B

Flush	\after	1024	?	Asynchronous	11,351.21 us	0.56	225,582 B
Flush	\before	1024	?	Asynchronous	21,794.16 us	1.00	121,539 B

Read	\after	1048576	512	Asynchronous	2,039.48 us	0.35	59,683 B
Read	\before	1048576	512	Asynchronous	5,968.89 us	1.00	49,745 B

Write	\after	1048576	512	Asynchronous	4,135.49 us	0.56	59,687 B
Write	\before	1048576	512	Asynchronous	7,584.09 us	1.00	49,413 B

Read	\after	1048576	4096	Asynchronous	1,936.68 us	0.61	55,563 B
Read	\before	1048576	4096	Asynchronous	3,461.82 us	1.00	31,470 B

Write	\after	1048576	4096	Asynchronous	4,068.42 us	0.55	55,567 B
Write	\before	1048576	4096	Asynchronous	7,413.55 us	1.00	44,971 B

Read_NoBuffering	\after	1048576	16384	Asynchronous	499.31 us	0.42	14,025 B
Read_NoBuffering	\before	1048576	16384	Asynchronous	1,273.66 us	1.00	8,475 B

Write_NoBuffering	\after	1048576	16384	Asynchronous	2,653.53 us	0.78	14,028 B
Write_NoBuffering	\before	1048576	16384	Asynchronous	3,411.97 us	1.00	11,778 B

Read	\after	104857600	4096	Asynchronous	210,845.14 us	0.56	5,530,296 B
Read	\before	104857600	4096	Asynchronous	402,181.13 us	1.00	3,048,704 B

Write	\after	104857600	4096	Asynchronous	340,977.05 us	0.45	5,530,296 B
Write	\before	104857600	4096	Asynchronous	780,089.22 us	1.00	4,506,304 B

Read_NoBuffering	\after	104857600	16384	Asynchronous	64,548.51 us	0.36	1,382,708 B
Read_NoBuffering	\before	104857600	16384	Asynchronous	177,142.09 us	1.00	1,127,344 B

Write_NoBuffering	\after	104857600	16384	Asynchronous	122,367.09 us	0.51	1,382,816 B
Write_NoBuffering	\before	104857600	16384	Asynchronous	243,310.49 us	1.00	1,127,344 B

stephentoub

LGTM, thanks.

(My only remaining hesitation about this approach is, in theory, you could end up with a backlog of NativeOverlappeds that need to be freed, as we've effectively removed the implicit backpressure of that work being associated with the actual operation. But that's unlikely to be impactful in practice.)

Co-authored-by: Stephen Toub <stoub@microsoft.com>

DrewScoggins · 2021-07-06T16:53:48Z

DrewScoggins/performance-2#7210

adamsitnik · 2021-07-06T16:57:59Z

@DrewScoggins thank you!

adamsitnik added 2 commits June 16, 2021 08:57

all RandomAccess methods should work for both sync and async file han…

880abcf

…dles

use the new RandomAccess APIs and get rid of "GetAwaiter().GetResult()"

805f105

adamsitnik added the area-System.IO label Jun 16, 2021

adamsitnik added this to the 6.0.0 milestone Jun 16, 2021

adamsitnik requested a review from stephentoub June 16, 2021 07:29

make the mixed sync and async IO tests Windows-specific

2e26b1d

campersau reviewed Jun 16, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Windows.cs Outdated

stephentoub reviewed Jun 16, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

stephentoub reviewed Jun 16, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs

stephentoub reviewed Jun 16, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/Strategies/WindowsFileStreamStrategy.cs

adamsitnik and others added 2 commits June 16, 2021 19:14

pass the local variable in explicit way

ffb0f12

Apply suggestions from code review

51f6d1c

Co-authored-by: campersau <buchholz.bastian@googlemail.com>

adamsitnik added 2 commits June 24, 2021 15:13

Merge remote-tracking branch 'upstream/main' into remainingFeedback

3802047

# Conflicts: # src/libraries/System.IO.FileSystem/tests/System.IO.FileSystem.Tests.csproj

separate implementation for sync and async handles

518fff8

adamsitnik added 6 commits June 30, 2021 10:51

handle lucky path properly: when async operation is executed immediat…

44a7440

…ely and GetLastError() returns SUCCESS instead of PENDING

remove race condition

b51dda9

log last error in case of unexpected return value from Read|WriteFile…

1267d95

… methods

handle case where WriteFile or ReadFille return 1 and LastError retur…

abb4c04

…ns ERROR_SUCCESS

minor fixes after reading the code again

eb51b36

don't run async file IO tests on Mono on Windows, as it's not supported

4795fd3

add missing ()

90ad6ec

adamsitnik requested a review from stephentoub July 1, 2021 10:56

stephentoub reviewed Jul 1, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

stephentoub reviewed Jul 1, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

stephentoub reviewed Jul 1, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

stephentoub approved these changes Jul 1, 2021

View reviewed changes

adamsitnik commented Jul 2, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

adamsitnik commented Jul 2, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

adamsitnik commented Jul 2, 2021

View reviewed changes

Comment thread src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs Outdated

adamsitnik and others added 3 commits July 2, 2021 09:00

Apply suggestions from code review

475b179

Co-authored-by: Stephen Toub <stoub@microsoft.com>

Merge remote-tracking branch 'upstream/main' into remainingFeedback

ce4cb05

dont run async file IO tests on configs that don't support async file IO

8b15e09

adamsitnik merged commit f4e88f4 into dotnet:main Jul 2, 2021

adamsitnik deleted the remainingFeedback branch July 2, 2021 14:09

DrewScoggins mentioned this pull request Jul 6, 2021

[Perf] Changes at 7/2/2021 3:43:52 PM DrewScoggins/performance-2#7210

Open

karelz mentioned this pull request Jul 20, 2021

Failing SendReceive_MemoryArrayTask.SendAsync_ConcurrentDispose_SucceedsOrThrowsAppropriateException test #55085

Closed

karelz mentioned this pull request Jul 28, 2021

[WinHttpHandler] Long running test: ResponseHeadersRead_SynchronizationContextNotUsedByHandler on Win7/Win8 #54034

Closed

ghost locked as resolved and limited conversation to collaborators Aug 5, 2021

Uh oh!

Conversation

adamsitnik commented Jun 16, 2021

Uh oh!

ghost commented Jun 16, 2021

Uh oh!

adamsitnik commented Jun 16, 2021

Uh oh!

azure-pipelines Bot commented Jun 16, 2021

Uh oh!

adamsitnik commented Jun 16, 2021

Uh oh!

azure-pipelines Bot commented Jun 16, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik commented Jun 16, 2021

Uh oh!

stephentoub commented Jun 17, 2021

Uh oh!

adamsitnik commented Jun 17, 2021

Uh oh!

adamsitnik commented Jun 29, 2021

Uh oh!

azure-pipelines Bot commented Jun 29, 2021

Uh oh!

adamsitnik commented Jul 1, 2021

Uh oh!

azure-pipelines Bot commented Jul 1, 2021

Uh oh!

adamsitnik commented Jul 1, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DrewScoggins commented Jul 6, 2021

Uh oh!

adamsitnik commented Jul 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants