Remove FileStream construction cost out of unrelated FileStream benchmarks #841

adamsitnik · 2019-07-25T12:22:56Z

git clone https://github.com/dotnet/performance.git
cd performance
# if you don't have cli installed and want python script to download the latest cli for you
py .\scripts\benchmarks_ci.py -f netcoreapp2.2 netcoreapp3.0 --filter System.IO.Tests.Perf_FileStream.ReadByte
# if you do
dotnet run -p .\src\benchmarks\micro\MicroBenchmarks.csproj -c Release -f netcoreapp2.2 --runtimes netcoreapp2.2 netcoreapp3.0 --filter System.IO.Tests.Perf_FileStream.ReadByte

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 512, TotalSize: 200000)

Conclusion	Base	Diff	Base/Diff	Modality	Operating System	Arch	Processor Name	Base Runtime	Diff Runtime
Slower	811254.07	856664.60	0.95		ubuntu 18.04	64bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1135065.67	1252114.71	0.91		Windows 10.0.18362	64bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1269167.75	1383255.57	0.92		ubuntu 16.04	64bit	Intel Xeon CPU E5-2673 v4 2.30GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-01
Slower	1286991.20	1447953.07	0.89		ubuntu 18.04	64bit	Intel Xeon CPU E5-2673 v4 2.30GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-01
Faster	1090675.72	1029506.42	1.06		macOS Mojave 10.14.5	64bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Same	1374811.62	1477960.48	0.93		Windows 10.0.18362	64bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1028851.74	1207637.50	0.85		Windows 10.0.18362	64bit	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Same	1494007.70	1497116.45	1.00		Windows 10.0.18362	64bit	AMD Ryzen 7 1800X	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	951703.90	1092421.54	0.87		Windows 10.0.18362	64bit	AMD Ryzen 9 3900X	.NET Core 2.2.6	.NET Core 3.0.0-preview8-28373-12
Slower	1053184.30	1276481.15	0.83		Windows 10.0.18362	32bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1298226.59	1487528.35	0.87	bimodal	Windows 10.0.18362	32bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-28373-17

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 200000, TotalSize: 200000)

Conclusion	Base	Diff	Base/Diff	Modality	Operating System	Arch	Processor Name	Base Runtime	Diff Runtime
Same	814052.98	856351.86	0.95		ubuntu 18.04	64bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Same	1174059.04	1220253.92	0.96		Windows 10.0.18362	64bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1290140.05	1383311.57	0.93		ubuntu 16.04	64bit	Intel Xeon CPU E5-2673 v4 2.30GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-01
Slower	1278546.35	1573405.84	0.81		ubuntu 18.04	64bit	Intel Xeon CPU E5-2673 v4 2.30GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-01
Faster	1093245.98	1028285.87	1.06		macOS Mojave 10.14.5	64bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Same	1356568.33	1408530.34	0.96		Windows 10.0.18362	64bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	964374.36	1203574.69	0.80		Windows 10.0.18362	64bit	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Same	1476829.06	1532662.36	0.96		Windows 10.0.18362	64bit	AMD Ryzen 7 1800X	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	951317.08	1093435.65	0.87		Windows 10.0.18362	64bit	AMD Ryzen 9 3900X	.NET Core 2.2.6	.NET Core 3.0.0-preview8-28373-12
Slower	1098744.50	1273262.66	0.86	several?	Windows 10.0.18362	32bit	Intel Xeon CPU E5-1650 v4 3.60GHz	.NET Core 2.2.6	.NET Core 3.0.0-preview8-27919-09
Slower	1208625.41	1474770.06	0.82		Windows 10.0.18362	32bit	Intel Core i7-5557U CPU 3.10GHz (Broadwell)	.NET Core 2.2.6	.NET Core 3.0.0-preview8-28373-17

/cc @danmosemsft @billwert @DrewScoggins

JeremyKuhne · 2019-07-25T18:51:35Z

Nothing sticks out with a cursory glance at the sources. :( @stephentoub, wouldn't the buffer be a good candidate for the new internal GC non zeroing API?

cc: @carlossanlop

stephentoub · 2019-07-25T18:54:40Z

wouldn't the buffer be a good candidate for the new internal GC non zeroing API

I'd prefer to use ArrayPool :)

But... that's also not relevant to ReadByte, right (other than that it uses the buffer)? Cost of getting the buffer would only affect first access... presumably we're measuring already-initialized timing here? If not, I'd argue the test is bad :)

JeremyKuhne · 2019-07-25T18:59:09Z

performance/src/benchmarks/micro/corefx/System.IO.FileSystem/Perf.FileStream.cs

Lines 36 to 49 in 0c626c0

    
           public int ReadByte() 
        
           { 
        
               int result = default; 
        
               using (FileStream reader = new FileStream(_filePath, FileMode.Open, FileAccess.Read, FileShare.Read, DefaultBufferSize, FileOptions.None)) 
        
               { 
        
                   for (int i = 0; i < TotalSize; i++) 
        
                   { 
        
                       result += reader.ReadByte(); 
        
                   } 
        
               } 
        
               return result; 
        
           }

stephentoub · 2019-07-25T19:01:00Z

Thanks. With "only" 200K iterations inside the test, the cost of creating the FileStream and everything it does, syscalls and allocation and all, could certainly affect the overall perf of the test, considering that ReadByte on most accesses is a few checks and an array access.

JeremyKuhne · 2019-08-15T18:00:27Z

We need to update this test to take the construction out of the measurement and make sure we have tests that specifically measure FileStream creation.

stephentoub transferred this issue from dotnet/corefx Jul 25, 2019

danmosemsft transferred this issue from dotnet/coreclr Aug 31, 2019

danmosemsft changed the title ~~Performance regression: FileStream.ReadByte 5-10% slower~~ Remove FileStream construction cost out of unrelated FileStream benchmarks Aug 31, 2019

danmosemsft added good first issue up-for-grabs labels Sep 3, 2019

dotnet / performance

Remove FileStream construction cost out of unrelated FileStream benchmarks #841

Remove FileStream construction cost out of unrelated FileStream benchmarks #841

adamsitnik commented Jul 25, 2019

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 512, TotalSize: 200000)

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 200000, TotalSize: 200000)

JeremyKuhne commented Jul 25, 2019

stephentoub commented Jul 25, 2019 •

edited

JeremyKuhne commented Jul 25, 2019

stephentoub commented Jul 25, 2019

JeremyKuhne commented Aug 15, 2019

dotnet / performance

Join GitHub today

GitHub is where the world builds software

Remove FileStream construction cost out of unrelated FileStream benchmarks #841

Remove FileStream construction cost out of unrelated FileStream benchmarks #841

Comments

adamsitnik commented Jul 25, 2019

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 512, TotalSize: 200000)

System.IO.Tests.Perf_FileStream.ReadByte(BufferSize: 200000, TotalSize: 200000)

JeremyKuhne commented Jul 25, 2019

stephentoub commented Jul 25, 2019 • edited

JeremyKuhne commented Jul 25, 2019

stephentoub commented Jul 25, 2019

JeremyKuhne commented Aug 15, 2019

Essential cookies

Always active

Analytics cookies

stephentoub commented Jul 25, 2019 •

edited