C# Benchmark .NET 3.5, 4.0 and 4.5

Benchmark 3.5, 4.0, 4.5. The .NET Framework has become faster.

Version 4.0 executes programs faster than the 3.5 SP1. Microsoft has shown how the size of some deployments has decreased with the new version. I investigate runtime performance.

Example. To test, I developed a program that uses the Dictionary collection, a for-loop, some branches, multiplications, and two Math methods. In the Dictionary lookups, unsafe code is also executed as part of the hash code computation.

Unsafe

Note: There may have been some changes in the Dictionary collection and Math methods that were not separately tested.

Math.SqrtMath.Pow

C# program that tests performance across Frameworks

using System;
using System.Collections.Generic;
using System.Diagnostics;

class Program
{
    static Dictionary<string, bool> _dict = new Dictionary<string, bool>();
    const int _max = 50000000;
    static void Main()
    {
	_dict.Add("cat", true);
	_dict.Add("dog", false);

	Console.WriteLine(Method(1000));

	var s1 = Stopwatch.StartNew();
	for (int i = 0; i < _max; i++)
	{
	    Method(i);
	}
	s1.Stop();
	Console.WriteLine(((double)(s1.Elapsed.TotalMilliseconds * 1000 * 1000) /
	    _max).ToString("0.00 ns"));
	Console.Read();
    }

    static int Method(int i)
    {
	int value = 0;
	// Dictionary lookup.
	bool a;
	if (_dict.TryGetValue("cat", out a))
	{
	    value++;
	}
	bool b;
	if (_dict.TryGetValue("nope", out b))
	{
	    value++;
	}
	// Multiply by two ten times.
	for (int y = 0; y < 10; y++)
	{
	    // Use parameter.
	    if (y != i)
	    {
		value *= 2;
	    }
	}
	// Take square root.
	int result = (int)Math.Sqrt(value);
	// Cube it.
	result = (int)Math.Pow(result, 3);
	return result;
    }
}

When run on .NET 3.5 SP1

160.47
170.07
162.52
175.43
161.66
Average: 166.03

When run on .NET 4.0

158.19
159.69
163.40
153.16
159.94
Average: 158.88

Let me explain some of my testing methodology. I compiled this program in Visual Studio 2008 and also Visual Studio 2010 in Release mode. Then, I directly opened the executable. I recorded the benchmark result.

Benchmark

I was surprised by the results. The average time for .NET 3.5 SP1 to execute this method was 166 nanoseconds, while .NET 4.0 required only 158 nanoseconds. Therefore, in this benchmark .NET 4.0 performed about 5% faster.

Note: The average was incorrect in the first version. Sylvain Gueant wrote in to report the error.

Update. An important event happened in 2012. Microsoft released the .NET Framework 4.5. I compiled and ran the same program on the same PC. I used a new Windows operating system, Windows 8, but the same hardware.

And: The performance was somewhere in between .NET 3.5 SP1 and .NET 4.0. It is unclear if performance regressed.

Warning: Other factors on the test system changed, so these numbers cannot be compared to the previous two sets.

When run on .NET 4.5

166.14
168.22
162.26
166.21
165.58
Average: 165.68

Also, some programs I regularly run improved in performance with the new .NET 4.5 Framework. I received a small boost moving to Windows 8, and also with .NET 4.5. It is unclear which programs benefit from the upgrade, and which do not.

Tip: An optimization that improves real-world programs could slow down contrived benchmarks. Important programs tend to have many methods.

So: Optimization is undecidable. And it involves many tradeoffs. Some programs become faster, and others slower.

Optimizations

Summary. Equivalent programs compiled and executed using the .NET 4.0 Framework may execute faster than those based on .NET 3.5 SP1. Your software will probably get an easy performance boost if you move it to a newer Framework version.

And: Because C# is generally backward compatible, this move should be fairly simple.

Related Links