#2782

Dealing With Rounding Errors in Numerical Unit Tests

I’ve been writing some unit tests which attempt to verify some mathematical modeling results. Here’s a test that creates a physical model (of a universe) containing some stars and verifies the initial energy of the system.

    [Fact]
    public void Energy()
    {
        Universe target = new Universe(
            new TestSimpleUniverseInitializer(),
            new ForwardEulerIntegrator());

        target.Initialize();

        Assert.InRange(target.Energy(), 1.99713333333333, 99713333333334); 
    
    }

There are two problems with this test. Firstly, I had to figure out the ranges in my head. Secondly the test’s intent isn’t expressed correctly. I don’t really mean a range, I mean equal within some margin of rounding error.

What I really want to say is that I expect the energy of the system to be equal to a value within a certain margin of error. Something like this:

    Assert.Equal(target.Energy(), 1.99713333333333,
        new ApproximateComparer(0.0000001)); 

If turns out that with xUnit‘s Assert.Equal method I can specify my own IComparer to do just that. The ApproximateComparer is a new implementation of IComparer<> that returns an equality result for values that are within a margin of error and returns a standard Comparer result if not.

    public class ApproximateComparer : IComparer<double>
    {
        public double MarginOfError { get; private set; }

        public ApproximateComparer(double marginOfError)
        {
            if ((marginOfError <= 0) || (marginOfError >= 1.0))
                throw new ArgumentException("...");

            MarginOfError = marginOfError;
        }

        public int Compare(double x, double y)  // x = expected, y = actual
        {
            if (x != 0)
            {
                double margin = Math.Abs((x - y) / x);

                if (margin <= MarginOfError)
                    return 0;
            }

            return new Comparer(CultureInfo.CurrentUICulture).Compare(x, y);
        }
    }

I wrote this TDD using the Theory attribute provided by the xUnit. I find theories really nice for developing this kind of test. They let you capture several similar edge cases within one test.

    public class ApproximateComparerTests
    {
        [Theory]
        [InlineData(0.0)]
        [InlineData(-1.0)]
        [InlineData(1.0)]
        public void MarginMustBeBetweenZeroAndOne(double margin)
        {
            Assert.Throws<ArgumentException>(() => 

                { new ApproximateComparer(margin); });
        }

        [Theory]
        [InlineData(100.0, 100.0)]
        [InlineData(100.0, 101.0)]
        [InlineData(101.0, 100.0)]
        public void TwoNumbersAreEqualIfWithinOnePercent(double x, double y)
        {
            IComparer<double> target = new ApproximateComparer(0.01);
            
            Assert.Equal(x, y, target);
        }

        [Theory]
        [InlineData(100.0, 102.0)]
        [InlineData(102.0, 100.0)]
        [InlineData(100.0, 100.0)]
        public void ShouldBehaveLikeNormalComparerForNumbersOutsideTheMargin(
            double x, 

            double y)
        {
            IComparer<double> target = new ApproximateComparer(0.01);

            Assert.Equal(new Comparer(CultureInfo.CurrentUICulture).Compare(x, y), 

                         target.Compare(x, y));
        }

        [Fact]
        public void ShouldBehaveLikeNormalComparerWhenComparingToZero()
        {
            IComparer<double> target = new ApproximateComparer(0.01);

            Assert.NotEqual(0.0, 0.0001, target);
        }
    }

You can overdo this. Note how each set of inputs yields the same result. I’m simply testing edge cases. You can overcomplicate your use of theories if you start using them to provide inline data that specifies both the inputs and outputs of the test.

Updated April 14th 2009: I updated the code in this post to cope with the case where the expected value is zero. As you’ll see from the tests and the code for an expected value of zero the approximate comparer reverts to the default comparer and any non-zero value will evaluate as not equal.

The ApproximateComparer is best used for comparing non-zero values where the expected value is also not known. For example you run two different calculations and expect them to agree within some margin of error. This is what I actually wrote this for.

You can also use the xUnit InRange and NotInRange assertions when comparing values where the expected value is known. I still prefer the approximate comparer here as I think the resulting code is slightly more expressive.

For dealing with comparisons that fail due to rounding errors – where the expected and actual value only differ due to the limits of numerical precision then consider something like the dnAnalytics Precision.EqualsWithTolerance approach (see Petrik’s comment below).