How To Prioritize xUnit Test Methods Using TestPriority Attributes and ITestCaseOrderer Interface

Hello,

A requirement for a project wherein we want to prioritize tests given that a certain algorithm depends on the execution of another one, we need to create tests priorities so that test methods are executed in a sequential order. After doing some research, I found an article here that demonstrates on how to control the execution order of test methods. Given the sample code below in a console application, I would like to set a priority for each method using the MDAS (Multiplication->Division->Addition->Subtraction) rule using the concepts from that article.

public class MathOperations
{
	public double Addition(double a, double b)
	{
		return a + b;
	}

	public double Subtraction(double a, double b)
	{
		return a - b;
	}

	public double Multiplication(double a, double b)
	{
		var result = a * b;
		return result;
	}

	public double Division(double a, double b)
	{
		var result = a / b;
		return result;
	}
}

The methodology that I would like to apply are using the TestPriority attribute and implementation of ITestCaseOrderer interface. In the xUnit test project, I defined a class below that inherits the Attribute class which is then decorated with AttributeUsage class. The constructor then receives an integer priority for the order of test execution.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
    public TestPriorityAttribute(int priority)
    {
        Priority = priority;
    }

    public int Priority { get; private set; }
}

I then defined another class that implements the ITestCaseOrderer interface. The method will then perform sorting the tests according to their designated priorities. The TestPriorityAttribute and it's Priority property are referenced in this method.

public class TestSequenceOrder : ITestCaseOrderer
{
    public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
    {
        var sortedMethods = new SortedDictionary<int, List<TTestCase>>();

        foreach (TTestCase testCase in testCases)
        {
            int priority = 0;

            foreach (IAttributeInfo attr in testCase.TestMethod.Method.GetCustomAttributes((typeof(TestPriorityAttribute).AssemblyQualifiedName)))
                priority = attr.GetNamedArgument<int>("Priority");

            GetOrCreate(sortedMethods, priority).Add(testCase);
        }

        foreach (var list in sortedMethods.Keys.Select(priority => sortedMethods[priority]))
        {
            list.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
            foreach (TTestCase testCase in list)
                yield return testCase;
        }
    }

    private static TValue GetOrCreate<TKey, TValue>(IDictionary<TKey, TValue> dictionary, TKey key)
        where TValue : new()
    {
        TValue result;

        if (dictionary.TryGetValue(key, out result))
            return result;

        result = new TValue();
        dictionary[key] = result;

        return result;
    }
}

To test the console application math operation methods, I created a class called MathOpTests decorated with the TestCaseOrderer attribute and their parameters are the TestSequenceOrder class and the fully qualified namespace. For each method, I then call the TestPriority class and pass an integer that signifies it's execution order. In the sample code below TestMultiplication() test is 1, TestDivision() test is 2, TestAddition() is 3 and TestSubtraction() is 4. When all tests are executed, the execution order are in place.

[TestCaseOrderer("TestsOrderByPriority.Tests.TestSequenceOrder", "TestsOrderByPriority.Tests")]
public class MathOpTests
{
    private MathOperations mathOperations;
    private readonly ITestOutputHelper output;
    private static int ctr = 0;

    public MathOpTests(ITestOutputHelper output)
    {
        this.output = output;
        mathOperations = new MathOperations();
    }

    [Fact, TestPriority(3)]
    public void TestAddition()
    {
        //arrange
        int num1 = 5;
        int num2 = 5;
        double result = 0;

        //act
        ctr++;
        result = mathOperations.Addition(num1, num2);
        output.WriteLine($"Test Addition priority {ctr}");

        //assert
        Assert.Equal(10, result);
    }

    [Fact, TestPriority(4)]
    public void TestSubtraction()
    {
        //arrange
        int num1 = 5;
        int num2 = 5;
        double result = 0;

        //act
        ctr++;
        result = mathOperations.Subtraction(num1, num2);
        output.WriteLine($"Test Subtraction priority {ctr}");

        //assert
        Assert.Equal(0, result);
    }

    [Fact, TestPriority(1)]
    public void TestMultiplication()
    {
        //arrange
        int num1 = 5;
        int num2 = 5;
        double result = 0;

        //act
        ctr++;
        result = mathOperations.Multiplication(num1, num2);
        output.WriteLine($"Test Multiplication priority {ctr}");

        //assert
        Assert.Equal(25, result);
    }

    [Fact, TestPriority(2)]
    public void TestDivision()
    {
        //arrange
        int num1 = 5;
        int num2 = 5;
        double result = 0;

        //act
        ctr++;
        result = mathOperations.Division(num1, num2);
        output.WriteLine($"Test Division priority {ctr}");

        //assert
        Assert.Equal(1.0, result);
    }
}

Output

Multiplication Test

Division Test

Addition Test

Subtraction Test

Source Code: Tests Methods Order By Priority