3
\$\begingroup\$

I build a C# Rest Client library / framework. I noticed that it seemed to be significantly faster than RestSharp for GET at least. So, I decided to put together some benchmarks comparing it to other clients. I'm a bit new to bench-marking, but I don't want to get hung up on best practice.

I'd just like to know,

  • Have I introduced bias here?

  • Have I made any obvious mistakes?

  • Have I missed any comparable Rest Clients?

The four libraries are RestSharp, DalSoft RestClient, Flurl.Http and my library RestClient.Net. The platform is .NET Core 3.1. I've included a test for serialization with the new System.Text.Json (case insensitive properties). I'm testing the Task based async version of all libraries.

These results are from a Windows parallel machine, but will post OSX results when ready. Here I run 6 batches of tests, and exclude the first batch for each library to remove engine startup time skewing.The results are the average of the total time. Results are in milliseconds with 250 repeats. It doesn't test concurrency.


Here are the results for GET

DalSoft 3527.41

Flurl 1500.3981

RestClient.Net System.Text.Json 2092.2710

RestClient.Net Newtsonsoft 1606.2478

RestSharp 9033.2331


Here are the results for POST

DalSoft 850.5417

Flurl 830.1859

RestClient.Net System.Text.Json 1699.9605

RestClient.Net Newtsonsoft 917.9309

RestSharp 894.0506


Unit Tests

Code Reference

[TestClass]
public class PerformanceTests
{
    #region Misc
    [AssemblyInitialize]
    public static void Initialize(TestContext testContext)
    {
        //Load all the assemblies in to the app domain so this loading doesn't skew results
        var flurlClient = new FlurlClient(PeopleUrl);
        var countryCodeClient = new Client(new NewtonsoftSerializationAdapter(), new Uri(PeopleUrl));
        var restSharpClient = new RestSharp.RestClient(PeopleUrl);
        var dalSoftClient = new DalSoft.RestClient.RestClient(PeopleUrl);
        var personJson = JsonConvert.SerializeObject(new Person());
        personJson = System.Text.Json.JsonSerializer.Serialize(new Person());
    }

    private const int Repeats = 250;
    private const string PeopleUrl = "https://localhost:44337/JsonPerson/people";
    private const string Path = "Results.csv";
    private static FileStream stream;

    static PerformanceTests()
    {
        if (File.Exists(Path)) File.Delete(Path);
        stream = new FileStream(Path, FileMode.Append);
        WriteText("Client,Method,First Call,All Calls,Total\r\n");
    }

    private static void WriteText(string text)
    {
        var bytes = Encoding.UTF8.GetBytes(text);
        stream.Write(bytes, 0, bytes.Length);
    }

    [AssemblyCleanup()]
    public static void AssemblyCleanup() 
    {
        stream.Close();
    }
    #endregion

    #region Flurl
    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestGetFlurl()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var flurlClient = new FlurlClient(PeopleUrl);

        startTime = DateTime.Now;
        var people = await flurlClient.Request().GetJsonAsync<List<Person>>();
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        startTime = DateTime.Now;
        for (var i = 0; i < Repeats; i++)
        {
            people = await flurlClient.Request().GetJsonAsync<List<Person>>();
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"Flurl,GET,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestPostFlurl()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new FlurlClient(PeopleUrl);

        var peopleRequest = new List<Person>();
        for (var i = 0; i < 10; i++)
        {
            peopleRequest.Add(new Person { FirstName = "Test" + i });
        }

        startTime = DateTime.Now;
        var people = await ReadPostResponseAsync(countryCodeClient, peopleRequest);
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await ReadPostResponseAsync(countryCodeClient, peopleRequest);
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"Flurl,POST,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    private static async Task<List<Person>> ReadPostResponseAsync(FlurlClient countryCodeClient, List<Person> peopleRequest)
    {
        var response = await countryCodeClient.Request().PostJsonAsync(peopleRequest);
        var json = await response.Content.ReadAsStringAsync();
        var people = JsonConvert.DeserializeObject<List<Person>>(json);
        return people;
    }
    #endregion

    #region RestClient.Net
    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestGetRestClientNewtonSoft()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new Client(new NewtonsoftSerializationAdapter(), new Uri(PeopleUrl));

        startTime = DateTime.Now;
        List<Person> people = await countryCodeClient.GetAsync<List<Person>>();
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.GetAsync<List<Person>>();
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestClient.Net Newtonsoft,GET,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestGetRestClient()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new Client(new Uri(PeopleUrl));

        startTime = DateTime.Now;
        List<Person> people = await countryCodeClient.GetAsync<List<Person>>();
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.GetAsync<List<Person>>();
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestClient.Net,GET,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestPostRestClient()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new Client(new Uri(PeopleUrl));

        var peopleRequest = new List<Person>();
        for (var i = 0; i < 10; i++)
        {
            peopleRequest.Add(new Person { FirstName = "Test" + i });
        }

        startTime = DateTime.Now;
        List<Person> people = await countryCodeClient.PostAsync<List<Person>, List<Person>>(peopleRequest);
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.PostAsync<List<Person>, List<Person>>(peopleRequest);
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestClient.Net,POST,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestPostRestClientNewtonsoft()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new Client(new NewtonsoftSerializationAdapter(), new Uri(PeopleUrl));
        countryCodeClient.SetJsonContentTypeHeader();

        var peopleRequest = new List<Person>();
        for (var i = 0; i < 10; i++)
        {
            peopleRequest.Add(new Person { FirstName = "Test" + i });
        }

        startTime = DateTime.Now;
        List<Person> people = await countryCodeClient.PostAsync<List<Person>, List<Person>>(peopleRequest);
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.PostAsync<List<Person>, List<Person>>(peopleRequest);
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestClient.Net Newtonsoft,POST,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }
    #endregion

    #region RestSharp
    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestGetRestSharp()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new RestSharp.RestClient(PeopleUrl);

        startTime = DateTime.Now;
        var people = await countryCodeClient.ExecuteGetTaskAsync<List<Person>>(new RestRequest { Method = Method.GET });
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.ExecuteGetTaskAsync<List<Person>>(new RestRequest { Method = Method.GET });
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Data.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestSharp,GET,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestPostRestSharp()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new RestSharp.RestClient(new Uri(PeopleUrl));

        var peopleRequest = new List<Person>();
        for (var i = 0; i < 10; i++)
        {
            peopleRequest.Add(new Person { FirstName = "Test" + i });
        }

        startTime = DateTime.Now;
        var peopleRestRequest = new RestRequest { Method = Method.POST, Body = new RequestBody("application/json", "Person", peopleRequest) };
        var people = await countryCodeClient.ExecutePostTaskAsync<List<Person>>(peopleRestRequest);
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.ExecutePostTaskAsync<List<Person>>(peopleRestRequest);
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Data.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"RestSharp,POST,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }
    #endregion

    #region DALSoft
    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestGetDALSoft()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new DalSoft.RestClient.RestClient(PeopleUrl);

        startTime = DateTime.Now;
        var people = await countryCodeClient.Get<List<Person>>();
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.Get<List<Person>>();
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"DalSoft,GET,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }

    [TestMethod]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    [DataRow]
    public async Task TestPostDALSoft()
    {
        var startTime = DateTime.Now;
        var originalStartTime = DateTime.Now;
        var countryCodeClient = new DalSoft.RestClient.RestClient(PeopleUrl);

        var peopleRequest = new List<Person>();
        for (var i = 0; i < 10; i++)
        {
            peopleRequest.Add(new Person { FirstName = "Test" + i });
        }

        startTime = DateTime.Now;
        var people = await countryCodeClient.Post<List<Person>, List<Person>>(peopleRequest);
        var timesOne = (DateTime.Now - startTime).TotalMilliseconds;

        for (var i = 0; i < Repeats; i++)
        {
            people = await countryCodeClient.Post<List<Person>, List<Person>>(peopleRequest);
            Assert.IsTrue(people != null);
            Assert.IsTrue(people.Count > 0);
        }

        var timesRepeats = (DateTime.Now - startTime).TotalMilliseconds;
        var total = (DateTime.Now - originalStartTime).TotalMilliseconds;

        var message = $"DalSoft,POST,{timesOne},{timesRepeats},{total}\r\n";
        WriteText(message);
        Console.WriteLine(message);
    }
    #endregion
}

System.Text.Json is looking promising in case sensitive mode. It seems to be faster than Newtonsoft case insensitive. But, there's still investigation to be done on whether or not it can be faster in other situations.

\$\endgroup\$
4
  • 3
    \$\begingroup\$ Use Benchmark.NET to measure your performance. Using wall clock time (DateTime.Now) is a very inaccurate way to measure the actual performance of your code versus other tasks running. \$\endgroup\$ Commented Jan 19, 2020 at 21:44
  • 1
    \$\begingroup\$ Use StopWatch instead of DateTime, and also, use three loads which simulates the consuming loads (heavy, medium, light). Keep monitoring the resources and also the memory. and make at least three tests on each lib, then get the average. \$\endgroup\$
    – iSR5
    Commented Jan 19, 2020 at 22:58
  • \$\begingroup\$ I'm wondering if you would be willing to share any updated information you may have had from utilizing suggested timing improvements. \$\endgroup\$ Commented Dec 11, 2020 at 18:49
  • \$\begingroup\$ I I need revisit this. Unfortunately, I found Benchmark.Net for .NET Core didn't match the documentation and struggled to get it working. I will take another shot at this for the next release of RestClient.Net \$\endgroup\$ Commented Dec 13, 2020 at 1:58

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.