Looping JSON in WebAPI Controller and Add new Property

Question

I have a json array that is being passed into a function. Whenever the function comes across a field (call it Field1) in a record with a value that starts with "@!!!@" the function compile them into a list to fire off to another server. I have code that looks like this in the initial function:

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    dynamic request = await req.Content.ReadAsAsync<dynamic>();

    JObject data = (JObject)request;
    var directive = data["Directive"];

    var json = data.Last.First;

    string url = null;

    if (directive.ToString() == "D")
    {
        url = "registry/get";
    }
    else if (directive.ToString() == "S")
    {
        url = "registry/sanitize";
    }



    JArray payloadArray = (JArray)json;

    string newToken = null;

    List<JObject> objList = new List<JObject>();

    for (int i = 0; i <= payloadArray.Count() - 1; i++)
    {
        string newJson = null;
        foreach (var prop in payloadArray[i])
        {
            newJson = newJson + prop.ToString() + ",";
            if (prop.ToString().Contains("@!!!@"))
            {
                JObject newProp = new JObject();
                newProp.Add(prop);
                newJson = newJson + "\"" + newProp.Properties().First().Name + "\":\"sanitize|" + prop.First + "\"," + "\"@" + newProp.Properties().First().Name.ToString() + "\":\"" + prop.First + "\",";
                // newJson = newJson + "\"@" + newProp.Properties().First().Name.ToString() + "\":\"sanitize|" + prop.First + "\",";
                newToken = newToken + "{\"Token\":\"" + prop.First + "\",\"ProcessId\":\"" + prop.First.ToString().Replace("@!!!@", "") + "\"},";
            }
        }
        objList.Add(JObject.Parse("{" + newJson + "}"));

    }

    string outGoingPayload = "{\"Registry\":[" + newToken + "]}";

      var content = new StringContent(outGoingPayload.ToString(), Encoding.UTF8, "application/json");

       HttpResponseMessage response = MakeRequest(outGoingPayload.ToString());


 var responseBodyAsText = response.Content.ReadAsStringAsync();
JObject responseJson = JObject.Parse(responseBodyAsText.Result);
int counter = 0;
foreach(JObject item in objList)
{
    foreach(var itm in item)
    {
        if (itm.Value.ToString().Contains("sanitize|@!!!@"))
        {
            foreach(var resItem in responseJson)
            {
                if (resItem.Value[counter]["ProcessId"].ToString() == itm.Value.ToString().Replace("sanitize|@!!!@",""))
                {
                    try
                    {
                        itm.Value.Replace(resItem.Value[counter]["Value"].ToString());
                    }
                    catch (Exception ex)
                    {
                        itm.Value.Replace("Token not found");
                    }     
                    counter++;
                }
            }
        }
    }

}

string jsonStr = null;
foreach (var val in objList)
{
    jsonStr = jsonStr + val + ",";
}

jsonStr = "[" + jsonStr.TrimEnd(',') + "]";
var returnArray = JsonConvert.DeserializeObject(jsonStr);


    return req.CreateResponse(HttpStatusCode.OK, "returnArray");
}

For a Json payload of 1000 records this take 5000ms to run. What can I do to improve performance here? Setup of the system is the payload passed into the function will contain all records, I must build up a new payload to pass into the remote service to get corresponding values. I do this as one HttpClient request. So this means I loop the initial payload, build up the new payload, call the remove service, return all matches and loop the initial payload and add the extra field where appropriate. I am trying to get this function to return a bit faster. I have tried using Linq to JSON I have tried treating the json as a string. The code I have posted seems to be the fastest. I can provide more information if needed.

Sample payload to send to this function:

{
"Directive":"S",
"Wrapper":[
{
 "Field1":"Apple",
 "Field2":"Peach",
 "Field3":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922",
 "Field4":"Kiwi"
},
{
 "Field1":"Apple2",
 "Field2":"Peach2",
 "Field3":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6",
 "Field4":"Kiwi2"
}]}

Sample to send the intermediate function (I cannot touch this one but it's already optimized) My function should build this from the above payload:

{ "Wrapper":[
{"Token":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6"},
{"Token":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922"}
]}

Return from intermediate service:

{ "Wrapper":[
{"Token":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6", "Value":"test"},
{"Token":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922", "Value":"test2"}
]}

Back in my function both the original payload and the return payload from the intermediate function should be merged and returned like this:

  {
    "Wrapper":[
       {
         "Field1":"Apple",
         "Field2":"Peach",
 "Field3":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922",
  "@Field3":"test2",
  "Field4":"Kiwi"
     },
{
 "Field1":"Apple2",
 "Field2":"Peach2",
 "Field3":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6 ",


"@Field3":"test"
           "Field4":"Kiwi2"
           }]}

Answer 1

Here are what in my opinion are the primary areas of concern.

Misuse of Json.NET

Code like JObject.Parse("{" + newJson + "}") is redundant. You are manually forming a JSON string and then deserialising it back into a JObject. You can just build JTokens directly, avoiding the cost of deserialization and having to mess around with formatting.

Inefficient Token Replacement

After the service call, you update the original items very inefficiently. Lets say you have 1000 items each with 3 tokenised fields. That would be 3000 results from your service call. Your code iterates through all 3000 results for each of the 3000 tokens, resulting in a total of ~9000000 iterations. I’ll show you below how to reduce this to just 3000.

A Possible Solution

If you create some classes to deserialize to/from, the code will be much easier to work with, removing the need to deal with JTokens at all. To fix your 9 million iterations problem, you could use a dictionary to create a lookup table when you initially loop over the properties. The key could be the token, and the value could be a reference to the item that needs to be updated. Then once you have your token values, you just need to iterate over them, updating the dictionary (fast). I've put a console app together here that works with your example JSON to demonstrate. The key parts are as follows.

Form service request and lookup table

foreach (var item in inputData.Wrapper)
{
    foreach (var kvp in item.Where(property => property.Value.StartsWith("@!!!@")))
    {
        propertiesLookup[kvp.Value] = new ItemUpdate
        {
            Properties = item,
            UpdateKey = kvp.Key
        };
        propertiesRequest.Wrapper.Add(new PropertySearch
        {
            Token = kvp.Value
        });
    }
}

Use lookup table to add tokenized fields to items

foreach (var item in intermediateData.Wrapper)
{
    var itemToUpdate = propertiesLookup[item.Token];
    itemToUpdate.Properties[$"@{itemToUpdate.UpdateKey}"] = item.Value;
}