2
\$\begingroup\$

I have a json array that is being passed into a function. Whenever the function comes across a field (call it Field1) in a record with a value that starts with "@!!!@" the function compile them into a list to fire off to another server. I have code that looks like this in the initial function:

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    dynamic request = await req.Content.ReadAsAsync<dynamic>();

    JObject data = (JObject)request;
    var directive = data["Directive"];

    var json = data.Last.First;

    string url = null;

    if (directive.ToString() == "D")
    {
        url = "registry/get";
    }
    else if (directive.ToString() == "S")
    {
        url = "registry/sanitize";
    }



    JArray payloadArray = (JArray)json;

    string newToken = null;

    List<JObject> objList = new List<JObject>();

    for (int i = 0; i <= payloadArray.Count() - 1; i++)
    {
        string newJson = null;
        foreach (var prop in payloadArray[i])
        {
            newJson = newJson + prop.ToString() + ",";
            if (prop.ToString().Contains("@!!!@"))
            {
                JObject newProp = new JObject();
                newProp.Add(prop);
                newJson = newJson + "\"" + newProp.Properties().First().Name + "\":\"sanitize|" + prop.First + "\"," + "\"@" + newProp.Properties().First().Name.ToString() + "\":\"" + prop.First + "\",";
                // newJson = newJson + "\"@" + newProp.Properties().First().Name.ToString() + "\":\"sanitize|" + prop.First + "\",";
                newToken = newToken + "{\"Token\":\"" + prop.First + "\",\"ProcessId\":\"" + prop.First.ToString().Replace("@!!!@", "") + "\"},";
            }
        }
        objList.Add(JObject.Parse("{" + newJson + "}"));

    }

    string outGoingPayload = "{\"Registry\":[" + newToken + "]}";

      var content = new StringContent(outGoingPayload.ToString(), Encoding.UTF8, "application/json");

       HttpResponseMessage response = MakeRequest(outGoingPayload.ToString());


 var responseBodyAsText = response.Content.ReadAsStringAsync();
JObject responseJson = JObject.Parse(responseBodyAsText.Result);
int counter = 0;
foreach(JObject item in objList)
{
    foreach(var itm in item)
    {
        if (itm.Value.ToString().Contains("sanitize|@!!!@"))
        {
            foreach(var resItem in responseJson)
            {
                if (resItem.Value[counter]["ProcessId"].ToString() == itm.Value.ToString().Replace("sanitize|@!!!@",""))
                {
                    try
                    {
                        itm.Value.Replace(resItem.Value[counter]["Value"].ToString());
                    }
                    catch (Exception ex)
                    {
                        itm.Value.Replace("Token not found");
                    }     
                    counter++;
                }
            }
        }
    }

}

string jsonStr = null;
foreach (var val in objList)
{
    jsonStr = jsonStr + val + ",";
}

jsonStr = "[" + jsonStr.TrimEnd(',') + "]";
var returnArray = JsonConvert.DeserializeObject(jsonStr);


    return req.CreateResponse(HttpStatusCode.OK, "returnArray");
}

For a Json payload of 1000 records this take 5000ms to run. What can I do to improve performance here? Setup of the system is the payload passed into the function will contain all records, I must build up a new payload to pass into the remote service to get corresponding values. I do this as one HttpClient request. So this means I loop the initial payload, build up the new payload, call the remove service, return all matches and loop the initial payload and add the extra field where appropriate. I am trying to get this function to return a bit faster. I have tried using Linq to JSON I have tried treating the json as a string. The code I have posted seems to be the fastest. I can provide more information if needed.

Sample payload to send to this function:

{
"Directive":"S",
"Wrapper":[
{
 "Field1":"Apple",
 "Field2":"Peach",
 "Field3":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922",
 "Field4":"Kiwi"
},
{
 "Field1":"Apple2",
 "Field2":"Peach2",
 "Field3":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6",
 "Field4":"Kiwi2"
}]}

Sample to send the intermediate function (I cannot touch this one but it's already optimized) My function should build this from the above payload:

{ "Wrapper":[
{"Token":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6"},
{"Token":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922"}
]}

Return from intermediate service:

{ "Wrapper":[
{"Token":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6", "Value":"test"},
{"Token":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922", "Value":"test2"}
]}

Back in my function both the original payload and the return payload from the intermediate function should be merged and returned like this:

  {
    "Wrapper":[
       {
         "Field1":"Apple",
         "Field2":"Peach",
 "Field3":"@!!!@008d613d1ca60885468bf274daa693cc778430fc8a539bdf2e7dc2dec88cd922",
  "@Field3":"test2",
  "Field4":"Kiwi"
     },
{
 "Field1":"Apple2",
 "Field2":"Peach2",
 "Field3":"@!!!@17e9ad37968e25893e96855ba3d633e250a401a6584b2bc9c7288f9fc458a9b6 ",


"@Field3":"test"
           "Field4":"Kiwi2"
           }]}
\$\endgroup\$
  • \$\begingroup\$ Please post the complete method, not a random snippet. If possible the class. \$\endgroup\$ – t3chb0t Jul 13 '18 at 17:55
  • \$\begingroup\$ sure give me a moment and I will do that.... just FYI the bottleneck is in the first function the remote service called cannot be modified and is already optimized. Let me upload the entire class. \$\endgroup\$ – VinnyGuitara Jul 13 '18 at 17:56
  • \$\begingroup\$ Was only able to put the function in not the whole class. The only other method in the class is the call to the remote service. \$\endgroup\$ – VinnyGuitara Jul 13 '18 at 18:02
  • \$\begingroup\$ This looks much better now ;-) I think the whole class isn't necessary here. The method provides enough context. \$\endgroup\$ – t3chb0t Jul 13 '18 at 18:04
  • 1
    \$\begingroup\$ @t3chb0t - I was able to add the payload/response example JSON... Hopefully that helps. \$\endgroup\$ – VinnyGuitara Jul 16 '18 at 14:00
2
\$\begingroup\$

Here are what in my opinion are the primary areas of concern.

Misuse of Json.NET

Code like JObject.Parse("{" + newJson + "}") is redundant. You are manually forming a JSON string and then deserialising it back into a JObject. You can just build JTokens directly, avoiding the cost of deserialization and having to mess around with formatting.

Inefficient Token Replacement

After the service call, you update the original items very inefficiently. Lets say you have 1000 items each with 3 tokenised fields. That would be 3000 results from your service call. Your code iterates through all 3000 results for each of the 3000 tokens, resulting in a total of ~9000000 iterations. I’ll show you below how to reduce this to just 3000.

A Possible Solution

If you create some classes to deserialize to/from, the code will be much easier to work with, removing the need to deal with JTokens at all. To fix your 9 million iterations problem, you could use a dictionary to create a lookup table when you initially loop over the properties. The key could be the token, and the value could be a reference to the item that needs to be updated. Then once you have your token values, you just need to iterate over them, updating the dictionary (fast). I've put a console app together here that works with your example JSON to demonstrate. The key parts are as follows.

Form service request and lookup table

foreach (var item in inputData.Wrapper)
{
    foreach (var kvp in item.Where(property => property.Value.StartsWith("@!!!@")))
    {
        propertiesLookup[kvp.Value] = new ItemUpdate
        {
            Properties = item,
            UpdateKey = kvp.Key
        };
        propertiesRequest.Wrapper.Add(new PropertySearch
        {
            Token = kvp.Value
        });
    }
}

Use lookup table to add tokenized fields to items

foreach (var item in intermediateData.Wrapper)
{
    var itemToUpdate = propertiesLookup[item.Token];
    itemToUpdate.Properties[$"@{itemToUpdate.UpdateKey}"] = item.Value;
}
\$\endgroup\$
  • \$\begingroup\$ It would be much cooler if you included the example code in your answer. \$\endgroup\$ – t3chb0t Jul 25 '18 at 17:22
  • \$\begingroup\$ Thanks so much for your suggestions and help on this. I am going to take a look at this attachment now. \$\endgroup\$ – VinnyGuitara Jul 25 '18 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.