1
\$\begingroup\$

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure, I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();
public async Task ProcessStart()
{
    string messageData = "{\"name\":\"DemoData\",\"no\":\"111\"}";
    RegistryManager registryManager;

    Parallel.ForEach(deviceList, async (device) =>
    {
        // get details for each device and use key to send message
        device = await registryManager.GetDeviceAsync(device.DeviceId);
        SendMessages(device.DeviceId, device.Key, messageData);
    });

    if (taskEventList.Count > 0)
    {
        await Task.WhenAll(taskEventList);
    }
}

private void SendMessages(string deviceId, string Key, string messageData)
{
    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
    //created separate Task
    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));
    taskEventList.Add(taskEvents);
}


private async Task ProcessMessages(string deviceId, string messageData)
{
    var startTime = DateTime.UtcNow;
    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))
    {
        await deviceClient.SendEventAsync(messageData);
    }
}
\$\endgroup\$
  • 2
    \$\begingroup\$ You are probably getting into a thread starvation as you just scheduling tasks and also borrowing threads from the thread pool. You need some kind of batching or throttling, as the thread pool is limited ( a thread has 2 MB, imagine 10k requests for threads) \$\endgroup\$ – Adrian Iftode Mar 15 '19 at 17:34
  • 1
    \$\begingroup\$ Also if DeviceClient.Create creates and disposes a new HttpClient everytime, then you'll get out of available sockets \$\endgroup\$ – Adrian Iftode Mar 15 '19 at 17:40
  • 3
    \$\begingroup\$ Hey @AdrianIftode that seems to be the beginnings of an answer. Are you sure you don't want to write one, now that the post has been reopened? \$\endgroup\$ – Vogel612 Mar 15 '19 at 18:47
  • \$\begingroup\$ Thanks, I would like also to know about DeviceClient.Create, possible @Neo ? \$\endgroup\$ – Adrian Iftode Mar 15 '19 at 20:40
  • \$\begingroup\$ @Neo, are you sure everything is ok with this check: while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15)) ? \$\endgroup\$ – Adrian Iftode Mar 15 '19 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.