0
\$\begingroup\$

I've a Linq query that performs poorly:

var minuteValues = ValuesInMemory.Where(o => o.Key.Day == tmpValue.QuoteDate.Day &&
                                  o.Key.Hour == tmpValue.QuoteDate.Hour &&
                                  o.Key.Minute == tmpValue.QuoteDate.Minute &&
                                  o.Key.Second >= 0).Select(p => p.Value).ToList();

Is there a way to do this better?

private readonly ConcurrentDictionary<DateTime, double> ValuesInMemory = new ConcurrentDictionary<DateTime, double>();

ValuesInMemory holds about 6700 records 24/7. Every second records are added and every 2 hours the oldest values are stored in SQL Server. But the average amount of values is 6700.

\$\endgroup\$
  • 1
    \$\begingroup\$ could you give us more context code please? \$\endgroup\$ – Malachi Aug 22 '14 at 14:21
3
\$\begingroup\$

I think the crux of your problem here is you are using the wrong tool for the job. The purpose of a dictionary is usually for direct retrieval i.e. give me the value associated with X key. I'm not sure how optimized they are for O(n) queries.

To rule out the ConcurrentDictionary<TKey, TValue> being the bottleneck, you could try switching it out for something like a ConcurrentBag<T>.

\$\endgroup\$
1
\$\begingroup\$

The last condition in your Where() expression is unnecessary and should be removed:

o.Key.Second >= 0

Regarding performance:

Your linq query will perform a linear scan of all the elements in your dictionary to find all "matching" keys. This is slow. In general, Dictionaries are fast when you retrieve a value by its key like this:

var val = dict[key];

It looks like you're dealing with time-series data, so instead of mapping DateTimes to doubles you should make a custom class called DataPoint which encapsulates a value and its Timestamp.

To make this query efficient you have two options, neither are trivial:

  1. Store your DataPoint objects in an ordered list and perform a binary search over the list using a custom IComparer<T>. http://msdn.microsoft.com/en-us/library/3f90y839(v=vs.110).aspx

  2. Use a dictionary to map between DateTimes and a list of DataPoint objects. Whenever you add a new DataPoint to the dictionary use an adjusted DateTime value as the key and add the DataPoint to the corresponding list. Generate adjusted DateTime values by making a new DateTime object based on the value in your DataPoint with the Second, and Millisecond properties set to zero. Next, you need to enforce this adjustment architecturally in one of two ways:

\$\endgroup\$
0
\$\begingroup\$

We can extract the tmpValue.QuoteDate's Day, Hour, Minute to separate variables.

int day = tmpValue.QuoteDate.Day;
int hour = tmpValue.QuoteDate.Hour;
int minute = tmpValue.QuoteDate.Minute;

In this way these properties doesn't need to be queried each time

var minuteValues = ValuesInMemory.Where(o => o.Key.Day == day &&
                              o.Key.Hour == hour &&
                              o.Key.Minute == minute &&
                              o.Key.Second >= 0).Select(p => p.Value).ToList();  

This can help, but maybe the compiler has optimized your query.

\$\endgroup\$
  • 1
    \$\begingroup\$ I really can't imagine this making much of a difference. \$\endgroup\$ – James Aug 22 '14 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.