Update cache with minimal blocking

Question

As part of the task, it is allowed to use slightly outdated data. It is required that only one thread per key is involved in the critical section, while the remaining threads use data from the cache despite the expired TTL. The following code is an attempt to write a function calling the delegate within a blocking section based on SemaphoreSlim:

/// <summary>
/// Ensures consistency and
/// avoid extra requests to
/// datastore.
/// </summary>
/// 
/// <exception cref="ArgumentNullException"/>
/// <exception cref="ObjectDisposedException"/>
/// <exception cref="OperationCanceledException"/>
protected virtual (bool err, T val) AsCriticalLocal<T>(string k, string region, Func<(bool err, ICacheWrapper<T> val)> preCheck, Func<(bool err, T val)> method, CancellationToken token)
{
    k = HelperIO(k, region);

    /*
     * Since this is a local cache, it
     * checks for the presence of data
     * in it.
     */

    (bool err, ICacheWrapper<T> val) preCheckRes = preCheck();

    SemaphoreSlim semaphore = Lock.GetOrAdd(k, f => new
    SemaphoreSlim
    (1, 1));

    int m_currentCount = semaphore.CurrentCount;

    // Means that the value has not been removed from the cache.
    if (!preCheckRes.err)
    {
        // Call Wait will be blocking.
        if (m_currentCount == 0)
        {
            // The update is already running, so don't care about TTL.
            return (preCheckRes.err, preCheckRes.val.Val);
        }
        else
        {
            // The update has not been started and the data is still relevant.
            if (!preCheckRes.val.IsExpired)
            {
                return (preCheckRes.err, preCheckRes.val.Val);
            }
        }
    }

    (bool err, T val) output = default;

    try
    {
        Logger.Debug($"Cache: { Name }. Key: { k }. Lock: BLOCKED.  Date: { DateTime.Now }. Type: { typeof(T) }");

        semaphore.Wait(GlobalTimeout, token);

        preCheckRes = preCheck();

        // Means that the value has not been removed from the cache.
        if (!preCheckRes.err)
        {
            /*
             * If the semaphore is involved after
             * updating the data in the cache and
             * until the TTL expires
             */

            if (!preCheckRes.val.IsExpired)
            {
                return (preCheckRes.err, preCheckRes.val.Val);
            }
        }

        output = method();
    }
    catch (Exception exception)
    {
        Logger.Error(exception);

        throw;
    }
    finally
    {
        if (output.err)
        {
            /*
             * Remove data from the cache in order to avoid
             * the use of outdated data since an exception
             * was thrown in the update section.
             */

            preCheckRes = preCheck();

            if (!preCheckRes.err)
            {
                if (preCheckRes.val.IsExpired)
                {
                    Instance.Remove(HelperIO(k, region));
                }
            }
        }

        semaphore.Release();

        Logger.Debug($"Cache: { Name }. Key: { k }. Lock: RELEASED. Date: { DateTime.Now }. Type: { typeof(T) }");
    }

    return output;
}

---

Сhecks that there is data in the cache and the semaphore is already used to block when executing the delegate returning data from the datastore (database, etc). If the update is not started and the data is still relevant, immediately returns it:

int m_currentCount = semaphore.CurrentCount;

// Means that the value has not been removed from the cache.
if (!preCheckRes.err)
{
    // Call Wait will be blocking.
    if (m_currentCount == 0)
    {
        // The update is already running, so don't care about TTL.
        return (preCheckRes.err, preCheckRes.val.Val);
    }
    else
    {
        // The update has not been started and the data is still relevant.
        if (!preCheckRes.val.IsExpired)
        {
            return (preCheckRes.err, preCheckRes.val.Val);
        }
    }
}

---

If the method was not returned in the previous two paragraphs, then the Wait method is called and the data is extracted from cache and verified again: if the data is not removed and the data is still relevant, immediately returns it. If none of the above is done, then the delegate is called, inside which the query to the datastore is executed:

semaphore.Wait(GlobalTimeout, token);

preCheckRes = preCheck();

// Means that the value has not been removed from the cache.
if (!preCheckRes.err)
{
    /*
     * If the semaphore is involved after
     * updating the data in the cache and
     * until the TTL expires
     */

    if (!preCheckRes.val.IsExpired)
    {
        return (preCheckRes.err, preCheckRes.val.Val);
    }
}

output = method();

---

If an error occurs during operation, the data will be deleted from the cache to avoid the use of non-updated data:

finally
{
    if (output.err)
    {
        /*
         * Remove data from the cache in order to avoid
         * the use of outdated data since an exception
         * was thrown in the update section.
         */

        preCheckRes = preCheck();

        if (!preCheckRes.err)
        {
            if (preCheckRes.val.IsExpired)
            {
                Instance.Remove(HelperIO(k, region));
            }
        }
    }

    semaphore.Release();

    Logger.Debug($"Cache: { Name }. Key: { k }. Lock: RELEASED. Date: { DateTime.Now }. Type: { typeof(T) }");
}

---

UPD:

preCheck is a simple delegate that calls the Get method internally and retrieves the data from the MemoryCache and returns a tuple of the form (bool err, T val):

{
    return AsCriticalLocal(k, region, () =>
    {
        ICacheWrapper<T> item = Get<T>(k, region);

        return (Common.IsDefault(item), item);
    },
    () =>
    {
        T val = method();

        return Set(k, TTL, val, true, region);
    },

    token);
}

---

Questions:

1. Will this code be thread safe?
2. Is the described goal achieved?
3. What changes are worth making in terms of code structure?
4. Assuming that this code works as expected, is there any point in duplicating the code where the asynchronous version of Wait - WaitAsync will be used for async methods? It is Implied that there will not be a large number of blocked threads.

Answer 1

I have design and implementation suggestions as well.

Cache type

There are several different caching strategies, like: read-through, write-through, refresh-ahead, cache-aside, write-behind, ...

Let's examine the following three: read-through, refresh-ahead, cache-aside

• Read-Through works in the following way:
  • If the content is present in the cache then it will be served from the cache
  • otherwise it will be fetched from the storage (the source of truth)
    • after the retrieval is succeeded then it will store a local copy in the cache
    • and finally the request is served from the cache
• Refresh-Ahead works in the following way:
  • The content is always served from the cache
  • If the content expires then it will fetch the data automatically without any external request.
• Cache-Aside works in the following way:
  • If the content is present in the cache then it will be served from the cache
  • When it is not present then a separate flow / process fetches the data from the source and stores the data into the cache

So, let's compare them:

• Read-Through vs Refresh-Ahead: reactive (serves on-demand) vs proactive (fetches in the background)
• Refresh-Ahead vs Cache-Aside: fetch is done by the cache itself vs retrieval process is done by an external provider
• Read-Through vs Cache-Aside: fetch by itself on-demand vs fetch by external provider on demand or in the background

Based on the requirements you should be able to decide which one is the right one for you. For further details, please this article.

---

Responsiveness

Serving stale data during the freshest retrieval has three separate stages:

1. Determining the freshness of the data and then branching based on the result
2. Retrieving data, while serving other requests
3. Updating the data in the cache

I would not spend words on the first and the last one because they are the easy ones. The second phase is where concurrency / parallelism comes into the play. You need to initiate a background retrieval process to be able to serve other requests simultaneously. You also need to synchronize the state between these threads. In order to do so you will utilize one of the sync primitives that are provided by .NET. You have chosen the SemaphoreSlim, which might not be the best for this problem.

There are different sync primitive categories:

• Locking constructions (Monitor, SpinLock, Semaphore, etc.)
• Blocking constructions (CountdownEvent, ManaulResetEvent, SpinWait, Task.Wait, etc.)
• Non-blocking constructions (Interlocked, MemoryBarrier, etc.)

I would highly recommend the following webpages for further details: 1, 2, 3

SemaphoreSlim is a generalization of the Monitor, which means rather than guaranteeing exclusive access to a single thread rather than it can allow n threads to access the same resource. In my opinion this is not what you need. Your semaphore instance (most probably) allow only a single access, and the state of the lock is used for branching. The same could be achieved with SpinLock and its IsHeld property, but I would not recommended that because it designed for really short locking in order to prevent context-switches.

The best fit (in my opinion) is to use one of the signaling approaches. Because what you try to achieve is that: "until a given condition is not met (the freshest data is not available) I would like to use my fallback (give back the stale data)"

WaitHandle and EventWaitHandle base classes do not expose their state like IsHeld, but calling the WaitOne with zero timeout will tell you instantly whether or not the other thread has signaled (by calling the Set method).

---

MemoryCache

I would like to also highlight that there are more than one MemoryCache. There is one under the System.Runtime.Caching namespace and there is another under the Microsoft.Extensions.Caching.Memory. Latter suits better for ASP.NET Core. Fortunately both of them are thread-safe by default.

---

Last but not least, I have two other suggestions:

1. Try to split your logic into smaller functions
2. First try to solve the problem with sync API then when you are familiar with all the components / primitives then try to achieve the async API