I was going to post this code as an answer to a recent question, but I wrote this code a little while ago (like, a year ago; if I recall correctly I wrote this after reading this article) and I'd like to get some feedback on it instead.
The idea is to address a cross-cutting concern using DI interception, by implementing a CircuitBreakerInterceptor
that intercepts calls and wraps them in a "retry" logic.
Actually MSDN is more precise about it:
The purpose of the Circuit Breaker pattern is different from that of the Retry Pattern. The Retry Pattern enables an application to retry an operation in the expectation that it will succeed. The Circuit Breaker pattern prevents an application from performing an operation that is likely to fail. An application may combine these two patterns by using the Retry pattern to invoke an operation through a circuit breaker. However, the retry logic should be sensitive to any exceptions returned by the circuit breaker and abandon retry attempts if the circuit breaker indicates that a fault is not transient.
Here's the interface for the circuit breaker:
/// <summary>
/// An interface for a circuit breaker, which allows wrapping a feature in a "circuit" that can only execute when closed or half-closed.
/// </summary>
public interface ICircuitBreaker
{
/// <summary>
/// A <see cref="TimeSpan"/> indicating the duration of the <see cref="OpenState"/> state.
/// </summary>
TimeSpan Timeout { get; }
/// <summary>
/// Changes the state of the breaker to <see cref="ClosedState"/>, allowing normal execution.
/// </summary>
void ToClosedState();
/// <summary>
/// Trips the breaker and changes the state to <see cref="OpenState"/>, blocking normal execution.
/// </summary>
void ToOpenState();
/// <summary>
/// Changes the state of the breaker to <see cref="HalfOpenState"/>, allowing normal execution.
/// </summary>
void ToHalfOpenState();
/// <summary>
/// Attempts to execute the specified action.
/// </summary>
/// <param name="action">A method that takes no parameters and returns void.</param>
void Execute(Action action);
/// <summary>
/// Attempts to execute the specified action.
/// </summary>
/// <typeparam name="T">The return type.</typeparam>
/// <param name="action">A method that takes no parameters and returns a value.</param>
/// <returns></returns>
T Execute<T>(Func<T> action);
/// <summary>
/// Increments the failure count.
/// </summary>
void IncrementFailCount();
/// <summary>
/// Resets the failure count.
/// </summary>
void ResetFailCount();
/// <summary>
/// Returns true if failure count has reached a threshold and breaker should be tripped.
/// </summary>
/// <returns></returns>
bool IsThresholdReached();
/// <summary>
/// Implements actions to take upon circuit breaker tripping.
/// </summary>
/// <param name="e"></param>
void OnCircuitBreakerTripped(Exception e);
}
With a base class for all circuit breaker states:
/// <summary>
/// A base class for all <see cref="CircuitBreaker"/> states.
/// </summary>
public abstract class CircuitBreakerStateBase
{
/// <summary>
/// The constructor-injected circuit breaker.
/// </summary>
protected readonly ICircuitBreaker CircuitBreaker;
/// <summary>
/// Initializes a new cicuit breaker state.
/// </summary>
/// <param name="circuitBreaker">The circuit breaker.</param>
protected CircuitBreakerStateBase(ICircuitBreaker circuitBreaker)
{
CircuitBreaker = circuitBreaker;
}
/// <summary>
/// A method that runs before the protected code.
/// </summary>
public virtual void BeforeExecute() { }
/// <summary>
/// A method that runs after the protected code.
/// </summary>
public virtual void AfterExecute() { }
/// <summary>
/// A method that runs upon failure of the protected code.
/// </summary>
/// <param name="e">The exception that was caught.</param>
public virtual void OnException(Exception e)
{
CircuitBreaker.IncrementFailCount();
}
}
ClosedState
allows normal execution of the "protected code", and if that code throws, OnException
will increment the failure count and switch the circuit breaker to OpenState
if the code has thrown a given number of times:
/// <summary>
/// A state that indicates that the circuit is closed and normal execution is allowed.
/// </summary>
public class ClosedState : CircuitBreakerStateBase
{
/// <summary>
/// Initializes a <see cref="ClosedState"/> circuit breaker state, meaning circuit is closed and operating normally.
/// </summary>
/// <param name="circuitBreaker">The circuit breaker.</param>
public ClosedState(ICircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
CircuitBreaker.ResetFailCount();
}
/// <summary>
/// A method that runs upon failure of the protected code.
/// </summary>
/// <param name="e">The exception that was caught.</param>
public override void OnException(Exception e)
{
base.OnException(e);
if (!CircuitBreaker.IsThresholdReached()) return;
CircuitBreaker.ToOpenState();
CircuitBreaker.OnCircuitBreakerTripped(e);
}
}
When the circuit is in OpenState
, normal execution is completely blocked for a given timeout delay - any attempt to run the protected code in that state will fire up an OpenCircuitException
. After the timeout has elapsed, the circuit goes into HalfOpenState
:
/// <summary>
/// A state that indicates that the circuit is opened and normal execution is blocked.
/// </summary>
public class OpenState : CircuitBreakerStateBase
{
private readonly Timer _timer;
/// <summary>
/// Initializes a <see cref="OpenState"/> circuit breaker state, meaning circuit is open and will not operate normally.
/// </summary>
/// <param name="circuitBreaker">The circuit breaker.</param>
public OpenState(ICircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
_timer = new Timer(CircuitBreaker.Timeout.TotalMilliseconds);
_timer.Elapsed += TimeoutReached;
_timer.AutoReset = false;
_timer.Start();
}
private void TimeoutReached(object sender, ElapsedEventArgs e)
{
CircuitBreaker.ToHalfOpenState();
}
/// <summary>
/// A method that runs before the protected code.
/// </summary>
/// <exception cref="OpenCircuitException"></exception>
public override void BeforeExecute()
{
base.BeforeExecute();
throw new OpenCircuitException();
}
}
A half-open circuit breaker will try to run the protected code and close the circuit if it succeeds, or fall back to OpenState
if it fails again:
/// <summary>
/// A state that indicates that the circuit is half-opened; successful execution will close the circuit, failure will re-open it.
/// </summary>
public class HalfOpenState : CircuitBreakerStateBase
{
/// <summary>
/// Initializes a <see cref="HalfOpenState"/> circuit breaker state, meaning circuit is half-open and will operate normally.
/// </summary>
/// <param name="circuitBreaker">The circuit breaker.</param>
public HalfOpenState(ICircuitBreaker circuitBreaker)
: base(circuitBreaker)
{ }
/// <summary>
/// A method that runs upon failure of the protected code.
/// </summary>
/// <param name="e">The exception that was caught.</param>
public override void OnException(Exception e)
{
base.OnException(e);
CircuitBreaker.ToOpenState();
CircuitBreaker.OnCircuitBreakerTripped(e);
}
/// <summary>
/// A method that runs after the protected code.
/// </summary>
public override void AfterExecute()
{
base.AfterExecute();
CircuitBreaker.ToClosedState();
}
}
Then you can address this cross-cutting concern with DI Interception, with an interceptor that can intercept the calls and wrap them in your circuit breaker:
/// <summary>
/// An interceptor that implements a CircuitBreaker pattern.
/// </summary>
public class CircuitBreakerInterceptor : IInterceptor
{
private readonly ICircuitBreaker _breaker;
/// <summary>
/// Initializes a new instance of the <see cref="CircuitBreakerInterceptor"/> class.
/// </summary>
/// <param name="breaker">An implemtation of a CircuitBreaker.</param>
public CircuitBreakerInterceptor(ICircuitBreaker breaker)
{
_breaker = breaker;
}
/// <summary>
/// A method that intercepts specified invocation, wrapping the call with the CircuitBreaker.
/// </summary>
/// <exception cref="OpenCircuitException"> thrown when circuit breaker is in <see cref="OpenState"/> state.</exception>
/// <param name="invocation"></param>
public void Intercept(IInvocation invocation)
{
_breaker.Execute(invocation.Proceed);
}
}
An implementation of an ICircuitBreaker
could look like this - here I'm just taking in a logger, but it could just as well be a MessageBoxCircuitBreaker
that displays a message instead:
public class LoggingCircuitBreaker : ICircuitBreaker
{
private readonly ILogger _logger;
private readonly object _monitor = new object();
private CircuitBreakerStateBase _state;
private static int _failCount;
/// <summary>
/// Gets the failure count.
/// </summary>
/// <value>
/// The failure count.
/// </value>
public int FailCount { get { return _failCount; } }
/// <summary>
/// Gets the failure threshold, representing the number of allowed failures before circuit switches to <see cref="OpenState"/>
/// </summary>
/// <value>
/// The failure threshold.
/// </value>
public int Threshold { get; private set; }
/// <summary>
/// A <see cref="TimeSpan" /> indicating the duration of the <see cref="OpenState" /> state.
/// </summary>
public TimeSpan Timeout { get; private set; }
/// <summary>
/// Creates a new CircuitBreaker that logs exceptions.
/// </summary>
/// <param name="threshold">Number of allowed failures in Closed state.</param>
/// <param name="timeout">A <see cref="TimeSpan"/> that determines how long a breaker remains in Open state.</param>
/// <param name="logger"></param>
public LoggingCircuitBreaker(int threshold, TimeSpan timeout, ILogger logger)
{
if (threshold < 1) throw new ArgumentException("threshold", "Threshold must be greater than 1.");
if (timeout.TotalMilliseconds < 1) throw new ArgumentException("timeout", "Timeout must be greater than 1 millisecond.");
_logger = logger;
Threshold = threshold;
Timeout = timeout;
ToClosedState();
}
/// <summary>
/// Changes the state of the breaker to <see cref="ClosedState" />, allowing normal execution.
/// </summary>
public void ToClosedState()
{
lock (_monitor)
{
_state = new ClosedState(this);
}
}
/// <summary>
/// Trips the breaker and changes the state to <see cref="OpenState" />, blocking normal execution.
/// </summary>
public void ToOpenState()
{
lock (_monitor)
{
_state = new OpenState(this);
}
}
/// <summary>
/// Changes the state of the breaker to <see cref="HalfOpenState" />, allowing normal execution.
/// </summary>
public void ToHalfOpenState()
{
lock (_monitor)
{
_state = new HalfOpenState(this);
}
}
/// <summary>
/// Increments the failure count.
/// </summary>
public void IncrementFailCount()
{
Interlocked.Increment(ref _failCount);
}
/// <summary>
/// Resets the failure count.
/// </summary>
public void ResetFailCount()
{
lock (_monitor)
{
_failCount = 0;
}
}
/// <summary>
/// Returns true if failure count has reached a threshold and breaker should be tripped.
/// </summary>
/// <returns></returns>
public bool IsThresholdReached()
{
return _failCount >= Threshold;
}
/// <summary>
/// Attempts to execute the specified action.
/// </summary>
/// <param name="action">A method that takes no parameters and returns void.</param>
public void Execute(Action action)
{
lock (_monitor) { _state.BeforeExecute(); }
try
{
action();
}
catch (Exception e)
{
lock (_monitor) { _state.OnException(e); }
throw;
}
lock (_monitor) { _state.AfterExecute(); }
}
/// <summary>
/// Attempts to execute the specified action.
/// </summary>
/// <typeparam name="T">The return type.</typeparam>
/// <param name="action">A method that takes no parameters and returns a value.</param>
/// <returns></returns>
public T Execute<T>(Func<T> action)
{
lock (_monitor) { _state.BeforeExecute(); }
T result;
try
{
result = action();
}
catch (Exception e)
{
lock (_monitor) { _state.OnException(e); }
throw;
}
lock (_monitor) { _state.AfterExecute(); }
return result;
}
/// <summary>
/// Implements actions to take upon circuit breaker tripping.
/// </summary>
/// <param name="e">The exception that tripped the circuit breaker.</param>
public void OnCircuitBreakerTripped(Exception e)
{
_logger.ErrorException("Circuit breaker tripped!", e);
}
}
The above implementation is meant to be thread-safe - but thread safety isn't exactly my cup of tea and I'd like to know if the code has potential race conditions or other threading issues I'm not aware of.
Of course every other aspect of the code is open to constructive comments (I'm seeing quite a few code style /readability points that I'd write differently today).
ICircuitBreaker
include methods likeToClosedState()
? Shouldn't those be internal to the implementation? When would a user ever call those? \$\endgroup\$