2
\$\begingroup\$

Introduction

I've created a C++ fiber library for use in an emulator project I'm working on. My goals were to make the fibers simple to use and to not have much overhead (compared to making the system calls directly).

Code rundown:

  • Fiber.h / Fiber.cpp: Main fiber management code
  • FiberImpl.h: Selects platform implementation
  • WindowsFiberImpl.h / WindowsFiberImpl.cpp: Windows fiber implementation
  • PosixFiberImpl.h / PosixFiberImpl.cpp: POSIX fiber implementation
  • Common.h: Some commonly used types
  • Scheduler.h: Fiber scheduler base class
  • RoundRobinScheduler.h: Round robin scheduler implementation (used as the default scheduler)

Known issues

  • <ucontext.h> has been deprecated. Unfortunately, there is no direct replacement, so it seems my only other portable option would be pthreads, but using them would introduce a significant cost compared to the current implementation.
  • I use assertions throughout the code. I would like to provide the ability for users to either provide their own assert() implementation, or disable it entirely, but I'm not sure of an easy way to do so.

I'd like to get feedback about the above issues, and am interested in hearing any other general comments.

For those who are interested, the project is available at https://github.com/aaronmjacobs/Flax


Code

Fiber.h

#ifndef FLAX_FIBER_H
#define FLAX_FIBER_H

#include "Flax/FiberImpl.h"

#if FLAX_USE_SCHEDULER
#include "Flax/Scheduler.h"
#include <vector>
#endif // FLAX_USE_SCHEDULER

#include <functional>
#include <memory>
#include <string>

namespace flax {

/*!
 * Class that represents a Fiber (lightweight thread of execution that allows for cooperative multitasking).
 * Fibers are bound to the thread they're created on.
 */
class Fiber {
public:

   /*!
    * Gets the (thread local) main Fiber singleton.
    */
   static Fiber& getMainFiber();

   /*!
    * Gets the (thread local) active Fiber. If no Fibers have been created (including the main fiber), nullptr will be returned.
    */
   static Fiber* getActiveFiber();

   /*!
    * Creates a new Fiber on the current thread that will run |function|. Returns a null pointer on failure.
    * When a created Fiber finishes executing, if a scheduler is being used, the next chosen Fiber will be yielded to.
    * If a scheduler is not being used, the Fiber will yield to the main Fiber.
    */
   static std::unique_ptr<Fiber> create(const std::function<void()>& function, const std::string& name = "Fiber");

   /*!
    * Yields execution to the given Fiber. You should not yield to the active Fiber or to a Fiber that has finished executing.
    */
   static void yieldTo(Fiber& fiber);

#if FLAX_USE_SCHEDULER
   /*!
    * Yields execution to the next Fiber chosen by the scheduler. If the scheduler can't select a Fiber, the current Fiber will continue execution.
    */
   static void yield();

   /*!
    * Sets the scheduler used to select Fibers when yield() is called. If no scheduler is set, a round robin scheduler is used by default.
    */
   static void setScheduler(std::unique_ptr<Scheduler> newScheduler);
#endif // FLAX_USE_SCHEDULER

   ~Fiber();

   /*!
    * Returns the name of the Fiber.
    */
   const std::string& name() const {
      return fiberName;
   }

   /*!
    * Returns whether the Fiber is the active Fiber.
    */
   bool isActive() const {
      return data->activeFiber == this;
   }

   /*!
    * Returns whether the Fiber has finished executing.
    */
   bool isFinished() const {
      return finished;
   }

   /*!
    * Returns whether the Fiber is the main Fiber.
    */
   bool isMainFiber() const {
      return mainFiber;
   }

private:
   static void fiberMain(Fiber* fiber);

   Fiber(const std::function<void()>& func, const std::string& name, bool isMainFiber);
   Fiber(const Fiber& other) = delete;
   Fiber(Fiber&& other) = delete;
   Fiber& operator=(const Fiber& other) = delete;
   Fiber& operator=(Fiber&& other) = delete;

   bool isValid() const {
      return impl.isValid();
   }

   void finish();

   struct ThreadLocalData {
#if FLAX_USE_SCHEDULER
      ThreadLocalData(std::unique_ptr<Scheduler> initialScheduler)
         : activeFiber(nullptr), scheduler(std::move(initialScheduler)) {
      }
#else
      ThreadLocalData()
         : activeFiber(nullptr) {
      }
#endif // FLAX_USE_SCHEDULER

      Fiber* activeFiber;
#if FLAX_USE_SCHEDULER
      std::unique_ptr<Scheduler> scheduler;
      std::vector<Fiber*> fibers;
#endif // FLAX_USE_SCHEDULER
   };

   static thread_local std::shared_ptr<ThreadLocalData> threadLocalData;

   std::function<void()> function;
   std::string fiberName;
   const bool mainFiber;
   bool finished;
   FiberImpl impl;
   std::shared_ptr<ThreadLocalData> data;
};

} // namespace flax

#endif

Fiber.cpp

#include "Flax/Fiber.h"
#include "Flax/RoundRobinScheduler.h"

#include <algorithm>
#include <cassert>

namespace flax {

#if FLAX_USE_SCHEDULER
// static
thread_local std::shared_ptr<Fiber::ThreadLocalData> Fiber::threadLocalData(new Fiber::ThreadLocalData(std::make_unique<RoundRobinScheduler>()));
#else
// static
thread_local std::shared_ptr<Fiber::ThreadLocalData> Fiber::threadLocalData(new Fiber::ThreadLocalData);
#endif // FLAX_USE_SCHEDULER

// static
Fiber& Fiber::getMainFiber() {
   static thread_local Fiber mainFiber(nullptr, "Main Fiber", true);
   assert(mainFiber.isValid() && !mainFiber.isFinished() && mainFiber.isMainFiber());

   return mainFiber;
}

// static
Fiber* Fiber::getActiveFiber() {
   assert(threadLocalData);
   return threadLocalData->activeFiber;
}

// static
std::unique_ptr<Fiber> Fiber::create(const std::function<void()>& function, const std::string& name) {
   // Make sure the main fiber is initialized
   getMainFiber();

   std::unique_ptr<Fiber> newFiber(new Fiber(function, name, false));
   if (!newFiber->isValid()) {
      return nullptr;
   }

   return newFiber;
}

// static
void Fiber::yieldTo(Fiber& fiber) {
   assert(fiber.isValid() && !fiber.isActive() && !fiber.isFinished());
   assert(threadLocalData && threadLocalData->activeFiber);
   assert(threadLocalData->activeFiber->isValid() && threadLocalData->activeFiber->isActive());
   assert(&fiber != threadLocalData->activeFiber);

#if FLAX_USE_SCHEDULER
   threadLocalData->scheduler->onFiberYieldedTo(&fiber);
#endif // FLAX_USE_SCHEDULER

   FiberImpl& from = threadLocalData->activeFiber->impl;
   FiberImpl& to = fiber.impl;
   threadLocalData->activeFiber = &fiber;
   FiberImpl::swap(from, to);
}

#if FLAX_USE_SCHEDULER
// static
void Fiber::yield() {
   assert(threadLocalData && threadLocalData->activeFiber && threadLocalData->scheduler);

   Fiber* nextFiber = threadLocalData->scheduler->next();
   if (!nextFiber) {
      return;
   }

   assert(nextFiber->isValid() && !nextFiber->isActive() && !nextFiber->isFinished());
   assert(nextFiber != threadLocalData->activeFiber);

   yieldTo(*nextFiber);
}

// static
void Fiber::setScheduler(std::unique_ptr<Scheduler> newScheduler) {
   assert(threadLocalData);

   if (newScheduler) {
      threadLocalData->scheduler = std::move(newScheduler);

      for (Fiber* fiber : threadLocalData->fibers) {
         assert(fiber->isValid());
         threadLocalData->scheduler->onFiberCreated(fiber);
      }

      if (threadLocalData->activeFiber) {
         threadLocalData->scheduler->onFiberYieldedTo(threadLocalData->activeFiber);
      }
   }
}
#endif // FLAX_USE_SCHEDULER

// static
void Fiber::fiberMain(Fiber* fiber) {
   assert(fiber && fiber->isValid() && fiber->isActive() && !fiber->isFinished() && !fiber->isMainFiber() && fiber->function != nullptr);

   fiber->function();

   fiber->finish();
#if FLAX_USE_SCHEDULER
   fiber->yield();
#else
   fiber->yieldTo(getMainFiber());
#endif // FLAX_USE_SCHEDULER

   assert(false); // Should never get here
}

Fiber::Fiber(const std::function<void()>& func, const std::string& name, bool isMainFiber)
   : function(func), fiberName(name), mainFiber(isMainFiber), finished(false), impl(FiberAndMain(this, &Fiber::fiberMain), isMainFiber) {
   if (isMainFiber) {
      assert(isValid() && function == nullptr && threadLocalData && threadLocalData->activeFiber == nullptr);
      threadLocalData->activeFiber = this;
   }

   data = threadLocalData;
   assert(data);

#if FLAX_USE_SCHEDULER
   if (isValid()) {
      data->fibers.push_back(this);
      data->scheduler->onFiberCreated(this);

      if (isMainFiber) {
         data->scheduler->onFiberYieldedTo(this);
      }
   }
#endif // FLAX_USE_SCHEDULER
}

Fiber::~Fiber() {
   assert(isActive() == isMainFiber());

   if (isValid() && !isFinished()) {
      finish();
   }
}

void Fiber::finish() {
   assert(isValid() && !isFinished());

   finished = true;

#if FLAX_USE_SCHEDULER
   data->scheduler->onFiberFinished(this);
   data->fibers.erase(std::remove_if(data->fibers.begin(), data->fibers.end(), [this](const Fiber* fiber) { return fiber == this; }), data->fibers.end());
#endif // FLAX_USE_SCHEDULER
}

} // namespace flax

Common.h

#ifndef FLAX_COMMON_H
#define FLAX_COMMON_H

namespace flax {

class Fiber;

using FiberMainFunction = void(*)(Fiber*);

struct FiberAndMain {
   Fiber* fiber;
   FiberMainFunction mainFunction;

   FiberAndMain(Fiber* fib, FiberMainFunction mainFunc)
      : fiber(fib), mainFunction(mainFunc) {
   }
};

} // namespace flax

#endif

FiberImpl.h

#ifndef FLAX_FIBER_IMPL_H
#define FLAX_FIBER_IMPL_H

#if FLAX_PLATFORM_WIN

#include "WindowsFiberImpl.h"
namespace flax {
using FiberImpl = WindowsFiberImpl;
} // namespace flax

#elif FLAX_PLATFORM_MAC || FLAX_PLATFORM_LINUX

#include "PosixFiberImpl.h"
namespace flax {
using FiberImpl = PosixFiberImpl;
} // namespace flax

#else

#error "Invalid platform"

#endif // FLAX_PLATFORM_<>

#endif

WindowsFiberImpl.h

#ifndef FLAX_WINDOWS_FIBER_IMPL_H
#define FLAX_WINDOWS_FIBER_IMPL_H

#include "Flax/Common.h"

namespace flax {

class WindowsFiberImpl {
public:
   WindowsFiberImpl(FiberAndMain fiberData, bool isMainFiber);

   ~WindowsFiberImpl();

   bool isValid() const {
      return address != nullptr;
   }

   static void swap(WindowsFiberImpl& from, WindowsFiberImpl& to);

private:
   void* address;
   bool mainFiber;
   FiberAndMain fiberAndMain;
};

} // namespace flax

#endif

WindowsFiberImpl.cpp

#include "Flax/WindowsFiberImpl.h"

#include <cassert>

#define WIN32_LEAN_AND_MEAN
#include <Windows.h>

namespace flax {

WindowsFiberImpl::WindowsFiberImpl(FiberAndMain fiberData, bool isMainFiber)
   : address(nullptr), mainFiber(isMainFiber), fiberAndMain(fiberData) {
   assert(fiberAndMain.fiber && fiberAndMain.mainFunction);

   if (mainFiber) {
      address = ConvertThreadToFiber(this);
      assert(address);
   } else {
      address = CreateFiber(0, [](LPVOID param) {
         FiberAndMain* fiberAndMain = reinterpret_cast<FiberAndMain*>(param);
         assert(fiberAndMain);

         fiberAndMain->mainFunction(fiberAndMain->fiber);
      }, &fiberAndMain);
   }
}

WindowsFiberImpl::~WindowsFiberImpl() {
   if (mainFiber) {
      assert(isValid());
      bool success = ConvertFiberToThread() != 0;
      assert(success);
   } else if (address) {
      DeleteFiber(address);
   }
}

// static
void WindowsFiberImpl::swap(WindowsFiberImpl& from, WindowsFiberImpl& to) {
   assert(from.isValid() && to.isValid() && GetCurrentFiber() == from.address && GetCurrentFiber() != to.address);
   SwitchToFiber(to.address);
}

} // namespace flax

PosixFiberImpl.h

#ifndef FLAX_POSIX_FIBER_IMPL_H
#define FLAX_POSIX_FIBER_IMPL_H

#include "Flax/Common.h"

#define _XOPEN_SOURCE
#include <ucontext.h>

#include <cstdint>
#include <memory>

namespace flax {

class PosixFiberImpl {
public:
   PosixFiberImpl(FiberAndMain fiberData, bool isMainFiber);

   ~PosixFiberImpl();

   bool isValid() const {
      return valid;
   }

   static void swap(PosixFiberImpl& from, PosixFiberImpl& to);

private:
   ucontext_t context;
   std::unique_ptr<uint8_t[]> stack;
   bool valid;
   FiberAndMain fiberAndMain;
};

} // namespace flax

#endif

PosixFiberImpl.cpp

#include "Flax/PosixFiberImpl.h"

#include <sys/resource.h>

#include <cassert>
#include <cstddef>

namespace flax {

namespace {

std::size_t getStackSize() {
   static std::size_t stackSize = 0;

   if (stackSize == 0) {
      rlimit limit = {};
      getrlimit(RLIMIT_STACK, &limit);

      stackSize = static_cast<std::size_t>(limit.rlim_cur);
   }

   return stackSize;
}

void fiberStart(FiberAndMain* fiberAndMain) {
   fiberAndMain->mainFunction(fiberAndMain->fiber);
}

} // namespace

PosixFiberImpl::PosixFiberImpl(FiberAndMain fiberData, bool isMainFiber)
   : context{}, stack(nullptr), valid(true), fiberAndMain(fiberData) {
   assert(fiberAndMain.fiber && fiberAndMain.mainFunction);

   valid = (getcontext(&context) == 0);

   if (isValid() && !isMainFiber) {
      std::size_t stackSize = getStackSize();
      stack = std::unique_ptr<uint8_t[]>(new uint8_t[stackSize]);

      context.uc_stack.ss_sp = stack.get();
      context.uc_stack.ss_size = stackSize;
      context.uc_link = nullptr;

      makecontext(&context, reinterpret_cast<void (*)()>(&fiberStart), 1, &fiberAndMain);
   }
}

PosixFiberImpl::~PosixFiberImpl() {
}

// static
void PosixFiberImpl::swap(PosixFiberImpl& from, PosixFiberImpl& to) {
   assert(from.isValid() && to.isValid());
   bool success = (swapcontext(&from.context, &to.context) == 0);
   assert(success);
}

} // namespace flax

Scheduler.h

#ifndef FLAX_SCHEDULER_H
#define FLAX_SCHEDULER_H

#include "Flax/Common.h"

namespace flax {

/*!
 * Scheduler used to select which Fiber to yield to when Fiber::yield() is called.
 */
class Scheduler {
public:
   virtual ~Scheduler() = default;

   /*!
    * Selects the next Fiber that should be yielded to. If there is no valid Fiber to yield to, it should return nullptr.
    */
   virtual Fiber* next() = 0;

   /*!
    * Called when a new Fiber is created. When a scheduler is set as the Fiber scheduler, this function will be called for all existing Fibers.
    */
   virtual void onFiberCreated(Fiber* fiber) = 0;

   /*!
    * Called when a Fiber finishes execution (or is destroyed).
    */
   virtual void onFiberFinished(Fiber* fiber) = 0;

   /*!
    * Called when a Fiber is yielded to (either due to being selected by this scheduler, or by being explicitly yielded to by the user).
    */
   virtual void onFiberYieldedTo(Fiber* fiber) = 0;
};

} // namespace flax

#endif

RoundRobinScheduler.h

#ifndef FLAX_ROUND_ROBIN_SCHEDULER_H
#define FLAX_ROUND_ROBIN_SCHEDULER_H

#include "Flax/Scheduler.h"

#include <algorithm>
#include <cassert>
#include <deque>

namespace flax {

/*!
 * Round robin Fiber scheduler.
 */
class RoundRobinScheduler : public Scheduler {
public:
   RoundRobinScheduler()
      : scheduledFiber(nullptr) {
   }

   virtual ~RoundRobinScheduler() = default;

   virtual Fiber* next() override {
      if (fiberQueue.empty()) {
         return nullptr;
      }

      Fiber* lastScheduledFiber = scheduledFiber;
      scheduledFiber = fiberQueue.front();
      fiberQueue.pop_front();

      if (lastScheduledFiber) {
         fiberQueue.push_back(lastScheduledFiber);
      }

      return scheduledFiber;
   }

   virtual void onFiberCreated(Fiber* fiber) override {
      assert(fiber && scheduledFiber != fiber && std::find(fiberQueue.begin(), fiberQueue.end(), fiber) == fiberQueue.end());

      fiberQueue.push_back(fiber);
   }

   virtual void onFiberFinished(Fiber* fiber) override {
      assert(fiber && (fiber == scheduledFiber || std::find(fiberQueue.begin(), fiberQueue.end(), fiber) != fiberQueue.end()));

      if (scheduledFiber == fiber) {
         scheduledFiber = nullptr;
      }

      fiberQueue.erase(std::remove_if(fiberQueue.begin(), fiberQueue.end(), [fiber](Fiber* element) { return element == fiber; }), fiberQueue.end());
   }

   virtual void onFiberYieldedTo(Fiber* fiber) override {
      assert(fiber && (fiber == scheduledFiber || std::find(fiberQueue.begin(), fiberQueue.end(), fiber) != fiberQueue.end()));

      while (fiber != scheduledFiber) {
         next();
      }
   }

private:
   std::deque<Fiber*> fiberQueue;
   Fiber* scheduledFiber;
};

} // namespace flax

#endif

Example

Code:

#include <Flax/Fiber.h>
#include <cstdio>

static void foo() {
   std::printf("foo() begin\n");
   flax::Fiber::yield();
   std::printf("foo() end\n");
}

static void bar() {
   std::printf("bar() begin\n");
   flax::Fiber::yield();
   std::printf("bar() end\n");
}

static void baz() {
   std::printf("baz() begin\n");
   flax::Fiber::yield();
   std::printf("baz() end\n");
}

int main(int argc, char* argv[]) {
   // Creating a fiber does not immediately begin its execution.
   std::unique_ptr<flax::Fiber> fooFiber = flax::Fiber::create(foo, "Foo Fiber");
   std::unique_ptr<flax::Fiber> barFiber = flax::Fiber::create(bar, "Bar Fiber");
   std::unique_ptr<flax::Fiber> bazFiber = flax::Fiber::create(baz, "Baz Fiber");

   // Calling Fiber::yield() uses a scheduler to select the next fiber.
   // By default, a round robin scheduler is used (though you can provide your own).
   std::printf("Before yield() calls\n");
   flax::Fiber::yield();
   std::printf("Between yield() calls\n");
   flax::Fiber::yield();
   std::printf("After yield() calls\n");

   // Fiber::create() takes a std::function, so you can pass lambdas that capture variables.
   int x = 0;
   std::unique_ptr<flax::Fiber> lambdaFiber = flax::Fiber::create([&x]() { ++x; }, "Lambda Fiber");

   // You can use Fiber::yieldTo() to explicitly yield to another Fiber.
   std::printf("Before yieldTo(): %d\n", x);
   flax::Fiber::yieldTo(*lambdaFiber);
   std::printf("After yieldTo(): %d\n", x);

   return 0;
}

Output:

Before yield() calls
foo() begin
bar() begin
baz() begin
Between yield() calls
foo() end
bar() end
baz() end
After yield() calls
Before yieldTo(): 0
After yieldTo(): 1
\$\endgroup\$
  • \$\begingroup\$ Just curious, why use printf instead of cout? \$\endgroup\$ – pacmaninbw May 14 '17 at 20:15
  • \$\begingroup\$ @pacmaninbw Just personal preference, it can be changed if the general consensus is that cout would be preferable. \$\endgroup\$ – Samusaaron3 May 14 '17 at 20:18
  • \$\begingroup\$ Have you successfully built this on a Mac, and if so, what version of OSX and what version of Xcode? \$\endgroup\$ – pacmaninbw May 14 '17 at 22:08
  • 1
    \$\begingroup\$ @pacmaninbw Yes, on macOS Sierra (10.12.4) and Xcode 8.3.2 (most recent version of both). \$\endgroup\$ – Samusaaron3 May 14 '17 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.