6
\$\begingroup\$

We're using std::chrono::steady_clock for most of our internal queues and relative timings. However, we're on a POSIX platform and steady_clock is implemented by using clock_gettime(CLOCK_MONOTONIC, ....

The CLOCK_MONOTONIC clock is not affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the clock), but is affected by the incremental adjustments performed by adjtime(3) and NTP. This clock does not count time that the system is suspended. All CLOCK_MONOTONIC variants guarantee that the time returned by consecutive calls will not go backwards, but successive calls may—depending on the architecture—return identical (not-increased) time values.

Since steady_clock is affected by certain clock adjustments, in some parts of our codebase clock_gettime(CLOCK_MONOTONIC_RAW, ... is therefore used directly instead, which is ...

Similar to CLOCK_MONOTONIC, but provides access to a raw hardware-based time that is not subject to NTP adjustments or the incremental adjustments performed by adjtime(3). This clock does not count time that the system is suspended.

I'd like to encapsulate it in a class that could work as a replacement for steady_clock and came up with the below which is inspired by the gcc steady_clock implementation.

  • I wonder if this is all there is to it or if I'm missing anything to make it fulfill the TrivialClock requirements?
#pragma once       // or a classic portable header guard

#include <chrono>
#include <ctime>

namespace foo {

struct monotonic_raw_clock {
    using duration = std::chrono::nanoseconds;
    using rep = duration::rep;
    using period = duration::period;
    using time_point = std::chrono::time_point<monotonic_raw_clock, duration>;

    static constexpr bool is_steady = true;

    static inline time_point now() noexcept {
        std::timespec tp;
        // The return value from clock_gettime is ignored in the gcc steady_clock
        // implementation too:
        static_cast<void>(clock_gettime(CLOCK_MONOTONIC_RAW, &tp));

        return time_point(duration(std::chrono::seconds(tp.tv_sec) +
                                   std::chrono::nanoseconds(tp.tv_nsec)));
    }
};

}  // namespace foo
\$\endgroup\$

2 Answers 2

6
\$\begingroup\$

The implementation looks correct to me. I do want to address some potential misunderstanding though:

Since steady_clock is affected by certain clock adjustments, […]

It sounds scary, but it's actually much more benign, and might even be desirable. The steady clock is certainly not "jumping". What happens is that you have the computer's internal crystal oscillator that is typically not of great quality, and is speeding up and down all the time because of temperature and voltage fluctuations. Its average speed might even be several percentage points slower or faster than wall clock time. Your computer corrects for that by checking it regularly against NTP time, and then compensating for the speed difference. So the adjustments done are very gradual, and in the end you have a clock which is much more steady and correct than CLOCK_MONOTIC_RAW is.

The only advantage of CLOCK_MONONOTIC_RAW might have over CLOCK_MONOTONIC is that it could be faster (but on Linux it likely has the same performance). So I recommend that you use it only if you call now() very often and need it to be as fast as possible, and don't care about the accuracy and precision of the clock.

\$\endgroup\$
7
  • \$\begingroup\$ Why (and how much?) is CLOCK_MONOTONIC_RAW faster? I'd have thought it would still use rdtsc and some scale factors to interpolate since the last tick of CLOCK_MONOTONIC_COARSE (which is the fast one, just reading the timestamp from the last timer interrupt). So my assumption was that the cost of tweaking scale factors and offsets for MONOTONIC vs. MONOTONIC_RAW would be paid in the timer interrupt, not the calls using it. Is that not correct, or does non-RAW have to do some additional work inside clock_gettime with an additional offset and/or scale factor? \$\endgroup\$ Commented Aug 28, 2023 at 20:01
  • 2
    \$\begingroup\$ (They're all fairly fast, since they're implemented via the VDSO on Linux; the kernel exports some code + data (with ELF metadata) into the address space of every process. So the scale factors, and tick updated by the kernel's timer interrupt, are directly readable from user-space.) \$\endgroup\$ Commented Aug 28, 2023 at 20:03
  • \$\begingroup\$ Indeed, it seems there is no performance difference on amd64 Linux; I guess to convert to nanoseconds you need an offset and scale factor anyway, so there's no extra cost to take into account the corrections. Only CLOCK_MONOTONIC_COARSE is faster on that platform. These results will likely be different on other platforms though. \$\endgroup\$
    – G. Sliepen
    Commented Aug 28, 2023 at 21:59
  • 2
    \$\begingroup\$ Thanks for the answer and insights! I asked around about the reasons for why we use CLOCK_MONOTONIC_RAW in some places and the feeling I got was that being slightly slow or fast wasn't as big of an issue as it would be if we got an adjustment that was "too big" (> ~4-5 milliseconds I think). @PeterCordes I did some measurements @ quick-bench Yesterday and the raw clock was consistently faster, but only extremely little. I haven't benched it on our little target node though. \$\endgroup\$
    – Ted Lyngmo
    Commented Aug 29, 2023 at 8:19
  • 2
    \$\begingroup\$ @TedLyngmo: Thanks for the quick-bench link. Yeah, it seems RAW is consistently a tiny bit faster, maybe about 1%. vs. COARSE being 100x faster when benchmarked for throughput in a loop. So caches stay hot, getting an L1d hit for the variable written by a timer interrupt. If it's not a per-core variable, that'll only happen under similarly heavy usage many times per ms. But the non-COARSE versions still have to run rdtsc. IIRC from single-stepping through it a while ago, they run rdtsc multiple times for some reason. It has 25c throughput on Skylake, 27c on Ice Lake, 36c on Zen 3. \$\endgroup\$ Commented Aug 29, 2023 at 8:45
2
\$\begingroup\$

The source code looks good. Merge to main and ship it.

Consider writing test code which highlights observed behavior difference between this and competing time sources.

I worry that using "raw" without frequency correction will produce output which isn't directly comparable with the S.I. second.


The code relies on some assumptions the OP explained:

Since steady_clock is affected by certain clock adjustments, in some parts of our codebase ...

I believe "affected" is a smaller concern than you feel it is. Some review context is missing. I could be more specific if the OP had described the app-level concerns about error magnitudes, and what portion of the error budget is consumed by non-clock sources of error.

(1.) apply offset

Yes, upon bootup we may notice the system time is e.g. more than an hour off from NTP time, so we immediately apply a large + or - offset.

In principle this can happen after the OS has been running for a while. Applications rightly fear this, and they properly choose a monotonic clock which will never go backwards. In practice, for an OS that boots up and runs continuously for some number of days, we wouldn't really expect to see more than one offset applied during those days. Even the laptop "suspend" scenario should work smoothly with time-sensitive apps.

(2.) slew rate

Dave Mills originally designed adjtime(2) to run the clock 10% faster or 10% slower until it drove the NTP error to approximately zero.

Modern usage is significantly more conservative. The (darwin) mac manpage explains:

adjtime – correct the time to allow synchronization of the system clock

adjtime() makes small adjustments to the system time, as returned by gettimeofday(2), advancing or retarding it by the time specified by the timeval delta. If delta is negative, the clock is slowed down by incrementing it more slowly than normal until the correction is complete. If delta is positive, a larger increment than normal is used.

The skew used to perform the correction is generally a fraction of one percent. Thus, the time is always a monotonically increasing function.

BSD source code explains that typically if the slew rate is non-zero it will adjust by just half a millisecond every second:

/* Apply any correction from adjtime(2). If more than one second off we slew at a rate of 5ms/s (5000 PPM) else 500us/s (500PPM) until the last second is slewed the final < 500 usecs. */

The linux source is less explicit:

/* If the clock is behind the NTP time, increase the multiplier by 1 to catch up with it. If it's ahead and there was a remainder in the tick division, the clock will slow down. Otherwise it will stay ahead until the tick length changes to a non-divisible value. */

Sorry, I wasn't able to divine any numeric bounds from that. But maybe someone can, and will post a comment?

BTW, time daemons on linux prefer this interface:

adjtimex(2) is Linux-specific and should not be used in programs intended to be portable. See adjtime(3) for a more portable, but less flexible, method of adjusting the system clock.


iburst

Time daemons would traditionally boot up then measure time and frequency offsets for at least 64 seconds, often for a few minutes, before issuing the first correction through a kernel interface. In part this conservative behavior was to avoid {DDoS, network load, traffic amplification} effects seen by central time servers.

And then came iburst, the "initial burst" of time queries. Trust me, you really want at least one reachable iburst server in your NTP config, it's pretty essential.

The idea is that biggest NTP error offset will typically be observed right around boot time, and we should deal with it fast, without awaiting the results of careful long-term observation. So instead of having "candidate" time servers for the first few minutes after boot, a time client will quickly settle on one and issue a local correction. As the minutes go by it might decide that actually another server appears to be optimal, and switch to that. But we're probably within 10 msec of true time at that point, so apps are unlikely to notice the switch.


leap second

In principle a POSIX app can observe "23:59:59" tick over to "23:59:60" during a leap second.

In practice only astronomers care about such details, and everyone else would rather pretend the earth isn't slowing down. Typically a time daemon will "smear" that one second of error across a few hours prior to UTC midnight.

You should either document such concerns for callers of your API and author relevant unit tests, or explicitly write down "it just doesn't matter!"


tl;dr: The implementation looks good. Depending on app details we don't know about, there possibly is no need to {implement, document, test, support} this API.

\$\endgroup\$
7
  • \$\begingroup\$ Thanks for this view on it too! I have considered diving into the documentation and if possible source code of the adjtime we use on target to see what kind of adjustments we could expect. If it follows the slow slew rate of BSD, it does sound like we should be ok. We want to keep it under ~4-5ms in "one jump" if I understood the requirements (that I got from a second hand source, not from anyone who actually had used CLOCK_MONOTONIC_RAW in our code). I doubt we'll be > 1s off at any time but even if so, 5ms/s slew is, I assume, not applied all at once. \$\endgroup\$
    – Ted Lyngmo
    Commented Aug 29, 2023 at 21:11
  • \$\begingroup\$ My point was that after we've booted up for a little while, there are essentially no "jump"s at all. "I assume, applied not all at once." I don't understand why you would make that remark, having gone to the trouble of implementing an alternate time source. In a sense every tiny adjustment is instantaneous. But the whole point of adjtime(2) is to spread out the adjustment. So if e.g. we have a twenty millisecond error w.r.t. NTP, the time daemon will ask the kernel to "fix it" and at 5 ms/s it will take four seconds to gradually drive the NTP error to zero, or forty at .5 ms/s. \$\endgroup\$
    – J_H
    Commented Aug 29, 2023 at 21:15
  • \$\begingroup\$ And about: "using "raw" without frequency correction will produce output which isn't directly comparable with the S.I. second" - Yes, and if I understood it correctly, the raw clock is used where it doesn't matter if it's a little slow or a little fast, just as long as there are no sudden "big" adjustments. \$\endgroup\$
    – Ted Lyngmo
    Commented Aug 29, 2023 at 21:15
  • 1
    \$\begingroup\$ Re: "My point was that after we've booted up for a little while, there are essentially no "jump"s at all." - True, but there's a lot happening when it boots. I will need to check if any of the uses of the raw clock is in those parts of the code. \$\endgroup\$
    – Ted Lyngmo
    Commented Aug 29, 2023 at 21:23
  • 1
    \$\begingroup\$ Thanks, these are great hints for where to dig! I got the feeling (on a git grep CLOCK_MONOTONIC_RAW level) that some of the uses were in driver code but for the hits I got in the layers above, we should probably just go for steady_clock. There might be some legacy concerns involved too that may not be issues anymore. \$\endgroup\$
    – Ted Lyngmo
    Commented Aug 29, 2023 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.