Seastar's gate utility

There are a few common scenarios that arise when using Seastar that require controlling the lifetime of resources referenced by asynchronous fibers (e.g. background fibers). In each of these cases it is generally necessary to wait until all possible references have been released before destroying a resource. This situation arises often enough that Seastar provides the seastar::gate utility which implements an easy-to-use pattern for solving the problem.

Consider the following example service that starts a background fiber performing units of work in a loop until the service is stopped (e.g. accepting network connections). The curious (void)func_returning_future(...) pattern creates and schedules a future, but explicitly ignores the returned future. The resulting fiber has no parent, no reference, and operates in the background.

class service {
public:
  seastar::future<> start() {
     (void)seastar::do_until(
       [this]() { return stopped_; },
       [this]() { return do_work(); });
     return seastar::now();
  }

  seastar::future<> stop() {
     stopped_ = true;
     // ???
  }

private:
  bool stopped_{false};

  seastar::future<> do_work() { ... }
};

Once it is time to shutdown a service, the question remains: when is it safe to delete the service instance? This question is important because it is common to see a pattern in which a service is stopped and then immediately deleted.

auto s = new service;
co_await s->start();
...
co_await s->stop();
delete s;

When stop is called and stopped_ is set to true we can be sure that the background work fiber will eventually stop, but if the background fiber has a reference to the service stop also needs to wait for the background worker fiber to finish because the caller is going to delete the service as soon as stop’s future resolves.

A seastar::gate can be thought of as a sort of one shot barrier providing a synchronization point, and is useful for solving this problem. Here is the updated example using a gate to address the issue. The work is now started inside the gate, and stop waits for the gate to close. A gate is closed once all fibers started inside the gate have resolved.

class service {
public:
  seastar::future<> start() {
     (void)seastar::with_gate(gate_, [this] {
        return seastar::do_until(
          [this]() { return stopped_; },
          [this]() { return do_work(); });
     });
     return seastar::now();
  }

  seastar::future<> stop() {
     stopped_ = true;
     return gate_.close();
  }

private:
  bool stopped_{false};
  seastar::gate gate_;

  seastar::future<> do_work() { ... }
};

Referring back to the shutdown scenario, the caller of service::stop can now reason about the point at which it is safe to destroy the service instance. Although there is only one fiber inside the gate in this example, a gate can handle any number. This makes it useful for also tracking futures returned to callers, such as references to a network connection maintained in the service which the caller interacts with asynchronously.

Gate internals #

A gate is effectively a counter that records the number of futures started inside the gate, and a flag that indicates that the gate is stopped (or closed per the interface-level terminology).

class gate {
    size_t _count = 0;
    compat::optional<promise<>> _stopped;

A fiber enters a gate by invoking gate::enter which increases the internal counter, provided that the gate is not already closed.

    void enter() {
        if (_stopped) {
            throw gate_closed_exception();
        }
        ++_count;
    }

Closing the gate with gate::close prevents any new fibers from entering the gate by setting the _stopped flag, which also serves as the promise that will be set when all futures leave the gate. Yes, the stopped flag and promise are the same, which is an example of low-level optimizations seen around the Seastar code base.

    future<> close() {
        assert(!_stopped && "seastar::gate::close() cannot be called more than once");
        _stopped = compat::make_optional(promise<>());
        if (!_count) {
            _stopped->set_value();
        }
        return _stopped->get_future();
    }

The final piece to the puzzle is leaving the gate. Leaving decreases the counter, and if the gate is closed and the fiber leaving is the last, it sets the promise. So, until the gate is closed, leaving only decreases the counter.

    void leave() {
        --_count;
        if (!_count && _stopped) {
            _stopped->set_value();
        }
    }

The careful observe will note that gates are safe to enter more than once provided that the gate is left an equal number of times. This reentrant property is rarely exploited, but I have encountered it a few times where shared code paths have multiple entry points.

Usage #

The most common way to use a gate is with the seastar::with_gate helper that attempts to enter the gate, and if successful, returns the passed continuation with an attached finally block so that it will always call gate::leave.

template <typename Func>
inline auto
with_gate(gate& g, Func&& func) {
    g.enter();
    return futurize_apply(std::forward<Func>(func)).finally([&g] { g.leave(); });
}

The last bit of seastar::gate that I’ll mention is gate::is_closed for checking if the gate has been closed (without throwing an exception). This turns out to be useful in a few different contexts. It is quite common for a gate to wrap a fairly heavy weight fiber that may use many resources or require a long execution time. Given the ability to query the state of the gate, such a fiber can often implement a form of fast shutdown by periodically checking if the gate is closed and returning without performing expensive operations.

    bool is_closed() const {
        return bool(_stopped);
    }

Used in this manner, the gate::is_closed method is effectively used as a signalling mechanism to fibers that they should shutdown. In later posts we’ll look at the Seastar abort source utility which provides a more feature rich form of signaling mechanism for stopping fibers.