What's new in Seastar - issue 4

Slowdown IO scheduler based on dispatched/completed ratio #1766 #

Modeling disks and I/O scheduling is hard. Really hard. There is plenty of discussion and mathematical equations in the pull request, but if I’m being honest, I didn’t read through most of it. I think for the most part this change shouldn’t be visible to users.

RPC metrics #1753 #

If you are using Seastar’s RPC framework, then this PR added some metrics available through the /metric Prometheus endpoint.

How would you implement coroutine cancellation? #

An interesting discussion on the mailing list about seastar::abort_source and creating coroutines that can be cancelled. The person who started the thread observes that users of seastar::abort_source need to pass references around explicitly, and that this adds lots of code (error prone) and expands parameter lists (ugly) and users still need to propagate the abort source to code that itself invokes.

This is a fair point. It’s something that we’ve had to deal with in Redpanda. I think the general consensus is of the thread was that while yes it would be possible to hide things like references to abort sources in the coroutine frame, and even propagate them via something like current_task, the solution would (1) add some overhead and (2) not be a general solution. Still, an interesting thought about how we can make Seastar easier to use.

Added ability to configure different credentials per HTTP listeners #1840 #

Seastar HTTP server listeners can use different credentials. At Redpanda this is useful because we may need to serve public and internal facing services with different sets of certificates.

Add file::list_directory() that co_yields entries #1688 #

At Redpanda we use C++ coroutines extensively, but for whatever reason, we rarely if ever find ourselves needing to use co_yield. So this PR is cool because I’ve been wanting to get some time experimenting with co_yield. It seems like it can produce some really nice looking solutions.

Here is directory listing with a coroutine generator. In contrast, we use Seastar’s version of directory listing which still uses continuation passing (i.e. passing in a callback function). It’s not nearly as nice looking as this.

    auto lister = f.experimental_list_directory();
    while (auto de = co_await lister()) {
        auto sd = co_await file_stat(de->name, follow_symlink::no);

And for the library writer, it seems like co_yield makes building generators really nice:

    while (true) {
        auto de = co_await ents.pop_eventually();
        if (!de) {
            break;
        }
        co_yield *de;
    }

tls: linearize small packets on send #1965 #

This PR from Redpanda is really nice. Our performance team observed that Redpanda had high CPU utilization and excessive calls to sendmsg when using TLS compared to non-TLS setups. It was eventually discovered that the fragmented memory represented by seastar::packet was being dispatched at the granularity of the fragments, often 3 or more small fragments per packet structure. It seems as though the code had assumed that either fragmentation wouldn’t be common, or the fragments would have been larger.

The solution in this PR was to linearize the fragments if the total size was under some threshold (e.g. 16 KB in the PR), so that the TLS machinery was only invoked once. And the results? Good stuff from the pull request cover letter:

p50: 7.2ms -> 5.6ms
p99: 35ms -> 15ms
p999: 74ms -> 43ms

At the same time reactor utilization goes down by ~3%. Note due to how
batching/lingering works in Kafka clients this is while doing ~5% more
RPS (with decresed batch size).

We can further confirm that the amount of sendmsg syscalls we do is
down by a factor of ~3 (before vs. with this patch):

shared_future: make available() immediate after set_value() #1958 #

Before this change there was a delay before seastar::shared_future::available would be true after setting a value on the associated promise. This can be surprising behavior for users coming from the non-shared variants of seastar::future and seastar::promise. This PR changes that so that available is updated immediately.