What's new in Seastar - issue 1

At Redpanda we recently updated the version of Seastar we use, as we do from time to time. I realized that even though we had been updating regularly, I hadn’t been paying close attention to all the interesting bits that have landed in some time. So I thought I’d take the opportunity to summarize what I found scanning the Git history one recent weekend afternoon.

The range covered here is roughly since the beginning of 2022. There was something like 1000+ commits. I wasn’t able to cover everything, but I think I was able to identify many of the highlights.

[noah@node seastar]$ git diff --shortstat cf0b43c52de0cd3c678879ea238dfe5edefb6a9a
 356 files changed, 20537 insertions(+), 4999 deletions(-)

In addition to the feature-shaped items listed below, there was a lot of change across the board that I don’t have time to mention individually. These changes included:

• Big changes to I/O queueing
• Documentation improvements
• Build system improvements and clean-up
• Generous use and improvement of C++ concepts
• Refactoring to simplify parts of the code base
• More coroutine utilities
• Fixes for memory leaks
• Fixes for use-after-free bugs
• Fixes for broken promises
• Fixes for DNS (UDP concurrency issue, and TCP errors)
• Fixes for HTTP header parsing
• Fixes for missed syscall thread reactor wakeup

And many more that will have to go unmentioned. Now let’s move on to the list of features and other general improvements.

Level 6 API is now the default #

Seastar uses a rolling source-level backwards compatibility policy, and when compiling Seastar an API level can be chosen. This commit bumps the default level to 6 allowing old code and interfaces to be retired completely from the tree.

commit b40743e9a3490fda61a373f482cae35963d36cee
Author: Avi Kivity <>
Date:   Wed Mar 29 14:26:34 2023 +0300

    future: make API level 6 mandatory
    
    In e215023c78b0e ("future: Make futures non variadic"), we made futures
    and related types non-variadic, and introduced API level 6 to ease
    transition. 2½ years later we retire variadic futures completely and
    make API level 6 mandatory.

With this version bump there is no longer support in Seastar for variadic type futures such as future<int, float>, only future<T>.

Websocket server #

Websocket server support has been trickling into the tree now for quite some time since late 2021, but I’ve not yet had a chance to use it. It looks like development is active and bugs are getting fixed.

This is a nice feature and I’m looking forward to using it.

commit 2dee550bc62664894995025edde1f645be86381e
Merge: f3749938 33006fdf
Author: Avi Kivity <>
Date:   Wed Jun 22 15:58:39 2022 +0300

    Merge 'Add initial support for websocket protocol' from Andrzej Stalke

commit 4051d8b66a4b075e7a56ab91605e1360fcde67f8
Author: Piotr Sarna <>
Date:   Thu Oct 14 16:42:52 2021 +0200

    websocket: add skeleton implementation of a websocket server
    
    This commit adds minimal foundations for a WebSocket server
    implementation for Seastar.

Faster reactor reactivity #

Some system calls are blocking so Seastar runs these on non-reactor threads. These systems calls are placed into a queue for processing. Once a system call completes the reactor may be sleeping and needs to be woken up. However, Seastar was processing all the system calls queued up before waking up the reactor, leading a sort of head of line blocking issue (in reverse) causing higher than necessary latency for system calls at the front of the queue. The change wakes up the reactor after each system call completes.

commit ecf98a507d88af8ea74716806f94f35906e85899
Author: Avi Kivity <>
Date:   Sun Feb 19 20:43:42 2023 +0200

    reactor: syscall thread: wakeup up reactor with finer granularity

Safety checks for semaphore and gate #

Semaphores generally need to be moved, such as when data structures are being created and wired up. But once things are ready to go, semaphores really shouldn’t be moved in order to maintain a stable memory address since things like RAII unit objects hold on to a reference. So instead of marking semaphores move-only, this commit adds an opt-in runtime check that moved-from semaphores aren’t used.

There is a similar debug feature added for the Seastar gate.

commit 9b6e181e424fd18b780ade4df59e2c0c3a806742
Author: Benny Halevy <>
Date:   Sun Feb 5 16:10:11 2023 +0200

    semaphore: disallow move after used

commit 695dcd4df1113301b15d9cc8f7494294b42636f3
Author: Benny Halevy <>
Date:   Sat Dec 3 11:24:10 2022 +0200

    gate: add gate holders debugging
    
    gate::holder keeps a naked pointer to the gate it holds.
    Its life time is independent of the gate and there is
    no mechanism to update this pointer if the gate is moved,
    therefore if the gate is moved or destroyed while there are
    outstanding holders pointing to it there is a risk for
    use-after-move or use-after-free.

Support shared library builds #

This is a huge quality of life improvement for big projects using Seastar. We recently enabled shared library builds for Redpanda where we have a lot of test binaries. We were able to knock off around 200 GB from the total on-disk build size.

commit f34335e5fac747a69800ecc67aa50897ce6f0f36
Author: Kefu Chai <>
Date:   Thu Feb 9 12:41:27 2023 +0800

    build: do not always build seastar as a static library

C++20 modules support #

Well, not quite yet. But this commit puts down some of the early ground work. Follow the next stage of development here https://github.com/scylladb/seastar/pull/1605.

commit 61c63abd1c9f43f942c1958aad16753df8030231
Author: Kefu Chai <>
Date:   Mon Feb 6 11:26:19 2023 +0800

    treewide: add non-seastar "#include"s

Additional memory reservation knob #

This looks like a reservation that reduces the amount of memory that the Seastar allocator sees on top of the reservation given to the OS for other resources. Presumably this is intended to be used when a Seastar application has a component that is bypassing the Seastar allocator. Now there is a mechanism for reserving that bucket of memory.

commit 88b85b882172fa09ae294e15316ecc1b73cfd38f
Author: Wojciech Mitros <>
Date:   Fri Dec 2 18:44:13 2022 +0100

    allocator: add an option to reserve additional memory for the OS
    
    This patch adds an option to specify the memory reserved for the
    OS not as a flat value, but as a modifier for the recommended
    amounts.
    
    The intended use case of this option is when the application
    is using both seastar and system allocators and it relies on
    the seastar defaults for the reservation of memory reserved
    for the OS but wants to make sure that there is enough memory
    not managed by seastar allocator for both the application
    and the rest of the system.

Related #

• https://github.com/scylladb/seastar/commit/4eb26d4add70c442775ef0a391966e8c3e097c85
• https://github.com/scylladb/seastar/commit/e15ca5f675e43de0131fda660702babc381f8f7a

Re-entrant abort source request #

Now abort_source users can request_abort any number of times. This change is a nice quality of life improvement. The restriction before seems to have just been related to an implementation detail rather than something fundamental about the intended semantics of the interface.

commit fc59dfaa2607489a3f28d204f2670c715ccab3dc
Author: Aleksandra Martyniuk <>
Date:   Mon Jan 16 14:43:42 2023 +0100

    abort_source: allow multiple request_abort()

Dynamic adjustments to metrics and other improvements #

Nice improvement to metrics allowing them to be dynamically changed in several ways like their labels as well as enabling and disabling them. This is useful for fine-tuning to reduce the amount of data published by a system to only what is necessary. Other improvements to reducing metrics size includes histogram summarization, and metrics aggregations which allow metrics to be rolled up into per-node rather than per-shard values.

commit 364ca84d6e59253b8f4a480b57030d187747f596
Author: Amnon Heiman <>
Date:   Thu Dec 15 18:46:17 2022 +0200

    Add an API for the metrics layer to manipulate metrics dynamically.

commit 49628ceea91ce33f7660886a54e92f61574cc73b
Merge: fe8fcd6d 39695cdd
Author: Nadav Har'El <>
Date:   Mon Nov 14 14:21:54 2022 +0200

    Merge 'Add Prometheus filtering capabilities by label' from Amnon Heiman

commit ce78808580f549b3c39c5aa80d2ce95e2ece37a0
Merge: dc591d21 84aa236f
Author: Nadav Har'El <>
Date:   Thu Jun 30 14:29:33 2022 +0300

    Merge 'Added summaries, remove empty, and aggregation to Prometheus' from Amnon Heiman

Use io_uring as the default backend #

Very cool to see this change. Seems that both disk and network I/O can now take advantage of io_uring.

commit eedca15f16c3b6eae3d3d8af9510624a93f5d186
Author: Kefu Chai <>
Date:   Wed Oct 12 18:31:54 2022 +0800

    reactor: make io_uring the default backend if available

commit df9c4c638d39e3500a69a279a531112c118937e8
Merge: b5ac3a63 5d8d5ae6
Author: Avi Kivity <>
Date:   Tue Oct 11 20:46:36 2022 +0300

    Merge 'reactor: support more network ops in io_uring backend' from Kefu Chai

Support for process control #

This is Seastar’s take on fork + exec, and it’s an interface for starting and communicating with an external process. Looks like one of the first use cases was to run an external program in Scylla to collect cluster topology information.

commit f009c4443f9885e7d289bb868e1b2fd35f25e4ae
Author: Kefu Chai <>
Date:   Fri Nov 4 02:31:16 2022 +0800

    util, core: add spawn_process() helper

commit f323da3bff2a58d172d73349b7230ba652337807
Author: Kefu Chai <>
Date:   Thu Dec 15 22:51:32 2022 +0800

    reactor: add process::terminate() and process::kill()

commit 2afaa12517846a0bfb5a4aa3220180c6ecbffc04
Author: Kefu Chai <>
Date:   Mon Nov 21 09:51:54 2022 +0800

    reactor: use make_pipe() in reactor::spawn()

New HTTP client #

This is a nice complement to the existing HTTP server available to Seastar applications. This wasn’t available when we needed an HTTP client in Redpanda and we ended up building one based on Boost Beast which was also custom tailored to our internal fragmented memory utility.

commit 67988bc874c0e20908489ac6d56c49ef0c2e443a
Merge: 925b4fb5 6a0f0016
Author: Avi Kivity <>
Date:   Tue Nov 22 17:21:46 2022 +0200

    Merge 'Add simple http client' from Pavel Emelyanov

Sharded parameter unwrapping #

Nice quality of life improvement. Commit message below speaks for itself.

commit 4ec599d1bc1b20c2ab0c3c102c7c84e11222ca36
Author: Pavel Emelyanov <>
Date:   Wed Aug 31 15:50:51 2022 +0300

    sharded: Support sharded parameters in sharded<>::invoke_on_all()
    
    Currently it's impossible to shortcut passing one sharded service
    reference into another sharded service method. Like this
    
        sharded<service> srv;
        sharded<dependency> dep;
        srv.invoke_on_all(&service::fn, std::ref(dep));
    
    where
    
        auto service::fn(dependency& dep);
    
    Instead, one needs to write a complete lambda as an argument to
    invoke_on_all(). However, calling srv.start(std::ref(dep)) is
    perfectly possible thanks to automagical sharded parameters
    unwrapping.

Condition variable coroutine optimizations #

Interesting optimization of condition variables for use in coroutine contexts. I think the original API will still work as expected in either case, but this could be nice for reducing some latency by avoiding memory allocations.

commit 91ed5a1b858ee12ae17872a4b27d729629f1c165
Author: Michał Chojnowski <>
Date:   Wed Apr 5 18:30:21 2023 +0200

    condition-variable: replace the coroutine wakeup task with a promise

commit 4e42a60199a26b3e784459c0ba476852e5deb371
Author: Calle Wilund <>
Date:   Mon Mar 7 11:41:57 2022 +0000

    condition_variable: Add coroutine-only "when" operation folding waiter into parent frame
    
    Puts the waiter structure into the coroutine awaiter object
    located in calling coroutine frame, allowing alloc-and-extra continuation
    free wait.