At Redpanda we recently updated the version of Seastar we use, as we do from time to time. I realized that even though we had been updating regularly, I hadn’t been paying close attention to all the interesting bits that have landed in some time. So I thought I’d take the opportunity to summarize what I found scanning the Git history one recent weekend afternoon.
The range covered here is roughly since the beginning of 2022. There was something like 1000+ commits. I wasn’t able to cover everything, but I think I was able to identify many of the highlights.
[noah@node seastar]$ git diff --shortstat cf0b43c52de0cd3c678879ea238dfe5edefb6a9a 356 files changed, 20537 insertions(+), 4999 deletions(-)
In addition to the feature-shaped items listed below, there was a lot of change across the board that I don’t have time to mention individually. These changes included:
- Big changes to I/O queueing
- Documentation improvements
- Build system improvements and clean-up
- Generous use and improvement of C++ concepts
- Refactoring to simplify parts of the code base
- More coroutine utilities
- Fixes for memory leaks
- Fixes for use-after-free bugs
- Fixes for broken promises
- Fixes for DNS (UDP concurrency issue, and TCP errors)
- Fixes for HTTP header parsing
- Fixes for missed syscall thread reactor wakeup
And many more that will have to go unmentioned. Now let’s move on to the list of features and other general improvements.
Level 6 API is now the default #
Seastar uses a rolling source-level backwards compatibility policy, and when compiling Seastar an API level can be chosen. This commit bumps the default level to 6 allowing old code and interfaces to be retired completely from the tree.
commit b40743e9a3490fda61a373f482cae35963d36cee Author: Avi Kivity <> Date: Wed Mar 29 14:26:34 2023 +0300 future: make API level 6 mandatory In e215023c78b0e ("future: Make futures non variadic"), we made futures and related types non-variadic, and introduced API level 6 to ease transition. 2½ years later we retire variadic futures completely and make API level 6 mandatory.
With this version bump there is no longer support in Seastar for variadic type
futures such as
future<int, float>, only
Websocket server #
Websocket server support has been trickling into the tree now for quite some time since late 2021, but I’ve not yet had a chance to use it. It looks like development is active and bugs are getting fixed.
This is a nice feature and I’m looking forward to using it.
commit 2dee550bc62664894995025edde1f645be86381e Merge: f3749938 33006fdf Author: Avi Kivity <> Date: Wed Jun 22 15:58:39 2022 +0300 Merge 'Add initial support for websocket protocol' from Andrzej Stalke commit 4051d8b66a4b075e7a56ab91605e1360fcde67f8 Author: Piotr Sarna <> Date: Thu Oct 14 16:42:52 2021 +0200 websocket: add skeleton implementation of a websocket server This commit adds minimal foundations for a WebSocket server implementation for Seastar.
Faster reactor reactivity #
Some system calls are blocking so Seastar runs these on non-reactor threads. These systems calls are placed into a queue for processing. Once a system call completes the reactor may be sleeping and needs to be woken up. However, Seastar was processing all the system calls queued up before waking up the reactor, leading a sort of head of line blocking issue (in reverse) causing higher than necessary latency for system calls at the front of the queue. The change wakes up the reactor after each system call completes.
commit ecf98a507d88af8ea74716806f94f35906e85899 Author: Avi Kivity <> Date: Sun Feb 19 20:43:42 2023 +0200 reactor: syscall thread: wakeup up reactor with finer granularity
Safety checks for semaphore and gate #
Semaphores generally need to be moved, such as when data structures are being created and wired up. But once things are ready to go, semaphores really shouldn’t be moved in order to maintain a stable memory address since things like RAII unit objects hold on to a reference. So instead of marking semaphores move-only, this commit adds an opt-in runtime check that moved-from semaphores aren’t used.
There is a similar debug feature added for the Seastar gate.
commit 9b6e181e424fd18b780ade4df59e2c0c3a806742 Author: Benny Halevy <> Date: Sun Feb 5 16:10:11 2023 +0200 semaphore: disallow move after used commit 695dcd4df1113301b15d9cc8f7494294b42636f3 Author: Benny Halevy <> Date: Sat Dec 3 11:24:10 2022 +0200 gate: add gate holders debugging gate::holder keeps a naked pointer to the gate it holds. Its life time is independent of the gate and there is no mechanism to update this pointer if the gate is moved, therefore if the gate is moved or destroyed while there are outstanding holders pointing to it there is a risk for use-after-move or use-after-free.
Support shared library builds #
This is a huge quality of life improvement for big projects using Seastar. We recently enabled shared library builds for Redpanda where we have a lot of test binaries. We were able to knock off around 200 GB from the total on-disk build size.
commit f34335e5fac747a69800ecc67aa50897ce6f0f36 Author: Kefu Chai <> Date: Thu Feb 9 12:41:27 2023 +0800 build: do not always build seastar as a static library
C++20 modules support #
Well, not quite yet. But this commit puts down some of the early ground work. Follow the next stage of development here https://github.com/scylladb/seastar/pull/1605.
commit 61c63abd1c9f43f942c1958aad16753df8030231 Author: Kefu Chai <> Date: Mon Feb 6 11:26:19 2023 +0800 treewide: add non-seastar "#include"s
Additional memory reservation knob #
This looks like a reservation that reduces the amount of memory that the Seastar allocator sees on top of the reservation given to the OS for other resources. Presumably this is intended to be used when a Seastar application has a component that is bypassing the Seastar allocator. Now there is a mechanism for reserving that bucket of memory.
commit 88b85b882172fa09ae294e15316ecc1b73cfd38f Author: Wojciech Mitros <> Date: Fri Dec 2 18:44:13 2022 +0100 allocator: add an option to reserve additional memory for the OS This patch adds an option to specify the memory reserved for the OS not as a flat value, but as a modifier for the recommended amounts. The intended use case of this option is when the application is using both seastar and system allocators and it relies on the seastar defaults for the reservation of memory reserved for the OS but wants to make sure that there is enough memory not managed by seastar allocator for both the application and the rest of the system.
Re-entrant abort source request #
abort_source users can
request_abort any number of times. This change
is a nice quality of life improvement. The restriction before seems to have just
been related to an implementation detail rather than something fundamental about
the intended semantics of the interface.
commit fc59dfaa2607489a3f28d204f2670c715ccab3dc Author: Aleksandra Martyniuk <> Date: Mon Jan 16 14:43:42 2023 +0100 abort_source: allow multiple request_abort()
Dynamic adjustments to metrics and other improvements #
Nice improvement to metrics allowing them to be dynamically changed in several ways like their labels as well as enabling and disabling them. This is useful for fine-tuning to reduce the amount of data published by a system to only what is necessary. Other improvements to reducing metrics size includes histogram summarization, and metrics aggregations which allow metrics to be rolled up into per-node rather than per-shard values.
commit 364ca84d6e59253b8f4a480b57030d187747f596 Author: Amnon Heiman <> Date: Thu Dec 15 18:46:17 2022 +0200 Add an API for the metrics layer to manipulate metrics dynamically. commit 49628ceea91ce33f7660886a54e92f61574cc73b Merge: fe8fcd6d 39695cdd Author: Nadav Har'El <> Date: Mon Nov 14 14:21:54 2022 +0200 Merge 'Add Prometheus filtering capabilities by label' from Amnon Heiman commit ce78808580f549b3c39c5aa80d2ce95e2ece37a0 Merge: dc591d21 84aa236f Author: Nadav Har'El <> Date: Thu Jun 30 14:29:33 2022 +0300 Merge 'Added summaries, remove empty, and aggregation to Prometheus' from Amnon Heiman
io_uring as the default backend #
Very cool to see this change. Seems that both disk and network I/O can now take
commit eedca15f16c3b6eae3d3d8af9510624a93f5d186 Author: Kefu Chai <> Date: Wed Oct 12 18:31:54 2022 +0800 reactor: make io_uring the default backend if available commit df9c4c638d39e3500a69a279a531112c118937e8 Merge: b5ac3a63 5d8d5ae6 Author: Avi Kivity <> Date: Tue Oct 11 20:46:36 2022 +0300 Merge 'reactor: support more network ops in io_uring backend' from Kefu Chai
Support for process control #
This is Seastar’s take on
fork + exec, and it’s an interface for starting and
communicating with an external process. Looks like one of the first use cases
was to run an external program in Scylla to collect cluster topology
commit f009c4443f9885e7d289bb868e1b2fd35f25e4ae Author: Kefu Chai <> Date: Fri Nov 4 02:31:16 2022 +0800 util, core: add spawn_process() helper commit f323da3bff2a58d172d73349b7230ba652337807 Author: Kefu Chai <> Date: Thu Dec 15 22:51:32 2022 +0800 reactor: add process::terminate() and process::kill() commit 2afaa12517846a0bfb5a4aa3220180c6ecbffc04 Author: Kefu Chai <> Date: Mon Nov 21 09:51:54 2022 +0800 reactor: use make_pipe() in reactor::spawn()
New HTTP client #
This is a nice complement to the existing HTTP server available to Seastar applications. This wasn’t available when we needed an HTTP client in Redpanda and we ended up building one based on Boost Beast which was also custom tailored to our internal fragmented memory utility.
commit 67988bc874c0e20908489ac6d56c49ef0c2e443a Merge: 925b4fb5 6a0f0016 Author: Avi Kivity <> Date: Tue Nov 22 17:21:46 2022 +0200 Merge 'Add simple http client' from Pavel Emelyanov
Sharded parameter unwrapping #
Nice quality of life improvement. Commit message below speaks for itself.
commit 4ec599d1bc1b20c2ab0c3c102c7c84e11222ca36 Author: Pavel Emelyanov <> Date: Wed Aug 31 15:50:51 2022 +0300 sharded: Support sharded parameters in sharded<>::invoke_on_all() Currently it's impossible to shortcut passing one sharded service reference into another sharded service method. Like this sharded<service> srv; sharded<dependency> dep; srv.invoke_on_all(&service::fn, std::ref(dep)); where auto service::fn(dependency& dep); Instead, one needs to write a complete lambda as an argument to invoke_on_all(). However, calling srv.start(std::ref(dep)) is perfectly possible thanks to automagical sharded parameters unwrapping.
Condition variable coroutine optimizations #
Interesting optimization of condition variables for use in coroutine contexts. I think the original API will still work as expected in either case, but this could be nice for reducing some latency by avoiding memory allocations.
commit 91ed5a1b858ee12ae17872a4b27d729629f1c165 Author: Michał Chojnowski <> Date: Wed Apr 5 18:30:21 2023 +0200 condition-variable: replace the coroutine wakeup task with a promise commit 4e42a60199a26b3e784459c0ba476852e5deb371 Author: Calle Wilund <> Date: Mon Mar 7 11:41:57 2022 +0000 condition_variable: Add coroutine-only "when" operation folding waiter into parent frame Puts the waiter structure into the coroutine awaiter object located in calling coroutine frame, allowing alloc-and-extra continuation free wait.