Seastar's deleter utility

In this post we are going to discuss a Seastar utility called seastar::deleter that is used for managing the lifetime of data. In many ways a deleter can be thought of as a unified approach to RAII for both raw memory pointers and objects, but it also adds other goodies like reference counting and deleter chaining.

It’s important to note that the deleter is a low-level utility that is generally not used directly by applications. Rather a deleter is typically used to build higher-level interfaces, like Seastar’s memory workhorse tool temporary_buffer<T>, and custom utilities built in support of application-specific scenarios.

In its simplest form the deleter acts as an RAII tool for raw memory. In the following example the allocated memory pointed to by p will be freed in the destructor of the deleter returned from make_free_deleter when d goes out of scope.

namespace ss = seastar;

{
  void *p = malloc(1024);
  auto d = ss::make_free_deleter(p);
}

A deleter can also manage object instances. In the example below the deleter that is returned by the make_object_deleter factory contains a moved-to instance of my_object. When the deleter goes out of scope both objects will be destroyed through normal calls to object destructors.

{
  MyObject my_object;
  auto d = ss::make_object_deleter(std::move(my_object));
}

That’s not particularly exciting, but the deleter has some interesting features, and having a unified type for managing the lifetime of data turns out to be useful when passing around ownership in a Seastar application. Let’s take a closer look at how the deleter works, and then look at some of the deleter’s features.

Deleter internals #

The factory function make_free_deleter builds a deleter object for managing a raw pointer, and passes the special deleter::raw_object_tag to the constructor to select the appropriate constructor overload.

deleter make_free_deleter(void* p) {
  return deleter(deleter::raw_object_tag(), p);
}

This translates into a call to the deleter’s constructor shown below that stores the result of from_raw_object(p) into the private _impl* member. Spoiler alert: the deleter effectively implements the pimpl idiom, but contains an optimization for raw pointers that avoids allocating an actual impl object. Part of that magic is contained in from_raw_object that we’ll look at next.

class deleter {
public:
  struct impl;
  struct raw_object_tag {};

private:
  impl* _impl = nullptr;

public:
  deleter(raw_object_tag tag, void* p)
    : _impl(from_raw_object(p)) {}

  ~deleter();

The call to from_raw_object is shown below. It returns the pointer it is passed but with a bit set to indicate that it is a pointer to raw memory. This is possible because modern hardware does not use all 64-bits to address memory.

impl* from_raw_object(void* p) {
   auto x = reinterpret_cast<uintptr_t>(p);
   return reinterpret_cast<impl*>(x | 1);
}

Consider what we have seen so far. We have an impl pointer initialized from a raw pointer to a region of memory (presumably allocated with malloc) and that pointer has an unused bit set to indicate this exact scenario. When reclaiming memory resources, this bit will determine the method of reclamation.

As we see below in the deleter’s destructor, if it is a raw pointer then std::free is used to clean up. The is_raw_object() method tests for that special bit, and to_raw_object returns the pointer with the bit cleared to give us back the original pointer that can be safely passed to std::free.

deleter::~deleter() {
  if (is_raw_object()) {
    std::free(to_raw_object());
    return;
  }
  if (_impl && --_impl->refs == 0) {
    delete _impl;
  }
}

But what is happening there in the destructor when _impl is not a raw pointer? In this case the _impl member is treated as a pointer to an instance of a deleter::impl and destroyed if its reference count drops to zero. If you’re thinking that the deleter doesn’t handle reference counting for raw memory, then be patient. We’ll get to that later.

The deleter::impl class has the definition shown below. Notice the embedded reference counter. The next member is used for deleter chaining, which we’ll talk about at the end of this article.

struct deleter::impl {
  unsigned refs = 1;
  deleter next;
  impl(deleter next) : next(std::move(next)) {}
  virtual ~impl() {}
};

Here is one deleter::impl specialization for holding an object instance. The call site moves the object into the object_deleter_impl.

template <typename Object>
struct object_deleter_impl final : deleter::impl {
  Object obj;
  object_deleter_impl(deleter next, Object&& obj)
    : impl(std::move(next)), obj(std::move(obj)) {}

template <typename T>
deleter make_object_deleter(T&& obj) {
  return deleter{
    new object_deleter_impl<Object>(deleter(), std::move(obj));
}

Notice below in the deleter constructor that despite passing in a pointer, the deleter does not treat this as a raw pointer since the constructor that takes an impl* does not accept the deleter::raw_object_tag parameter. Therefore, it won’t have the raw memory bit set on its pointer.

class deleter {
  explicit deleter(impl* i) : _impl(i) {}
...
};

Putting all this together, when the destructor runs and the reference count eventually drops to zero, the managed object embedded in the object_deleter_impl instance will be destroyed.

Why all this complexity? One reason is because it makes the case of managing raw memory very fast, requiring no additional allocations. But that’s really not enough reason to justify the deleter. The second reason is that it makes for a unified interface: a deleter instance managing a raw pointer can upgrade into a heavier weight object with extra features like reference counting and chaining only when needed.

Deleter sharing #

The copy constructor and assignment operator are disabled on the deleter type. Instead, a deleter may be explicitly shared using the deleter::share method which returns a new deleter instance that shares ownership of the underlying resource with the original deleter. Once a shared instance is created, both deleter instances must be destroyed before the managed resource is released.

We saw above how a deleter managing an object uses a reference count to control when a managed object is destroyed, but how does it work for a raw pointer? Here is the implementation of share which first checks if a raw pointer is being managed, and if so, replaces the internal _impl* member with a specialized deleter::impl instance that wraps the raw pointer. In effect, it upgrades the internal representation.

deleter deleter::share() {
  if (!_impl) {
    return deleter();
  }
  if (is_raw_object()) {
    _impl = new free_deleter_impl(to_raw_object());
  }
  ++_impl->refs;
  return deleter(_impl);
}

struct free_deleter_impl final : deleter::impl {
  void* obj;
  free_deleter_impl(void* obj) : impl(deleter()), obj(obj) {}
  virtual ~free_deleter_impl() override { std::free(obj); }
};

With a deleter::impl instance now in place, share increments the reference count and returns a new deleter that shares the same implementation. Unsurprisingly, free_deleter_impl calls std::free on the pointer it holds in its destructor.

You may have noticed that the reference counter is a non-atomic integer, and there is not a lock in sight to protect it. Like we saw in the discussion about Seastar smart pointers, a deleter must also not be shared across cores, otherwise that shared state is at risk of being corrupt, leading to memory leaks or accessing freed memory.

Deleter chaining #

Another feature of the deleter is that it can be used to create dependent chains of deleters. Recall the next member of the deleter::impl base class:

struct deleter::impl {
  unsigned refs = 1;
  deleter next;
  impl(deleter next) : next(std::move(next)) {}
  virtual ~impl() {}
};

When an instance of deleter::impl is destroyed, so too is next. Thus, it’s merely a matter of setting next in order to create deleter dependencies. Here is an example of creating a deleter chain. It’s just like we saw above, but uses a different make_object_deleter interface that accepts the next deleter, extending the chain. By building up a chain of deleters, complex clean-up processes can be conveniently represented as a single deleter.

template <typename T>
deleter make_object_deleter(deleter d, T&& obj) {
  return deleter{
    new object_deleter_impl<T>(std::move(d), std::move(obj))};
}

Building a chain doesn’t have to occur only at the time a deleter is created. If you happen to already have some deleter instances managing resources, you can use the deleter::append method to build a chain with them.