↓Skip to main content

Adding a new placement group operation in Ceph

·10 mins

In this post we are going to create a librados operation in Ceph that operates at the level of the placement group (most RADOS operations act upon objects). As a demonstration we’ll build an interface that computes the checksum of all object data in a placement group. This probably isn’t useful to anyone, but it exercises a lot of interesting internal machinery. The overall approach is adapted from the code paths used to list objects in a pool. We will simplify the problem by ignoring various error scenarios related to pool state changing during the operation.

The intended usage of the new interface is similar to:

int pg = 0;
for (;;) {
  int checksum;
  int ret = ioctx.pg_checksum(pg, &checksum);
  if (ret < 0) // -ERANGE: no more pgs
    break;
  std::cout << "pg " << pg << ": " << checksum << std::endl;
  pg++;
}

Build the operation #

There is some basic scaffolding that needs to be added on the client side to expose an operation through the librados interface and it includes editing librados.hpp and adding hooks into the IoCtx implementation. Here are the basics:

Add a public interface to librados:

diff --git a/src/include/rados/librados.hpp b/src/include/rados/librados.hpp
index 640e7cf..ea25fca 100644
--- a/src/include/rados/librados.hpp
+++ b/src/include/rados/librados.hpp
@@ -857,6 +857,8 @@ namespace librados
         ObjectCursor *split_start,
         ObjectCursor *split_finish);
 
+    int pg_checksum(int pg, int *checksum);

diff --git a/src/librados/librados.cc b/src/librados/librados.cc
index 55f0a5a..6d2a6f3 100644
--- a/src/librados/librados.cc
+++ b/src/librados/librados.cc
@@ -1773,6 +1773,11 @@ const librados::ObjectIterator& librados::IoCtx::objects_end() const
   return ObjectIterator::__EndObjectIterator;
 }
 
+int librados::IoCtx::pg_checksum(int pg, int *checksum)
+{
+  return io_ctx_impl->pg_checksum(pg, checksum);
+}

The public interface punches through to the implementation:

diff --git a/src/librados/IoCtxImpl.h b/src/librados/IoCtxImpl.h
index be77d8e..e58a57e 100644
--- a/src/librados/IoCtxImpl.h
+++ b/src/librados/IoCtxImpl.h
@@ -222,6 +222,8 @@ struct librados::IoCtxImpl {
   int pool_change_auid(unsigned long long auid);
   int pool_change_auid_async(unsigned long long auid, PoolAsyncCompletionImpl *c);
 
+  int pg_checksum(int pg, int *checksum);

diff --git a/src/librados/IoCtxImpl.cc b/src/librados/IoCtxImpl.cc
index 75f4dee..5f72f0b 100644
--- a/src/librados/IoCtxImpl.cc
+++ b/src/librados/IoCtxImpl.cc
@@ -1098,6 +1098,11 @@ int librados::IoCtxImpl::aio_cancel(AioCompletionImpl *c)
   return objecter->op_cancel(c->tid, -ECANCELED);
 }
 
+int librados::IoCtxImpl::pg_checksum(int pg, int *checksum)
+{
+  std::cout << "call objecter here..." << std::endl << std::flush;
+  return 0;
+}

This last method librados::IoCtxImpl::pg_checksum is where we will add more interesting bits. The Objecter talks to the OSDs and the IoCtx contains the context information for the client connection to a pool. We’ll yank out the necessary information from the IoCtx (like the target pool) and pass it to the Objecter where most of the work will happen.

The new version of librados::IoCtxImpl::pg_checksum passes a condition variable into Objecter::pg_checksum (which we’ll show next) along with any state information, and then waits for the condition variable to signal that the work is complete.

diff --git a/src/librados/IoCtxImpl.cc b/src/librados/IoCtxImpl.cc
index 75f4dee..7228bb6 100644
--- a/src/librados/IoCtxImpl.cc
+++ b/src/librados/IoCtxImpl.cc
@@ -1098,6 +1098,23 @@ int librados::IoCtxImpl::aio_cancel(AioCompletionImpl *c)
   return objecter->op_cancel(c->tid, -ECANCELED);
 }
 
+int librados::IoCtxImpl::pg_checksum(int pg, int *checksum)
+{
+  Cond cond;
+  bool done;
+  int r = 0;
+  Mutex mylock("IoCtxImpl::pg_checksums::mylock");
+
+  objecter->pg_checksum(poolid, snap_seq, oloc.nspace,
+      pg, checksum, new C_SafeCond(&mylock, &cond, &done, &r));
+
+  mylock.Lock();
+  while (!done)
+    cond.Wait(mylock);
+  mylock.Unlock();
+
+  return r;
+}

Before we build the operation to send off to the OSD we should make sure that the target placement group is valid. We do this by querying the OSDMap and comparing the number of placement groups to the target placement group specified by the client:

diff --git a/src/osdc/Objecter.cc b/src/osdc/Objecter.cc
index 8560a26..c0ac883 100644
--- a/src/osdc/Objecter.cc
+++ b/src/osdc/Objecter.cc
@@ -3427,6 +3427,34 @@ uint32_t Objecter::list_nobjects_seek(NListContext *list_context,
   return list_context->current_pg;
 }
 
+void Objecter::pg_checksum(int64_t poolid, int pool_snap_seq, string nspace,
+    int pg, int *checksum, Context *onfinish)
+{
+  // get pool state
+  shared_lock rl(rwlock);
+  const pg_pool_t *pool = osdmap->get_pg_pool(poolid);
+  if (!pool) {
+    rl.unlock();
+    onfinish->complete(-ENOENT);
+    return;
+  }
+  int pg_num = pool->get_pg_num();
+  rl.unlock();
+
+  // sanity check requested pg
+  if (pg >= pg_num) {
+    onfinish->complete(-ERANGE);
+    return;
+  }

Now we construct the operation. The call to ObjectOperation::pg_checksum builds a context for the operation. The C_PGChecksum object is a callback object responsible for handling the response. Finally, Objecter::pg_read is used to submit the operation. It returns immediately, and the calling client continues to block on the condition variable in IoCtxImpl::pg_checksum (see above). The condition is signaled by C_PGChecksum after the operation reply arrives from the target OSD.

+  // build and submit op
+  ObjectOperation op;
+  op.pg_checksum();

+  C_PGChecksum *onack = new C_PGChecksum(onfinish, checksum);

+  object_locator_t oloc(poolid, nspace);
+  pg_read(pg, oloc, op, &onack->bl, 0, onack, &onack->epoch, NULL);
+}
+
 void Objecter::list_nobjects(NListContext *list_context, Context *onfinish)
 {
   ldout(cct, 10) << "list_objects" << dendl;

Here is ObjectOperation::pg_checksum. It adds a new operation and tags it with the ID of the new checksum operation.

diff --git a/src/osdc/Objecter.h b/src/osdc/Objecter.h
index 984fb89..98748fa 100644
--- a/src/osdc/Objecter.h
+++ b/src/osdc/Objecter.h
@@ -215,6 +215,12 @@ struct ObjectOperation {
     flags |= CEPH_OSD_FLAG_PGOP;
   }
 
+  void pg_checksum() {
+    OSDOp& osd_op = add_op(CEPH_OSD_OP_PG_CHECKSUM);
+    (void)osd_op;
+    flags |= CEPH_OSD_FLAG_PGOP;
+  }
+
   void scrub_ls(const librados::object_id_t& start_after,
                uint64_t max_to_get,

This is the callback object for handling the OSD reply. The finish method is called automatically, and if there is no error the return payload is decoded. The final_finish context refers to the context that the calling client is waiting on (back in IoCtxImpl::pg_checksum), and checksum_out is a pointer to integer where the client wants the result stored.

@@ -1427,6 +1433,32 @@ public:
     }
   };
 
+  struct C_PGChecksum : public Context {
+    Context *final_finish;
+    epoch_t epoch;
+    int *checksum_out;
+    bufferlist bl;
+
+    C_PGChecksum(Context *finish, int *checksum) :
+      final_finish(finish), epoch(0), checksum_out(checksum)
+    {}
+
+    void finish(int r) {
+      if (r >= 0) {
+        try {
+          int checksum;
+          bufferlist::iterator it = bl.begin();
+          ::decode(checksum, it);
+          *checksum_out = checksum;
+          r = 0;
+        } catch (buffer::error& e) {
+          r = -EIO;
+        }
+      }
+      final_finish->complete(r);
+    }
+  };
+
   struct C_NList : public Context {
     NListContext *list_context;
     Context *final_finish;

To complete the story on the client side, here is where CEPH_OSD_OP_PG_CHECKSUM is defined:

diff --git a/src/include/rados.h b/src/include/rados.h
index 71d5f27..8c5c3b4 100644
--- a/src/include/rados.h
+++ b/src/include/rados.h
@@ -299,7 +299,8 @@ extern const char *ceph_osd_state_name(int s);
        f(PG_HITSET_GET, __CEPH_OSD_OP(RD, PG, 4),      "pg-hitset-get")    \
        f(PGNLS,        __CEPH_OSD_OP(RD, PG, 5),       "pgnls")            \
        f(PGNLS_FILTER, __CEPH_OSD_OP(RD, PG, 6),       "pgnls-filter")     \
-       f(SCRUBLS, __CEPH_OSD_OP(RD, PG, 7), "scrubls")
+       f(SCRUBLS, __CEPH_OSD_OP(RD, PG, 7), "scrubls")                 \
+       f(PG_CHECKSUM,  __CEPH_OSD_OP(RD, PG, 8),       "pg-checksum")     \
 
 enum {
 #define GENERATE_ENUM_ENTRY(op, opcode, str)   CEPH_OSD_OP_##op = (opcode),

If a client calls IoCtx::pg_checksum at this point it will receive an error from the OSD indicating that an unknown operation has been submitted. Next we’ll fix that.

Handling the operation in an OSD #

Before we dive into the weeds, let’s return a fixed value to make sure everything is wired up correctly. Below we add a handler for the checksum operation and return a fixed value of 543. If everything is working correctly your client will receive this value when calling IoCtx::pg_checksum on any valid placement group.

diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc
index cce8898..496a7bc 100644
--- a/src/osd/ReplicatedPG.cc
+++ b/src/osd/ReplicatedPG.cc
@@ -1470,6 +1470,14 @@ void ReplicatedPG::do_pg_op(OpRequestRef op)
       result = do_scrub_ls(m, &osd_op);
       break;
 
+   case CEPH_OSD_OP_PG_CHECKSUM:
+      {
+        int checksum = 543;
+        ::encode(checksum, osd_op.outdata);
+        result = 0;
+      }
+      break;
+
     default:
       result = -EINVAL;
       break;

Now let’s get to work implementing the server-side handler for computing the checksum. The first step is to enumerate the set of objects in the placement group. This is accomplished using the PGBackend::objects_list_partial method. This method takes an object as an enumeration starting point, and returns some number of objects that follow the starting point. The checksum integer will be the result of the operation, and osr->flush(); blocks until any queued transactions have completed.

Note that 2nd and 3rd arguments to PGBackend::objects_list_partial specify the min and max number of objects to return. In practice these should be larger than 1, but we keep them this way to exercise the iteration logic.

diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc
index 496a7bc..d093f67 100644
--- a/src/osd/ReplicatedPG.cc
+++ b/src/osd/ReplicatedPG.cc
@@ -1472,9 +1472,112 @@ void ReplicatedPG::do_pg_op(OpRequestRef op)
 
    case CEPH_OSD_OP_PG_CHECKSUM:
       {
+        int checksum = 0;
+
+        osr->flush();
+        hobject_t current = collection_list_handle_t();
+        for (;;) {
+          hobject_t next;
+          vector<hobject_t> sentries;
+          int r = pgbackend->objects_list_partial(
+              current, 1, 1, &sentries, &next);
+          if (r) {
+            result = -EINVAL;
+            break;
+          }

The way I interpret this next assertion (taken from the standard librados code path for enumerating objects) is that if we are enumerating objects within a snapshot (i.e. read-only) context, then there should be no missing objects. I believe missing objects can show up when the number of placement groups are changed, when there is a failure, and probably some other situations. This topic deserves more research, but for now we can act as though the union of the missing objects and the set of objects returned by PGBackend::objects_list_partial are the set of objects in the placement group.

+          assert(snapid == CEPH_NOSNAP ||
+              pg_log.get_missing().get_items().empty());
+
+          pg_log.resort_missing(get_sort_bitwise());

Here we obtain iterators to the list of objects from PGBackend and the missing set.

+          map<hobject_t, pg_missing_item, hobject_t::ComparatorWithDefault>::const_iterator missing_iter =
+            pg_log.get_missing().get_items().lower_bound(current);
+
+          vector<hobject_t>::iterator ls_iter = sentries.begin();
+          hobject_t _max = hobject_t::get_max();
+

If no objects were returned from the backend and there are no missing objects then we are done.

+
+          if (sentries.empty() && missing_iter == pg_log.get_missing().get_items().end()) {
+            result = 0;
+            break;
+          }
+

This loop processes one object at a time. The first step is to combine the set of existing objects with the set of missing objects. The process here is effectively the last part of a merge sort. The resulting variable candidate holds a hobject_t from one of the two sets.

+          for (;;) {
+            const hobject_t &mcand =
+              missing_iter == pg_log.get_missing().get_items().end() ?
+              _max :
+              missing_iter->first;
+
+            const hobject_t &lcand =
+              ls_iter == sentries.end() ?
+              _max :
+              *ls_iter;
+
+            hobject_t candidate;
+            if (mcand == lcand) {
+              candidate = mcand;
+              if (!mcand.is_max()) {
+                ++ls_iter;
+                ++missing_iter;
+              }
+            } else if (cmp(mcand, lcand, get_sort_bitwise()) < 0) {
+              candidate = mcand;
+              assert(!mcand.is_max());
+              ++missing_iter;
+            } else {
+              candidate = lcand;
+              assert(!lcand.is_max());
+              ++ls_iter;
+            }

The comparison here ends iteration if we go past the end point returned from PGBackend::objects_list_partial, which is probably possible because an arbitrary set of missing objects could sort into the same range.

+
+            if (cmp(candidate, next, get_sort_bitwise()) >= 0) {
+              break;
+            }
+

Objects internal to RADOS like snapdir objects and objects used for metadata are skipped as they are not user-visible.

+            // skip snapdir objects
+            if (candidate.snap == CEPH_SNAPDIR)
+              continue;
+
+            if (candidate.snap < snapid)
+              continue;
+
+            assert(snapid == CEPH_NOSNAP);
+
+            // skip internal namespace
+            if (candidate.get_namespace() == cct->_conf->osd_hit_set_namespace)
+              continue;
+
+            // skip wrong namespace
+            if (m->get_object_locator().nspace != librados::all_nspaces &&
+                candidate.get_namespace() != m->get_object_locator().nspace)
+              continue;
+

Finally we read the object data by calling PGBackend::objects_read_sync.

+            bufferlist bl;
+            int ret = pgbackend->objects_read_sync(
+                candidate, 0, 0, 0, &bl);
+            if (ret < 0) {
+              derr << "checksum read error " << ret << dendl;
+              result = ret;
+              break;
+            }
+

And the checksum is updated by adding together all of the object bytes cast to a numeric value (this is just for testing…). Iteration resumes with current = next, and after iteration is complete the checksum is encoded and returned to the client.

+            // update checksum
+            const size_t length = bl.length();
+            const uint8_t *data = (uint8_t*)bl.c_str();
+            for (size_t offset = 0; offset < length; offset++)
+              checksum += data[offset];
+          }
+          current = next;
+        }
+        ::encode(checksum, osd_op.outdata);
+      }
+      break;

And here is a sample output:

nwatkins@pl1:~/ceph/build$ bin/checksum-test
pg 0 ret 0 cs 126195159
pg 1 ret 0 cs 378485417
pg 2 ret 0 cs 378442111
pg 3 ret 0 cs 252377238
pg 4 ret 0 cs 378579803
pg 5 ret 0 cs 126198934
pg 6 ret 0 cs 126232920
pg 7 ret 0 cs 378636258
↑