↓Skip to main content

Ceph OSD state machine visualization

·3 mins

The Ceph OSD contains a state machine that encodes different operational aspects of the system such as peering and recovery. The state machine is built using the Boost Statechart library, and using a plugin for Clang that state machine can be programmatically extract and transformed into a graphical state machine representation. I’ve extracted the latest version of the state machine, and posted my notes on reproducing the visualization.

Original post #

A similar extraction of the state machine was done at least as far back as 2013, but I cannot find any references online to extractions on newer versions of Ceph. The image below is an SVG of the latest version of Ceph (as of writing this), so for best viewing open it up in its own browser tab (direct link dot source).

Ceph OSD state machine

Reproducing #

It was a bit of a challenge to get the tool to work. Here are the steps I took.

First build the Clang plugin. I used the version from https://github.com/Larsjep/boost-statechart-viewer. Below I assume that the plugin is cloned and built at /tmp/boost-statechart-viewer. After installing LLVM and Clang development packages, the plugin built fine by running make.

Next you’ll need to prepare the Ceph tree:

  1. Initialize the Ceph source tree by running cmake .
  2. I ran make just long enough for Boost to get built and setup
  3. Generate the tracing dependencies cd src/tracing; make
  4. Apply the small diff (shown below at end of this post)

Next create a file called src/osd/statechart.cc that contains the following amalgamation of source files that contain state chart bits (found with git grep -l statechart):

#include "PG.h"
#include "PrimaryLogPG.h"
#include "PGPeeringEvent.h"
#include "PG.cc"
#include "PrimaryLogPG.cc"
#include "OSD.cc"

Then run the following command in src/osd:

clang++ -I.. -I../../include -I../dmclock/support/src -std=c++1z \
   -Wno-inconsistent-missing-override -Xclang -load -Xclang \
   /tmp/boost-statechart-viewer/src/visualizer.so -Xclang -plugin -Xclang \
   visualize-statechart -c -o statechart.o statechart.cc

There were a couple warnings and errors, but the tool still produced the graph source file statechart.dot which you can now transform into any graphical form you want (e.g. dot -Tsvg statechart.dot > statechart.svg).

diff --git a/src/osd/PG.cc b/src/osd/PG.cc
index 649f7177d2..03ae285587 100644
--- a/src/osd/PG.cc
+++ b/src/osd/PG.cc
@@ -87,7 +87,8 @@ const string fastinfo_key("_fastinfo");
 template <class T>
 static ostream& _prefix(std::ostream *_dout, T *t)
 {
-  return *_dout << t->gen_prefix();
+  return *_dout;
+  //return *_dout << t->gen_prefix();
 }
 
 MEMPOOL_DEFINE_OBJECT_FACTORY(PGPeeringEvent, pg_peering_evt, osd);
diff --git a/src/osd/PrimaryLogPG.cc b/src/osd/PrimaryLogPG.cc
index dc194e0372..bf3e8cfbd3 100644
--- a/src/osd/PrimaryLogPG.cc
+++ b/src/osd/PrimaryLogPG.cc
@@ -63,10 +63,10 @@
 #define DOUT_PREFIX_ARGS this, osd->whoami, get_osdmap()
 #undef dout_prefix
 #define dout_prefix _prefix(_dout, this)
-template <typename T>
-static ostream& _prefix(std::ostream *_dout, T *pg) {
-  return *_dout << pg->gen_prefix();
-}
+//template <typename T>
+//static ostream& _prefix(std::ostream *_dout, T *pg) {
+//  return *_dout << pg->gen_prefix();
+//}

That’s it. If anyone knows of tools that can be used to produce visualizations that are easier to navigate in the browser, I’d love to hear about them.