↓Skip to main content

Load Lua RADOS classes from local file system

·19 mins

Object classes in RADOS written in Lua have up until now been limited to scripts that are embedded into every client request. This post describes how we have extended RADOS to load Lua scripts from the local file system, supporting a new way to manage object interfaces written in Lua.

Introduction #

The RADOS object store that powers Ceph supports an active storage-like feature called object classes that allow custom object interfaces to be defined. Object classes define a set of methods, each of which can perform arbitrary mutations on objects within a transactional context. The native object class interface is C++, and deploying object classes that are not in the upstream Ceph release means compiling and managing binary libraries and ensuring version compatibility. Alternatively, developers can define object classes in the embedded language Lua allowing for object classes to be defined on-the-fly. However in the current design every client request must be augmented with the Lua script to be executed. In this post we will describe a new form of interface management that allows object classes written in Lua to be loaded from the file system, allowing Lua scripts to effectively be installed into the system.

The remainder of this post will examine how existing object class modules are loaded into a running OSD, how Lua object classes are defined by attaching a Lua script to each request, and finally how we implement Lua class loading from the file system.

Object Class Loader #

I’ll now briefly describe how object classes are currently managed and loaded by Ceph. We’ll adopt a similar model for Lua scripts. The following is a directory listing of the object classes that are in the current upstream Ceph tree. Each directory corresponds to a separate class:

[nwatkins@kyoto ceph]$ ls -l src/cls/
total 84
drwxrwxr-x. 4 nwatkins nwatkins 4096 Dec  5 12:05 cephfs
drwxrwxr-x. 4 nwatkins nwatkins 4096 Dec  6 09:21 journal
drwxrwxr-x. 4 nwatkins nwatkins 4096 Dec  3 12:09 lock
drwxrwxr-x. 4 nwatkins nwatkins 4096 Dec  3 12:07 log
-rw-rw-r--. 1 nwatkins nwatkins 4265 Dec  5 10:40 Makefile-server.am
drwxrwxr-x. 4 nwatkins nwatkins 4096 Dec  3 17:17 numops
...

Each class is compiled into a shared library that can be loaded into the OSD process at runtime. The following is a directory listing of the installation path for each object class:

[nwatkins@kyoto ceph]$ ls -l /usr/lib/rados-classes/*.so | grep cls
-rwxr-xr-x. 1 root root   241360 Dec  5 10:45 /usr/lib/rados-classes/libcls_cephfs.so
-rwxr-xr-x. 1 root root   727840 Dec  5 10:45 /usr/lib/rados-classes/libcls_journal.so
-rwxr-xr-x. 1 root root  1176504 Dec  5 10:45 /usr/lib/rados-classes/libcls_kvs.so
-rwxr-xr-x. 1 root root   765960 Dec  5 10:45 /usr/lib/rados-classes/libcls_lock.so
-rwxr-xr-x. 1 root root   482920 Dec  5 10:45 /usr/lib/rados-classes/libcls_log.so
...

The naming convention of the libraries is important. The OSD assumes a naming convention in which an object class named foo can be loaded from the file libcls_foo.so. Concretely, when a RADOS client runs the command:

IoCtx::exec("oid", "foo", "foo-method", ...)

The OSD will check if the foo class has already been loaded, and if not, it will attempt to locate a shared library named libcls_foo.so and use dlopen to dynamically insert the library into the running process. Once the object class module has been loaded methods exported by the module may be invoked on behalf of the client (e.g.foo-method in the above example).

Loading Lua Classes #

The Lua object class feature is not a native OSD feature. Rather, it is implemented as a statically loaded object class written in C++ that is used as a runtime host for dynamically defined object interfaces. It exists alongside other object classes:

[nwatkins@kyoto ceph]$ ls -l src/cls/lua
total 1996
-rw-rw-r--. 1 nwatkins nwatkins   20472 Dec  4 15:08 cls_lua.cc
-rw-rw-r--. 1 nwatkins nwatkins    1823 Dec  4 14:59 cls_lua_client.cc
-rw-rw-r--. 1 nwatkins nwatkins     412 Dec  4 14:59 cls_lua_client.hpp
-rw-rw-r--. 1 nwatkins nwatkins     354 Dec  4 14:59 cls_lua.h
-rw-rw-r--. 1 nwatkins nwatkins    3739 Dec  4 14:59 lua_bufferlist.cc
-rw-rw-r--. 1 nwatkins nwatkins   22899 Dec  4 14:59 lua_cmsgpack.c

And is compiled and loaded like any other object class:

[nwatkins@kyoto ceph]$ ls -l /usr/lib/rados-classes/*.so | grep cls_lua
-rwxr-xr-x. 1 root root   241360 Dec  5 10:45 /usr/lib/rados-classes/libcls_lua.so

The lua object class is invoked like any other object class:

IoCtx::exec("oid", "lua", "eval_json", input, output);

What is happening here? The eval_json method is being called on the lua class which specifies that the input should be assumed to be a JSON encoded string (there are other encoding available such as bufferlist and `msgpack). The JSON string is simply the Lua script, actual method input, and the method in the Lua script to invoke. Here is an example of the input (using Python dictionary notation). The Lua script shown below treats the input as a string and returns the string with each character capitalized:

cmd = {
  "script": """
      function upper(input, output)
        input_str = input:str()
        upper_str = string.upper(input_str)
        output:append(upper_str)
      end
      cls.register(upper)
  """,
  "handler": "upper",
  "input": "this string was in lower case",
}

The encoded input can be provided to IoCtx::exec:

ret, data = ioctx.execute('oid', 'lua', 'eval_json', json.dumps(cmd))
print data[:ret]

And when run, we see the output that we expect:

[nwatkins@kyoto src]$ python test.py
THIS STRING WAS IN LOWER CASE

Rather than using the lua class directly, clients typically use a thin wrapper around IoCtx::exec that performs input encoding, providing the appearance that RADOS provides access to dynamically defined object classes. There are a couple drawbacks to this approach:

  1. Clients must send a script along with each request
  2. The vanilla exec interface cannot invoke dynamic classes directly

Both of these issues are limiting. Providing a script with each request can be expensive and may introduce a major hassle for applications that must manage a set of dynamic interfaces. The second issue limits the power of dynamic interfaces. Existing applications that are written assuming a given a object class would need to be recompiled to switch to Lua defined classes because they would need use the lua class. However, it would be nice to be able to simply swap out the implementation and allow the application to continue referencing the same object class methods.

We address both of these issues in the remainder of this post. To address the first issue we introduce the ability for Lua scripts to be loaded from a local file system in a manner analogous to how native C++ modules are managed. The second issue is effectively solved for free when object classes are allowed to be resolved to separate Lua scripts.

The Goal #

From a management perspective the goal of this project is to allow object classes written in Lua to be installed to and loaded from the file system with identical semantics to the existing infrastructure for loading native C++ object class modules. In the same way that object classes written in C++ can be compiled and installed manually, in this post we only consider Lua scripts managed in the Ceph tree (but out-of-tree management is much easier with Lua scripts as they do not require compilation).

We stash the Lua scripts in the same directory structure as the native extensions. Below are three object classes, one native and two written in Lua. The native object class hello is mirrored by the Lua object class named hellolua which will be used as a demonstration class. The compress class is in its own directory and is a Lua class that contains data compression methods.

[nwatkins@kyoto ceph]$ ls -l src/cls/hello/
total 480
-rw-rw-r--. 1 nwatkins nwatkins  10328 Dec  3 11:10 cls_hello.cc
-rw-rw-r--. 1 nwatkins nwatkins    289 Dec  3 12:24 cls_hello.lo
-rw-rw-r--. 1 nwatkins nwatkins    230 Dec  4 13:57 cls_hellolua.lua
-rw-rw-r--. 1 nwatkins nwatkins 468864 Dec  3 12:24 cls_hello.o
[nwatkins@kyoto ceph]$ ls -l src/cls/compress/
total 0
-rw-rw-r--. 1 nwatkins nwatkins 0 Dec  6 11:46 cls_compress.lua

Lua classes are installed like their native counterparts, although Lua scripts are installed in /usr/share/ceph/rados-classes as opposed to /usr/lib/rados-classes used for architecture-dependent binaries. The naming convention for Lua scripts is used for resolving class names to files, and native classes with the same name take precedence. Like native classes, Lua scripts are only loaded once. Changes to any Lua scripts will be reflected after an OSD restart. Note that this restriction isn’t fundamental and could be easily removed. Shipping a Lua script with each request still retains its dynamic nature, too.

Implementation #

First we’ll cover basic changes needed to install the Lua scripts. First some automake magic that will install the Lua scripts into the datadir (typically /usr/share/):

diff --git a/src/Makefile-env.am b/src/Makefile-env.am
index 3d8a252..0e265d8 100644
--- a/src/Makefile-env.am
+++ b/src/Makefile-env.am
@@ -20,6 +20,7 @@ lib_LTLIBRARIES =
 noinst_LTLIBRARIES = 
 noinst_LIBRARIES =
 radoslib_LTLIBRARIES =
+dist_radoslua_DATA =
 
 # like bin_PROGRAMS, but these targets are only built for debug builds
 bin_DEBUGPROGRAMS =
@@ -286,4 +287,5 @@ DENCODER_DEPS =
 
 
 radoslibdir = $(libdir)/rados-classes
+radosluadir = $(datadir)/ceph/rados-classes
 
diff --git a/src/cls/Makefile-server.am b/src/cls/Makefile-server.am
index 2221d6c..5b95f84 100644
--- a/src/cls/Makefile-server.am
+++ b/src/cls/Makefile-server.am
@@ -92,4 +92,7 @@ libcls_lua_la_LDFLAGS = ${AM_LDFLAGS} -version-info 1:0:0 -export-symbols-regex
 radoslib_LTLIBRARIES += libcls_lua.la
 endif
 
+dist_radoslua_DATA += \
+    cls/hello/cls_hellolua.lua
+
 endif # WITH_OSD

Similar changes need to be made for the CMake build system, Debian, and RPM packaging but these are omitted from this post. Next we need to be able to refer to the directory containing the Lua scripts so we can load them dynamically at runtime. For this we define a configuration option populated by the automake variable specifying the installation location:

diff --git a/src/Makefile-env.am b/src/Makefile-env.am
index 0e265d8..6672d37 100644
--- a/src/Makefile-env.am
+++ b/src/Makefile-env.am
@@ -87,6 +87,7 @@ AM_COMMON_CPPFLAGS = \
        -D_GNU_SOURCE \
        -DCEPH_LIBDIR=\"${libdir}\" \
        -DCEPH_PKGLIBDIR=\"${pkglibdir}\" \
+       -DCEPH_DATADIR=\"${datadir}\" \
        -DGTEST_USE_OWN_TR1_TUPLE=0
 
 if LINUX
diff --git a/src/common/config_opts.h b/src/common/config_opts.h
index a19cc5d..86eb42f 100644
--- a/src/common/config_opts.h
+++ b/src/common/config_opts.h
@@ -695,6 +695,7 @@ OPTION(osd_deep_scrub_stride, OPT_INT, 524288)
 OPTION(osd_deep_scrub_update_digest_min_age, OPT_INT, 2*60*60)   // objects must be this old (seconds) before we update the whole-object digest on scrub
 OPTION(osd_scan_list_ping_tp_interval, OPT_U64, 100)
 OPTION(osd_class_dir, OPT_STR, CEPH_LIBDIR "/rados-classes") // where rados plugins are stored
+OPTION(osd_lua_class_dir, OPT_STR, CEPH_DATADIR "/ceph/rados-classes") // where rados lua plugins are stored
 OPTION(osd_open_classes_on_start, OPT_BOOL, true)
 OPTION(osd_check_for_log_corruption, OPT_BOOL, false)
 OPTION(osd_use_stale_snap, OPT_BOOL, false)

Loading Lua Scripts From File System #

We’ll describe the implementation by tracing the execution of a IoCtx::exec operation and describing the changes required along that execution path. The first important stop is when the OSD performs pre-processing on the operation which obtains a reference to the object class requested, and optionally loads the class if it hasn’t already been loaded:

ClassHandler::ClassData *cls;
int r = class_handler->open_class(cname, &cls);
if (r) {
  derr << "class " << cname << " open got " << cpp_strerror(r) << dendl;
  if (r == -ENOENT)
    r = -EOPNOTSUPP;
  else
    r = -EIO;
  return r;
}

Class Setup #

The open_class method performs a lookup on the class name which will implicitly create a reference to a class with the name, but may not yet load it. Initially a class is not in the CLASS_OPEN state, so the call to open_class will attempt to load the class.

int ClassHandler::open_class(const string& cname, ClassData **pcls)
{
  Mutex::Locker lock(mutex);
  ClassData *cls = _get_class(cname);
  if (cls->status != ClassData::CLASS_OPEN) {
    int r = _load_class(cls);
    if (r)
      return r;
  }
  *pcls = cls;
  return 0;
}

The first thing that happens in _load_class is to handle the case that the class has not yet loaded the library which happens the first time the class loaded (status == CLASS_UNKNOWN), or each time after that until any of its dependencies are missing (status == CLASS_MISSING).

int ClassHandler::_load_class(ClassData *cls)
{
  if (cls->status == ClassData::CLASS_UNKNOWN ||
      cls->status == ClassData::CLASS_MISSING) {

The path to the library is constructed based on a naming convention (libcls_<<class-name>>.so) and then the OSD attempts to load the library using dlopen():

    char fname[PATH_MAX];

    // build the path to the shared library based on the class name
    snprintf(fname, sizeof(fname), "%s/" CLS_PREFIX "%s" CLS_SUFFIX,
         cct->_conf->osd_class_dir.c_str(),
         cls->name.c_str());

    cls->handle = dlopen(fname, RTLD_NOW);

The dlopen() function will return a non-NULL value on success meaning that a native object class was loaded which satisfies our goal of having native object classes take precedence. Here we consider the error case. First we stat the path as it may be the case that the file exists and cannot be loaded for a variety of reasons:

    if (!cls->handle) {
      struct stat st;
      int r = ::stat(fname, &st);

In the case that stat cannot find the file we look for a Lua script with the same object class name. All other errors are handled normally:

      if (r < 0) {
        r = -errno;
        dout(0) << __func__ << " could not stat class " << fname
                << ": " << cpp_strerror(r) << dendl;
        if (r == -ENOENT)
          r = _load_lua_class(cls);
      } else {
        dout(0) << "_load_class could not open class " << fname
                << " (dlopen failed): " << dlerror() << dendl;
        r = -EIO;
      }

If there was an error resolving the class name to a native module or a Lua script, then we change the state of the class to CLASS_MISSING and return an error.

      // dlopen error may be cleared if Lua class found
      if (r) {
        cls->status = ClassData::CLASS_MISSING;
        return r;
      }
    }

Let’s now look at what happens in _load_lua_class before continuing to describe this method. The first thing that we do is use a naming convention to construct a path name for a target Lua script. Notice that here we use the osd_lua_class_dir configuration option that we described previously in this post. Normally it points at the directory /usr/share/ceph/rados-classes/.

int ClassHandler::_load_lua_class(ClassData *cls)
{
  char fname[PATH_MAX];

  snprintf(fname, sizeof(fname),
      "%s/" CLS_LUA_PREFIX "%s" CLS_LUA_SUFFIX,
      cct->_conf->osd_lua_class_dir.c_str(),
      cls->name.c_str());

Next we see if the file exists using stat system call and bail if there is an error:

  struct stat st;
  int r = ::stat(fname, &st);
  if (r < 0) {
    r = -errno;
    dout(0) << __func__ << " could not stat class " << fname
      << ": " << cpp_strerror(r) << dendl;
    return r;
  }

The file exists, but lets add a sanity check before moving on:

  if (st.st_size > CLS_LUA_MAX_SIZE) {
    dout(0) << __func__ << " lua script too big " << fname
      << ": " << st.st_size << " bytes" << dendl;
    return -EIO;
  }

Finally we read the script into a string that is stored in the C++ object representing the class:

  cls->lua_script.reserve(st.st_size);
  std::ifstream ifs(fname);
  cls->lua_script.assign(std::istreambuf_iterator<char>(ifs),
      std::istreambuf_iterator<char>());

To finish up we manually add a dependency on the lua class which is needed to run the script we just read, and mark a special flag on the object class that indicates that this class is defined by a Lua script:

  /*
   * Mark 'lua' as a class dependency.
   */
  ClassData *cls_dep = _get_class("lua");
  cls->dependencies.insert(cls_dep);
  if (cls_dep->status != ClassData::CLASS_OPEN)
    cls->missing_dependencies.insert(cls_dep);

  cls->lua_handler = true;

  return 0;
}

This brings up an important implementation note. We use the object class machinery in the OSD to explicitly represent object classes that are backed by a Lua script. However, the existing representation of object classes assumes the class is backed by a dynamically loaded native shared library. In contrast, classes defined by a Lua script act as a proxy, forwarding their methods onto the native lua class that can evaluate Lua scripts that define object classes.

Now back to the _load_class method. At this point we have a new object class and have either loaded it from native shared library or a Lua script. Before we can use the class we need to make sure any dependencies are met. For native libraries a symbol within the shared library is resolved and interpreted as array of strings defining the set of dependencies. However, for classes backed by a Lua script there is no shared library to interact with. Thus, we need to be careful to ensure that existing code does not assume that a class is backed by a shared library. To do this we simply check that cls->handle is non-NULL (we omit the rest of the process of saving the set of dependencies).

    if (cls->handle) {
      cls_deps_t *(*cls_deps)();
      cls_deps = (cls_deps_t *(*)())dlsym(cls->handle, "class_deps");
      if (cls_deps) {
        cls_deps_t *deps = cls_deps();
        while (deps) {
...

The next step is to actually resolve the dependencies. The immediately proceeding code snippet populates the set of dependencies for shared libraries, but as we saw in _load_lua_class the lua dependency is manually attached to classes backed by Lua scripts. Resolving dependencies is a recursive call to _load_class and is omitted here. It is the same for all types of classes.

Next we configure the class as a proxy. First we grab a reference to the lua object class. The assertion is OK because we should have already resolved the dependency:

  if (cls->lua_handler) {
    ClassHandler::ClassData *lua_cls = _get_class("lua");
    assert(lua_cls && lua_cls->name == "lua" &&
        lua_cls->status == ClassData::CLASS_OPEN);

Next we grab a reference to the eval_bufferlist method on the lua class. Above we demonstrated how to encode operations via the eval_json method, and this is similar but uses internal Ceph encoding protocol.

    ClassHandler::ClassMethod *lua_method;
    lua_method = lua_cls->_get_method("eval_bufferlist");
    assert(lua_method);

Finally we setup a fixed method that will forward invocations to the lua class. Note that the method flags are set to read + write. This is a conservative view necessary because Lua scripts don’t currently have a callback method for exposing these flags. Finally, note that the cxx_func of this proxy method actually points to the method in the lua class.

    cls->lua_method_proxy.cxx_func = lua_method->cxx_func;
    cls->lua_method_proxy.name = "lua_method_proxy";
    cls->lua_method_proxy.flags = CLS_METHOD_RD | CLS_METHOD_WR;
    cls->lua_method_proxy.cls = cls;
  }

At this point I’ll briefly explain this method structure. The main structure representing an object class is ClassHandler::ClassData which contains the structure map<string, ClassMethod> methods_map; that maps the name of a method to a ClassHandler::ClassMethod structure. The ClassHandler::ClassMethod structure stores the method flags and a corresponding function pointer.

When a class is created that is backed by a Lua script it is configured such that a fixed method exists (ClassMethod lua_method_proxy;), and we modify the method resolution routine to always return the proxy method for Lua classes:

diff --git a/src/osd/ClassHandler.cc b/src/osd/ClassHandler.cc
index 5616307..e60783b 100644
--- a/src/osd/ClassHandler.cc
+++ b/src/osd/ClassHandler.cc
@@ -296,6 +296,8 @@ ClassHandler::ClassFilter *ClassHandler::ClassData::register_cxx_filter(
 
 ClassHandler::ClassMethod *ClassHandler::ClassData::_get_method(const char *mname)
 {
+  if (lua_handler)
+    return &lua_method_proxy;
   map<string, ClassHandler::ClassMethod>::iterator iter = methods_map.find(mname);
   if (iter == methods_map.end())
     return NULL;

The final operations in _load_class are to perform initialization on the class and activate the class by setting its status to CLASS_OPEN. Initialization is skipped for Lua classes:

  // initialize (non-Lua classes)
  if (cls->handle) {
    void (*cls_init)() = (void (*)())dlsym(cls->handle, "__cls_init");
    if (cls_init) {
      cls->status = ClassData::CLASS_INITIALIZING;
      cls_init();
    }
  }
  
  dout(10) << "_load_class " << cls->name << " success" << dendl;
  cls->status = ClassData::CLASS_OPEN;
  return 0;
}

Forward Method Calls #

Recall that the OSD performs a pre-processing step on IoCtx::exec operations that verify that the target object class exists. In addition to this step per-method flags are extracted that assist in the remaining processing of the operation. Once a reference to the target class has been obtained the OSD grabs the flags:

int flags = cls->get_method_flags(mname.c_str());
if (flags < 0) {
  if (flags == -ENOENT)
    r = -EOPNOTSUPP;
  else
    r = flags;
  return r;
}
is_read = flags & CLS_METHOD_RD;
is_write = flags & CLS_METHOD_WR;

Taking into account the description of how Lua classes are handled, we know that the class isn’t backed by a shared library. This call thus eventually resolves to the static method we defined that has its flags hard-coded (the _get_method call was shown above in which a Lua class simply returns a fixed proxy method):

int ClassHandler::ClassData::get_method_flags(const char *mname)
{
  Mutex::Locker l(handler->mutex);
  ClassMethod *method = _get_method(mname);
  if (!method)
    return -ENOENT;
  return method->flags;
}

The next stop in the OSD that we are interested in is when the actual operation is invoked. This happens in ReplicatedPG::do_osd_ops. First we grab a reference to the target class by name:

CassHandler::ClassData *cls;
result = osd->class_handler->open_class(cname, &cls);
assert(result == 0);   // init_op_flags() already verified this works.

Then the same for the target method:

ClassHandler::ClassMethod *method = cls->get_method(mname.c_str());
if (!method) {
  dout(10) << "call method " << cname << "." << mname << " does not exist" << dendl;
  result = -EOPNOTSUPP;
  break;
}

Finally the method is invoked. Note that the method name (mname) is passed to the method. Prior to the changes described in this post that parameter was not necessary because the method structure contained a direct function pointer to execute. However, when invoking a method on a Lua class the method doesn’t actually exist. Instead, we pass the target method name to the proxy method which forwards the request to the lua class.

result = method->exec((cls_method_context_t)&ctx, indata, outdata, mname);

Patching Method Execution #

What’s left in processing the request is simply preparing the parameters and forwarding the call to the lua class. Recall that we are using eval_bufferlist which expects the parameter order to be (1) Lua script, (2) method name, and (3) the original input data:

int ClassHandler::ClassMethod::exec(cls_method_context_t ctx,
    bufferlist& indata, bufferlist& outdata, const string& mname)
{
  if (cls->lua_handler) {
    bufferlist tmp_indata;
    ::encode(cls->lua_script, tmp_indata);
    ::encode(mname, tmp_indata);
    ::encode(indata, tmp_indata);
    indata = tmp_indata;
    assert(cxx_func); // we don't setup a C version
  }

The remainder of the exec is left unchanged because the proxy method was already setup with its cxx_func to point at the eval_bufferlist method of the lua class:

  int ret;
  if (cxx_func) {
    // C++ call version
    ret = cxx_func(ctx, &indata, &outdata);
  } else {
    // C version
    char *out = NULL;
    int olen = 0;
    ret = func(ctx, indata.c_str(), indata.length(), &out, &olen);
    if (out) {
      // assume *out was allocated via cls_alloc (which calls malloc!)
      buffer::ptr bp = buffer::claim_malloc(olen, out);
      outdata.push_back(bp);
    }
  }
  return ret;
}

And those are the only changes needed. The entire patch set is fairly small, and includes other important bits like fixing some assumptions on classes being backed by shared libraries.

Testing #

We’ll start with a test to make sure everything works as expected. Here is the Lua class we’ll be using. The say_hello method returns the same result at the hello class that is written in C++, but we append (from Lua) to the output to distinguish between the two.

[nwatkins@kyoto src]$ cat /usr/share/ceph/rados-classes/cls_hellolua.lua 
function say_hello(input, output)
    output:append("Hello, ")
    if #input > 0 then
        output:append(input:str())
    else
        output:append("world")
    end
    output:append("! (from Lua)")
end

cls.register(say_hello)

We’ll test with the following Python snippet:

ret, out = ioctx.execute('oid', 'hellolua', 'say_hello', "Ceph")
print out

When run we get the following output which is what we expected to get:

[nwatkins@kyoto src]$ python test.py
Hello, Ceph! (from Lua)

Next we’ll test that the scripts are re-loaded after each restart by modifying the script to append -- Updated to the message returned to the client:

[nwatkins@kyoto src]$ python test.py
Hello, Ceph! (from Lua) -- Updated

That worked just fine. Now we can test that native libraries take precedence. To do this we’ll rename cls_hellolua.lua to cls_hello.lua and then call the say_hello method on the hello class:

[nwatkins@kyoto src]$ python test.py
Hello, Ceph!

And that is what we expected from the native hello class. Now we’ll test the dependency mechanism. We’ll rename libcls_lua.so to prevent it from being loaded. Then we’ll try to run a Lua script class which should fail, and finally restore the naming of the lua library file and try again for success:

mv /usr/lib/rados-classes/libcls_lua.so /usr/lib/rados-classes/libcls_dont_load.so 

python test.py
[nwatkins@kyoto src]$ python test.py
rados.Error: Ioctx.exec(rbd): failed to exec hellolua:say_hello on oid: errno ENOTSUP

[nwatkins@kyoto src]$ cat out/osd.0.log | grep lua
2015-12-06 20:55:34.996635 7f75c61a3700  0 _load_class could not stat class ./usr/lib/rados-classes/libcls_lua.so: (2) No such file or directory

Now for the success case:

mv /usr/lib/rados-classes/libcls_dont_load.so /usr/lib/rados-classes/libcls_lua.so

[nwatkins@kyoto src]$ python test.py
Hello, Ceph! (from Lua)

Great, so everything is working as expected. In future posts we’ll present more advanced techniques for managing dynamic storage interfaces.