README.rst 11.1 KB
Newer Older
1
.. _modules-api:
2

3 4 5
*********************
Modules API reference
*********************
6 7

.. contents::
8
   :depth: 1
9
   :local:
10 11

Supported languages
12
===================
13

14 15
Currently modules written in C and LuaJIT are supported.
There is also a support for writing modules in Go 1.5+ |---| the library has no native Go bindings, library is accessible using CGO_.
16 17

The anatomy of an extension
18
===========================
19

Marek Vavruša's avatar
Marek Vavruša committed
20
A module is a shared object or script defining specific functions, here's an overview.
21

22
*Note* |---| the :ref:`Modules <lib_api_modules>` header documents the module loading and API.
23 24

.. csv-table::
25
   :header: "C/Go", "Lua", "Params", "Comment"
26

27 28 29 30 31 32
   "``X_api()`` [#]_", "",               "",                "API version"
   "``X_init()``",     "``X.init()``",   "``module``",      "Constructor"
   "``X_deinit()``",   "``X.deinit()``", "``module, key``", "Destructor"
   "``X_config()``",   "``X.config()``", "``module``",      "Configuration"
   "``X_layer()``",    "``X.layer``",    "``module``",      ":ref:`Module layer <lib-layers>`"
   "``X_props()``",    "",               "",                "List of properties"
33 34 35

.. [#] Mandatory symbol.

Marek Vavruša's avatar
Marek Vavruša committed
36
The ``X`` corresponds to the module name, if the module name is ``hints``, then the prefix for constructor would be ``hints_init()``.
37 38
This doesn't apply for Go, as it for now always implements `main` and requires capitalized first letter in order to export its symbol.

39 40 41 42
.. note::
   The resolution context :c:type:`struct kr_context` holds loaded modules for current context. A module can be registered with :c:func:`kr_context_register`, which triggers module constructor *immediately* after the load. Module destructor is automatically called when the resolution context closes.
   
   If the module exports a layer implementation, it is automatically discovered by :c:func:`kr_resolver` on resolution init and plugged in. The order in which the modules are registered corresponds to the call order of layers.
43

Marek Vavruša's avatar
Marek Vavruša committed
44
Writing a module in Lua
45
=======================
Marek Vavruša's avatar
Marek Vavruša committed
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84

The probably most convenient way of writing modules is Lua since you can use already installed modules
from system and have first-class access to the scripting engine. You can also tap to all the events, that
the C API has access to, but keep in mind that transitioning from the C to Lua function is slower than
the other way round.

.. note:: The Lua functions retrieve an additional first parameter compared to the C counterparts - a "state".
          There is no Lua wrapper for C structures used in the resolution context, until they're implemented
          you can inspect the structures using the `ffi <http://luajit.org/ext_ffi.html>`_ library.

The modules follow the `Lua way <http://lua-users.org/wiki/ModuleDefinition>`_, where the module interface is returned in a named table.

.. code-block:: lua

	--- @module Count incoming queries
	local counter = {}

	function counter.init(module)
		counter.total = 0
		counter.last = 0
		counter.failed = 0
	end

	function counter.deinit(module)
		print('counted', counter.total, 'queries')
	end

	-- @function Run the q/s counter with given interval.
	function counter.config(conf)
		-- We can use the scripting facilities here
		if counter.ev then event.cancel(counter.ev)
		event.recurrent(conf.interval, function ()
			print(counter.total - counter.last, 'q/s')
			counter.last = counter.total
		end)
	end

	return counter

Marek Vavruša's avatar
Marek Vavruša committed
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
.. tip:: The API functions may return an integer value just like in other languages, but they may also return a coroutine that will be continued asynchronously. A good use case for this approach is is a deferred initialization, e.g. loading a chunks of data or waiting for I/O.

.. code-block:: lua

	function counter.init(module)
		counter.total = 0
		counter.last = 0
		counter.failed = 0
		return coroutine.create(function ()
			for line in io.lines('/etc/hosts') do
				load(module, line)
				coroutine.yield()
			end
		end)
	end
100

Marek Vavruša's avatar
Marek Vavruša committed
101 102 103 104 105 106 107 108 109 110 111 112
The created module can be then loaded just like any other module, except it isn't very useful since it
doesn't provide any layer to capture events. The Lua module can however provide a processing layer, just
:ref:`like its C counterpart <lib-layers>`.

.. code-block:: lua

	-- Notice it isn't a function, but a table of functions
	counter.layer = {
		begin = function (state, data)
				counter.total = counter.total + 1
				return state
			end,
Marek Vavruša's avatar
Marek Vavruša committed
113 114
		finish = function (state, req, answer)
				if state == kres.FAIL then
Marek Vavruša's avatar
Marek Vavruša committed
115 116 117 118 119 120
					counter.failed = counter.failed + 1
				end
				return state
			end 
	}

121 122 123 124
There is currently an additional "feature" in comparison to C layer functions:
the ``consume``, ``produce`` and ``checkout`` functions do not get called at all
if ``state == kres.FAIL`` (note that ``finish`` does get called nevertheless).

Marek Vavruša's avatar
Marek Vavruša committed
125 126 127 128
Since the modules are like any other Lua modules, you can interact with them through the CLI and and any interface.

.. tip:: The module can be placed anywhere in the Lua search path, in the working directory or in the MODULESDIR.

129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181
Writing a module in C
=====================

As almost all the functions are optional, the minimal module looks like this:

.. code-block:: c

	#include "lib/module.h"
	/* Convenience macro to declare module API. */
	KR_MODULE_EXPORT(mymodule);


Let's define an observer thread for the module as well. It's going to be stub for the sake of brevity,
but you can for example create a condition, and notify the thread from query processing by declaring
module layer (see the :ref:`Writing layers <lib-layers>`).

.. code-block:: c

	static void* observe(void *arg)
	{
		/* ... do some observing ... */
	}

	int mymodule_init(struct kr_module *module)
	{
		/* Create a thread and start it in the background. */
		pthread_t thr_id;
		int ret = pthread_create(&thr_id, NULL, &observe, NULL);
		if (ret != 0) {
			return kr_error(errno);
		}

		/* Keep it in the thread */
		module->data = thr_id;
		return kr_ok();
	}

	int mymodule_deinit(struct kr_module *module)
	{
		/* ... signalize cancellation ... */
		void *res = NULL;
		pthread_t thr_id = (pthread_t) module->data;
		int ret = pthread_join(thr_id, res);
		if (ret != 0) {
			return kr_error(errno);
		}

		return kr_ok();
	}

This example shows how a module can run in the background, this enables you to, for example, observe
and publish data about query resolution.

182
Writing a module in Go
183
======================
184

185
The Go modules use CGO_ to interface C resolver library, there are no native bindings yet. Second issue is that layers are declared as a structure of function pointers, which are `not present in Go`_, the workaround is to declare them in CGO_ header. Each module must be the ``main`` package, here's a minimal example:
186 187 188 189 190 191 192 193 194 195 196

.. code-block:: go

	package main

	/*
	#include "lib/module.h"
	*/
	import "C"
	import "unsafe"

197 198
	/* Mandatory functions */

199 200
	//export mymodule_api
	func mymodule_api() C.uint32_t {
201 202
		return C.KR_MODULE_API
	}
203 204 205 206
	func main() {}

.. warning:: Do not forget to prefix function declarations with ``//export symbol_name``, as only these will be exported in module.

207 208 209 210 211 212 213
In order to integrate with query processing, you have to declare a helper function with function pointers to the
the layer implementation. Since the code prefacing ``import "C"`` is expanded in headers, you need the `static inline` trick
to avoid multiple declarations. Here's how the preface looks like:

.. code-block:: go

	/*
214
	#include "lib/layer.h"
215
	#include "lib/module.h"
216
	// Need a forward declaration of the function signature
217
	int finish(kr_layer_t *);
218
	// Workaround for layers composition
219
	static inline const kr_layer_api_t *_layer(void)
220
	{
221
		static const kr_layer_api_t api = {
222
			.finish = &finish
223 224 225 226 227 228 229
		};
		return &api;
	}
	*/
	import "C"
	import "unsafe"

230
Now we can add the implementations for the ``finish`` layer and finalize the module:
231 232 233

.. code-block:: go

234
	//export finish
235
	func finish(ctx *C.kr_layer_t) C.int {
236
		// Since the context is unsafe.Pointer, we need to cast it
237
		var param *C.struct_kr_request = (*C.struct_kr_request)(ctx.data)
238
		// Now we can use the C API as well
239
		fmt.Printf("[go] resolved %d queries\n", C.list_size(&param.rplan.resolved))
240 241 242
		return 0
	}

243
	//export mymodule_layer
244
	func mymodule_layer(module *C.struct_kr_module) *C.kr_layer_api_t {
245 246 247 248 249 250
		// Wrapping the inline trampoline function
		return C._layer()
	}

See the CGO_ for more information about type conversions and interoperability between the C/Go.

251 252 253 254 255 256 257 258
Gotchas
-------

* ``main()`` function is mandatory in each module, otherwise it won't compile.
* Module layer function implementation must be done in C during ``import "C"``, as Go doesn't support pointers to functions.
* The library doesn't have a Go-ified bindings yet, so interacting with it requires CGO shims, namely structure traversal and type conversions (strings, numbers).
* Other modules can be called through C call ``C.kr_module_call(kr_context, module_name, module_propery, input)``

259
Configuring modules
260
===================
261

Marek Vavruša's avatar
Marek Vavruša committed
262
There is a callback ``X_config()`` that you can implement, see hints module.
263

264 265
.. _mod-properties:

Marek Vavruša's avatar
Marek Vavruša committed
266
Exposing C/Go module properties
267
===============================
268 269 270 271 272

A module can offer NULL-terminated list of *properties*, each property is essentially a callable with free-form JSON input/output.
JSON was chosen as an interchangeable format that doesn't require any schema beforehand, so you can do two things - query the module properties
from external applications or between modules (i.e. `statistics` module can query `cache` module for memory usage).
JSON was chosen not because it's the most efficient protocol, but because it's easy to read and write and interface to outside world.
273 274 275 276

.. note:: The ``void *env`` is a generic module interface. Since we're implementing daemon modules, the pointer can be cast to ``struct engine*``.
          This is guaranteed by the implemented API version (see `Writing a module in C`_).

277 278 279 280
Here's an example how a module can expose its property:

.. code-block:: c

281
	char* get_size(void *env, struct kr_module *m,
282 283
	               const char *args)
	{
284 285
		/* Get cache from engine. */
		struct engine *engine = env;
286
        struct kr_cache *cache = &engine->resolver.cache;
287
		/* Read item count */
288
        int count = (cache->api)->count(cache->db);
289
		char *result = NULL;
290
		asprintf(&result, "{ \"result\": %d }", count);
291 292 293 294 295 296 297 298
		
		return result;
	}

	struct kr_prop *cache_props(void)
	{
		static struct kr_prop prop_list[] = {
			/* Callback,   Name,   Description */
299
			{&get_size, "get_size", "Return number of records."},
300 301 302 303 304 305 306
			{NULL, NULL, NULL}
		};
		return prop_list;
	}

	KR_MODULE_EXPORT(cache)

307 308
Once you load the module, you can call the module property from the interactive console.
*Note* |---| the JSON output will be transparently converted to Lua tables.
309 310 311

.. code-block:: bash

312
	$ kresd
313
	...
314 315
	[system] started in interactive mode, type 'help()'
	> modules.load('cached')
316
	> cached.get_size()
317
	[size] => 53
318 319 320

*Note* |---| this relies on function pointers, so the same ``static inline`` trick as for the ``Layer()`` is required for C/Go.

321 322 323 324 325 326
Special properties
------------------

If the module declares properties ``get`` or ``set``, they can be used in the Lua interpreter as
regular tables.

327 328 329
.. _`not present in Go`: http://blog.golang.org/gos-declaration-syntax
.. _CGO: http://golang.org/cmd/cgo/

330
.. |---| unicode:: U+02014 .. em dash