Hacking on rototiller / libtil: Introduction: This document primarily attempts to describe how one goes about hacking on new rototiller modules. Initially only a bare minimum module addition is described. This is a single-threaded, unconfigurable at runtime, simple module, requiring only a single rendering function be implemented. Later more advanced topics like threaded rendering and runtime configurability will be covered. These are completely optional and can safely be ignored until such facilities are desired. The creative process of developing a new module often starts with writing nothing more than a rendering function, later evolving to become more complex in pursuit of better performance via threaded rendering, and greater flexibility via runtime settings. Getting started: After acquiring a copy of the source, adding a new module to rototiller consists of four steps: 1. Giving the module a unique name, creating a directory as named under src/modules. This can be a temporary working name just to get started, what's important is that it not conflcit with any existing module names. 2. Implementing at least a ${new}_render_fragment() method for the module in a file placed in its directory at "src/modules/${new}/${new}.c". 3. Integrating the module into the build system by adding its directory to the existing "configure.ac" and "src/modules/Makefile.am" files, and creating its own "src/modules/${new}/Makefile.am" file. 4. Binding the module into libtil exposing it to the world by adding it to the modules[] array in "src/til.c". Most of these steps are self-explanatory after looking at the existing code/build-system files. It's common to bootstrap a new module by copying a "Makefile.am" and "${new}.c" file from one of the existing modules. There's also a "stub" branch provided in the git repository, adding a bare minimum module rendering a solid white canvas every frame. This is intended for use as a clean slate for bootstrapping new modules, there's no harm in deriving new modules from either this "stub" branch, or existing modules. Configuring and building the source: Rototiller uses GNU Autotools for its build system. Generally all that's required for building the source is the following sequence of shell commands: $ ./bootstrap $ mkdir build; cd build; ../configure $ make The source is all C, so a C compiler is required. Autotools is also required for `bootstrap` to succeed in generating the configure script and Makefile templates, `pkg-config` is used by configure, and a `make` program for executing the build. On Debian systems installing the "build-essential" meta-package and "libtool" should at least get things building successfully. To actually produce a `rototiller` binary usable for rendering visual output, libsdl2 and/or libdrm development packages will also be needed. Look at the `../configure` output for SDL and DRM lines to see which have been enabled. If both report "no" then the build will only produce a libtil library for potential use in other frontends, with no rototiller binary for the included CLI frontend. After successfully building rototiller with the CLI frontend, an executable will be at "src/rototiller" in the build tree. If the steps above were followed verbatim, that would be at "build/src/rototiller". Quickly testing modules via the CLI frontend: The included frontend supports both an interactive stdio-style setup and specifying those same settings via commandline arguments. If run without any arguments, i.e. just running `build/src/rototiller`, a comprehensive interactive setup will be performed for determining both module and video settings. Prior to actually proceeding with a given setup, the configured setup about to be used is always printed on stdout as valid commandline argument syntax. This may be copied and reused for an automated non-interactive startup using those settings. One can also partially specify any setup in the commandline arguments, resulting in an interactive setup for just the unspecified settings. When developing a new module it's common to specify the video settings, and just the module name under development, leaving the module's settings for interactive specification during the experimentation process. i.e.: $ build/src/rototiller --module=newmodule --video=sdl,fullscreen=off,size=640x480 This way, if "newmodule" implements settings, only those unspecified will be asked for interactively. The render function, a bare minimum module: All rendering in rototiller is performed using the CPU, in 24-bit "True color" pixel format, with 32-bits/4-bytes of space used per pixel. The surface for rendering into is described using a display system agnostic "framebuffer fragment" structure type named "til_fb_fragment_t", defined in "src/til_fb.h" as: typedef struct til_fb_fragment_t { uint32_t *buf; /* pointer to the first pixel in the fragment */ unsigned x, y; /* absolute coordinates of the upper left corner of this fragment */ unsigned width, height; /* width and height of this fragment */ unsigned frame_width; /* width of the frame this fragment is part of */ unsigned frame_height; /* height of the frame this fragment is part of */ unsigned stride; /* number of bytes from the end of one row to the start of the next */ unsigned pitch; /* number of bytes separating y from y + 1, including any padding */ unsigned number; /* this fragment's number as produced by fragmenting */ unsigned cleared:1; /* if this fragment has been cleared since last flip */ } til_fb_fragment_t; For most modules these members are simply used as provided, and there's no need to manipulate them. For simple non-threaded cases only the "buf" and "{width,height}" members are required, with "stride" or "pitch" becoming important for algorithms directly manipulating buf's memory contents to properly address rows of pixels since fragments may be discontiguous in buf at row boundaries for a variety of reasons. When using threaded rendering, the "frame_{width,height}" members become important as they describe a fragment's full-frame dimensions, while "{width,height}" describe the specific fragment within the frame being rendered by render_fragment(). In non-threaded scenarios these members have the same values, but threading employs subfragmenting the frame into independent fragments potentially rendered concurrently. The module_render() function prototype is declared within the "til_module_t" struct in "src/til.h" as: void (*render_fragment)(void *context, unsigned ticks, unsigned cpu, til_fb_fragment_t *fragment); Every module must provide a "til_module_t" instance having at least this "render_fragment" member initialized to its rendering function. This is typically done using a global instance named with the module's prefix. None of the other function pointer members in "til_module_t" are required, and the convention is to use designated initialization in assigning a module's "til_module_t" members ensuring zero-initialization of omitted members, i.e.: static void minimal_render_fragment(void *context, unsigned ticks, unsigned cpu, til_fb_fragment_t *fragment) { /* render into fragment->buf */ } til_module_t minimal_module = { .render_fragment = minimal_render_fragment, .name = "minimal", .description = "Minimal example module", } Note that the render_fragment() prototype has additional arguments than just the "til_fb_fragment_t *fragment": void *context: For modules implementing a create_context() function, this will be the pointer returned by that function. Intended for modules that require state persisted across frames rendered. unsigned ticks: A convenient time-like counter the frontend advances during operation. Instead of calling some kind of time function in every module which may become costly, "ticks" may be used to represent time. unsigned cpu: An identifier representing which logical CPU # the render function is executing on. This isn't interesting for simple single-threaded modules, but when implementing more advanced threaded renderers this may be useful for indexing per-cpu resources to avoid contention. For simple modules these can all be safely ignored, "ticks" does tend to be useful for even simple modules however. Rendering functions shouldn't make assumptions about the contents of "fragment->buf", in part because rototiller will always use multiple buffers for rendering which may be recycled in any order. Additionally, it's possible a given fragment will be further manipulated in composited scenarios. Consequently it's important that every render_fragment() function fully render the region described by the fragment. There tends to be two classes of rendering algorithms; those that always produce a substantial color for every pixel available in the output, and those producing more sparse output resembling an overlay. In the latter case it's common to require bulk-clearing the fragment before the algorithm draws its sparse overlay-like contents onto the canvas. To facilitate potential compositing of such modules, the "til_fb_fragment_t" structure contains a "cleared" member used to indicate if a given fragment's buf contents have been fully initialized yet for the current frame. When "cleared" is already set, the bulk clearing operation should be skipped, allowing the existing contents to serve as the logically blank canvas. A convenience helper for such modules is provided named til_fb_fragment_clear(). Simply call this at the start of the render_fragment() function, and the conditional cleared vs. non-cleared details will be handled automatically. Otherwise see the implementation in "src/til_fb.h" to see what's appropriate. To clarify, modules implementing algorithms that naturally always write every pixel in the fragment may completely ignore this aspect, and need not set the cleared member; that's handled automatically. Stateful rendering: It's common to require some state persisting from one frame to the next. Achieving this is a simple matter of providing create_context() and destroy_context() functions when initializing til_module_t, i.e.: typedef struct minimal_context_t { int stateful_variables; } minimal_context_t; static void * minimal_create_context(unsigned ticks, unsigned n_cpus, void *setup) { /* this can include more elaborate initialization of minimal_context_t as needed */ return calloc(1, sizeof(minimal_context_t)); } static void minimal_destroy_context(void *context) { free(context); } static void minimal_render_fragment(void *context, unsigned ticks, unsigned cpu, til_fb_fragment_t *fragment) { minimal_context_t *ctxt = context; /* render into fragment->buf utilizing/updating ctxt->stateful_variables */ } til_module_t minimal_module = { .create_context = minimal_create_context, .destroy_context = minimal_destroy_context, .render_fragment = minimal_render_fragment, .name = "minimal", .description = "Minimal example module", } Note that the create_context() function prototype includes some arguments: unsigned ticks: Same as render_fragment; a time-like counter. This is provided to the create_context() function in the event that some ticks-derived state must be initialized continuously with the ticks value subsequently passed to render_fragment(). This is often ignored. unsigned n_cpus: This is the number of logical CPUs rototiller is running atop, which is potentially relevant for threaded renderers. The "unsigned cpu" parameter supplied to render_fragment() will always be < this n_cpus value, and the two are intended to complement eachother. When creating the context, one may allocate per-cpu cache-aligned space in n_cpus quantity. Then the render_fragment() function would address this per-cpu space using the cpu parameter as an index into the n_cpus sized allocation. This is often ignored. void *setup: For modules implementing runtime-configuration by providing a setup() function in their til_module_t initializer, this will contain the pointer returned in res_setup by their setup() function. Unless implementing runtime configuration, this would be ignored. As mentioned above in describing the rendering function, this is entirely optional. One may create 100% valid modules implementing only the render_fragment() function. Runtime-configurable rendering: For myriad reasons ranging from debugging and creative experimentation, to aesthetic variety, it's important to support runtime configuration of modules. Everything configurable that is potentially interesting to a viewer is best exposed via runtime settings, as opposed to hidden behind compile-time constants like #defines or magic numbers in the source. It's implied that when adding runtime configuration to a module, it will also involve stateful rendering as described in the previous section. This isn't absolutely required, but without an allocated context to apply the runtime-configuration to, the configuration will be applied in some global fashion. Any modules to be merged upstream shouldn't apply their configuration globally if at all avoidable. Adding runtime configuration requires implementing a setup() function for a given module. This setup() function is then provided when initializing til_module_t. Building upon the previous minimal example from stateful rendering: typedef struct minimal_setup_t { int foobar; } minimal_setup_t; typedef struct minimal_context_t { int stateful_variables; } minimal_context_t; static void * minimal_create_context(unsigned ticks, unsigned n_cpus, void *setup) { minimal_context_t *ctxt; ctxt = calloc(1, sizeof(minimal_context_t)); if (!ctxt) return NULL; ctxt->stateful_variables = ((minimal_setup_t *)setup)->foobar; return ctxt; } static void minimal_destroy_context(void *context) { free(context); } static void minimal_render_fragment(void *context, unsigned ticks, unsigned cpu, til_fb_fragment_t *fragment) { minimal_context_t *ctxt = context; /* render into fragment->buf utilizing/updating ctxt->stateful_variables */ } static int minimal_setup(const til_settings_t *settings, til_setting_t **res_setting, const til_setting_desc_t **res_desc, void **res_setup) { const char *values[] = { "off", "on", NULL }; const char *foobar; int r; r = til_settings_get_and_describe_value(settings, &(til_setting_desc_t){ .name = "Foobar configurable setting", .key = "foobar", .regex = "^(off|on)", .preferred = values[0], .values = values, .annotations = NULL }, &foobar, res_setting, res_desc); if (r) return r; if (res_setup) { minimal_setup_t *setup; setup = calloc(1, sizeof(*setup)); if (!setup) return -ENOMEM; if (!strcasecmp(foobar, "on")) setup->foobar = 1; *res_setup = setup; } return 0; } til_module_t minimal_module = { .create_context = minimal_create_context, .destroy_context = minimal_destroy_context, .render_fragment = minimal_render_fragment, .setup = minimal_setup, .name = "minimal", .description = "Minimal example module", } In the above example the minimal module now has a "foobar" boolean style setting supporting the values "on" and "off". It may be specified at runtime to rototiller (or any other frontend) via the commandline argument: "--module=minimal,foobar=on" And if the "foobar=on" setting were omitted from the commandline, in rototiller's CLI frontend an interactive setup dialog would occur, i.e: Foobar configurable setting: 0: off 1: on Enter a value 0-1 [0 (off)]: Much can be said on the subject of settings, this introduction should be enough to get started. Use the existing modules as a reference on how to implement settings. The sparkler module in particular has one of the more complicated setup() functions involving dependencies where some settings become expected and described only if others are enabled. None of the frontends currently enforce the regex, but it's best to always populate it with something valid as enforcement will become implemented at some point in the future. Module authors should be able to largely assume the input is valid at least in terms of passing the regex. Note how the minimal_setup_t instance returned by setup() in res_setup is subsequently supplied to minimal_create_context() in its setup parameter. In the previous "Stateful rendering" example, this setup parameter was ignored as it would always be NULL lacking any setup() function. But here we use it to retrieve the "foobar" value wired up by the minimal_setup() function supplied as minimal_module.setup. Threaded rendering: Rototiller deliberately abstains from utilizing any GPU hardware- acceleration for rendering. Instead, all rendering is done using the CPU programmed simply in C, without incurring a bunch of GPU API complexity like OpenGL/Direct3D or any need to manage GPU resources. Modern systems tend to have multiple CPU cores, enabling parallel execution similar to how GPUs employ multiple execution units for parallel rendering of pixels. With some care and little effort rototiller modules may exploit these additional CPU resources. Rototiller takes care of the low-level minutia surrounding creating threads and efficiently scheduling rendering across them for every frame. The way modules integrate into this threaded rendering machinery is by implementing a prepare_frame() function that gets called at the start of every frame in a single-threaded fashion, followed by parallel execution of the module's render_fragment() function from potentially many threads. The prepare_frame() function prototype is declared within the "til_module_t" struct in "src/til.h" as: void (*prepare_frame)(void *context, unsigned ticks, unsigned n_cpus, til_fb_fragment_t *fragment, til_fragmenter_t *res_fragmenter); The context, ticks, n_cpus, and fragment parameters here are semantically identical to their use in the other til_module_t functions explained previously in this document. What's special here is the res_fragmenter parameter. This is where your module is expected to provide a fragmenter function rototiller will call repeatedly while breaking up the frame's fragment being rendered into smaller subfragments for passing to the module's render_fragment() in place of the larger frame's whole fragment. This effectively gives modules control over the order, quantity, size, and shape, of individually rendered subfragments. Logically speaking, one may view the fragments described by the fragmenter function returned in res_fragmenter as the potentially parallel units of work dispatched to the rendering threads. The fragmenter function's prototype is declared in the "til_fragmenter_t" typedef, also in "src/til.h": typedef int (*til_fragmenter_t)(void *context, const til_fb_fragment_t *fragment, unsigned number, til_fb_fragment_t *res_fragment); While rototiller renders a frame using the provided fragmenter, it repeatedly calls the fragmenter with an increasing number parameter until the fragmenter returns 0. The fragmenter is expected to return 1 when it describes another fragment for the supplied number in *res_fragment. For a given frame being rendered this way, the context and fragment parameters will be uniform throughout the frame. The produced fragment in *res_fragment is expected to describe a subset of the provided fragment. Some rudimentary fragmenting helpers have been provided in "src/til_fb.[ch]": int til_fb_fragment_slice_single(const til_fb_fragment_t *fragment, unsigned n_fragments, unsigned num, til_fb_fragment_t *res_fragment); int til_fb_fragment_tile_single(const til_fb_fragment_t *fragment, unsigned tile_size, unsigned num, til_fb_fragment_t *res_fragment); It's common for threaded modules to simply call one of these in their fragmenter function, i.e. in the "ray" module: static int ray_fragmenter(void *context, const til_fb_fragment_t *fragment, unsigned number, til_fb_fragment_t *res_fragment) { return til_fb_fragment_tile_single(fragment, 64, number, res_fragment); } This results in tiling the frame into 64x64 tiles which are then passed to the module's render_fragment(). The other helper, til_fb_fragment_slice_single(), instead slices up the input fragment into n_fragments horizontal slices. Which is preferable depends on the rendering algorithm. Use of these helpers is optional and provided just for convenience, modules are free to do whatever they wish here. Building upon the first minimal example from above, here's an example adding threaded (tiled) rendering: static int minimal_fragmenter(void *context, const til_fb_fragment_t *fragment, unsigned number, til_fb_fragment_t *res_fragment) { return til_fb_fragment_tile_single(fragment, 64, number, res_fragment); } static void minimal_prepare_frame)(void *context, unsigned ticks, unsigned n_cpus, til_fb_fragment_t *fragment, til_fragmenter_t *res_fragmenter) { *res_fragmenter = minimal_fragmenter; } static void minimal_render_fragment(void *context, unsigned ticks, unsigned cpu, til_fb_fragment_t *fragment) { /* render into fragment->buf, which will be a 64x64 tile within the frame (modulo clipping) */ /* Note: * fragment->frame_{width,height} reflect the dimensions of the * whole-frame fragment provided to prepare_frame() * * fragment->{x,y,width,height} describe this fragment's tile * within the frame, which fragment->buf points at the upper left * corner of. */ } til_module_t minimal_module = { .prepare_frame = minimal_prepare_frame, .render_fragment = minimal_render_fragment, .name = "minimal", .description = "Minimal threaded example module", } That's all one must do to achieve threaded rendering. Note however this places new constraints on what's safe to do from within the module's render_fragment() function. When using threaded rendering, any varying state accessed via render_fragment() must either be thread-local or synchronized using a mutex or atomic intrinsics. For performance reasons, the thread-local option is strongly preferred, as it avoids the need for any atomics. Both the create_context() and prepare_frame() functions receive an n_cpus parameter primarily for the purpose of preparing per-thread/per-cpu resources that may then be trivially indexed using the cpu parameter supplied to render_fragment(). When preparing such per-thread resources, care must be taken to avoid sharing of cache lines. A trivial (though wasteful) way to achieve this is to simply page-align the per-cpu allocation. With more intimate knowledge of the cache line size (64 bytes is very common), one can be more frugal. See the "snow" module for an example of using per-cpu state for lockless threaded stateful rendering.