ZAP 007 - Module system¶

14-17 min read · 3,533 words · View in Zensical Spark

Abstract: This ZAP establishes the requirements for the design of Zensical’s module system, the extensibility layer of its differential build runtime. To motivate and ground these requirements, it first examines the MkDocs plugin API in depth, identifying the architectural limitations that constrain its evolution: a callback-and-priority model with no explicit data dependencies, shared mutable state, hard-wired components, and no built-in support for caching, parallelism, or multi-project coordination. From this analysis, the ZAP derives the core design principles for Zensical’s module system, in which modules declare their data inputs and outputs explicitly so that the runtime can derive execution order, schedule concurrent operations, manage caching and I/O centrally, and produce correct differential builds. Module authoring is simplified, since key concerns are hoisted into the runtime.

Problem statement¶

Material for MkDocs has become a popular choice among Open Source projects, a growing number of professional users, and many others. Key to its success has been the combination of its own plugins with third-party plugins available in the wider MkDocs ecosystem.

At the same time, shortcomings in the architecture of MkDocs have held further development of essential functionality back for a long time. Plugin development for MkDocs is unnecessarily complicated and constrained by its architecture, requiring substantial boilerplate code and often necessitating complex and fragile workarounds. At the same time, many design choices like the callback-and-priority model in MkDocs limit its scalability and its uptake by organizations with very large documentation projects.

This is why we built Zensical. Our aim is to develop a fully modular, modern SSG that can scale even to the largest and most complex projects while offering a batteries-included approach that makes getting started and scaling up very easy. Key to this is a differential build runtime. The actual SSG is built as modules consisting of functions that declare their specific data inputs and outputs, allowing the runtime to orchestrate their operation.

In this ZAP, we outline the requirements Zensical’s module system must meet. This is an important step on our path to feature parity with Material for MkDocs and the wider MkDocs ecosystem. We expect to ship support for the functionality of many important plugins over the next 2-3 months, allowing more and more users of Material for MkDocs to transition to Zensical.

The module system needs to deliver the modularity and authoring experience we need to achieve this ambitious goal. Its design will also have a major impact on the maintainability of the code we and the community will write and maintain. With much of what is boilerplate in MkDocs plugins hoisted into the runtime and module system, adding new functionality to Zensical will be significantly easier than building and maintaining MkDocs plugins.

Purpose¶

This ZAP outlines the fundamental requirements for the architectural design of Zensical’s module system. It does not concern itself with defining a public API, since we will first road-test the module system ourselves before opening it up to the wider community. This iterative process ensures that we do not prematurely commit to an API before we have had the chance to learn from our own work on modules as part of achieving feature parity with MkDocs.

Background¶

Before we outline an architecture for the module system, it is important to identify what we can learn from the existing MkDocs API and its strengths and weaknesses. We focus on several aspects below before drawing conclusions for Zensical’s module system.

The MkDocs architecture and plugins¶

The MkDocs plugin API has enabled the ecosystem of plugins, which is pivotal to the success and industry-wide adoption of MkDocs. An advantage the API has is that it follows a familiar pattern. Plugins register callbacks that are invoked at specific points corresponding to a pre-defined set of events in the linear build process. There are global events and page-specific events. Since plugins are implemented as Python classes, plugin callbacks are allowed to have side effects, allowing plugins to store data and use it later in another callback.

This pattern is simple and easy for developers to understand. However, it is also the source of many of the problems that are constraining how MkDocs and its plugins could evolve. Below, we spell out what cannot be achieved without making fundamental changes that amount to a rewrite rather than an incremental improvement.

Imprecise contracts with plugins¶

MkDocs code runs from one event to the next, guaranteeing that specific work is completed before plugin callbacks are invoked. The documentation lists the guarantees made at each point. There is no way for plugins to specify their dependencies on build artifacts or other plugins.

For instance, a plugin that modifies only blog posts needs to wait until all Markdown files have been loaded into memory, even ones it will never touch. At the same time, the plugin might want to allow for blog posts to be generated by other plugins. Since plugins cannot explicitly define dependencies, the only thing our blog plugin can do is specify a very low priority for its callbacks and hope that other plugins that produce inputs choose higher ones.

The plugin callbacks are regular synchronization points that limit the possible performance benefits of parallel execution. At the time MkDocs was written, this was essentially an academic point since Python did not have a threading model. Parallelization across processes would not yield a speedup because of the overhead of serialization. With the introduction of free-threaded Python, this constraint was lifted, but the fixed execution flow of the plugin API and the presence of shared mutable state that plugins can write to and read from still stand in the way of achieving parallelization.

Likewise, the end of a plugin callback is a synchronization point. A plugin must complete its business before returning so that the callbacks of other plugins can run. Any modifications of data structures must be complete to ensure consistency.

This is to say that there is only very limited scope for work to progress in parallel, even with free-threaded Python, despite much of the work of a static site generator being embarrassingly parallel. The global synchronization points exist because there is no model of the data dependencies of the plugin callbacks.

No coordination between multiple builds¶

The projects plugin was an attempt to implement one of the most frequently requested features in the MkDocs ecosystem, to run builds of multiple subprojects at the same time. Subprojects are potentially useful for multi-language projects but also for larger projects that are split up for reasons of ownership or for performance.

The development of the projects plugin ran into fundamental obstacles rooted in MkDocs’ architecture. Initial progress was encouraging as the plugin gained support for nested projects and a tree structure where sub-projects can themselves contain further sub-projects. Difficulties emerged quickly, however, and they all trace back to the same underlying problem: MkDocs provides no programmatic API for coordinating multiple builds.

Cross-project navigation proved especially difficult to implement. When a page in one project links to a page in another, the link cannot be resolved until both projects have been built, yet navigation must be resolved before rendering can begin. The plugin works around this with a custom project:// URI scheme that is translated after the page is rendered. This requires the plugin to maintain a manifest, a global registry mapping project slugs to file paths, written to the cache directory and shared across process boundaries. Resolving these links then demands careful path arithmetic accounting for the relative positions of each project’s output directory in the final site hierarchy.

The concurrency problem compounds this. MkDocs is not thread-safe, which makes it impossible to build projects concurrently within the same process. The projects plugin is therefore forced to spawn each sub-project build as a separate OS process, adding significant overhead and making it difficult to share state between builds. The manifest stored on disk is essentially a workaround for the absence of any inter-process data-sharing mechanism in MkDocs.

These workarounds are complex and fragile. The projects plugin’s complexity is not incidental: it is the unavoidable cost of attempting to coordinate multiple builds within an architecture that was never designed to support it.

Hard-wired dependencies¶

Key components like the Python Markdown parser and the Jinja templating engine are hard-coded into MkDocs. This means that plugins have to work with the APIs these components provide. Crucially, the Python Markdown parser does not provide an abstract syntax tree for Markdown, only for HTML. This forces many plugins to use regular expressions, which means that the same Markdown text is scanned multiple times. Python Markdown itself uses regular expressions in large parts of its parser.

Plugins that need to manipulate the content end up being closely coupled to both Python Markdown and MkDocs, as they use the APIs of both projects. This means the code is usable only in the context of MkDocs, and that MkDocs cannot switch to a different Markdown parser without affecting its users and ecosystem.

What is more, Python Markdown is not a standardized Markdown dialect – it is defined by its single implementation. It differs from CommonMark in important respects, especially in how it handles edge cases and in its rigid whitespace and parsing rules. CommonMark prioritizes correctness and cross-implementation consistency. This means that many of the tools available to technical writers, such as editors, Markdown previews, or linters, typically target CommonMark.

Memory-based operation¶

MkDocs holds all data structures in memory for the entire duration of a build – nothing is flushed to disk until completion. For large projects, this makes memory consumption excessive. The problem is compounded by circular references between files and pages, unmanaged memory growth over the course of a build, and the fact that plugins can easily introduce memory leaks by holding references in instance variables without cleaning up.

In practice, larger projects may not even build on machines with limited available main memory.

Caching strategies¶

MkDocs does not provide any support for caching intermediate build artifacts. Every build runs every step of the build logic from scratch, including all the plugin callbacks. Thus, plugins that perform expensive operations such as parsing source code for API documentation or reading and optimizing image files would need to implement their own caching mechanisms. Most plugin developers do not have the effort available to implement a caching strategy as doing so can go wrong in many subtle ways.

Caching artifacts is a performance optimization that is best provided by the runtime rather than being left to every plugin developer to implement. The runtime has a view of the global build state - the file changes, configuration changes, dependencies, and build progress to implement an effective caching strategy that leads to efficient and correct builds. Unifying storage of build artifacts into a centrally managed cache is the only feasible way to persist cache state in a CI environment, since storage locations for caches are not guaranteed to be consistent between plugins.

Serving partial builds¶

Authors working on large documentation sites have to wait - sometimes 30 minutes or more - before mkdocs serve becomes usable, because MkDocs performs a full build before the preview server accepts connections. This is especially painful because many MkDocs extensions, such as tabbed and superfences, have no editor preview support. Authors depend entirely on the live server to see their work rendered correctly.

The existing --dirtyreload flag only speeds up subsequent builds, not the initial startup phase, and does so at the cost of correctness: it produces pages with invalid navigation and incomplete metadata. Moreover, most plugins do not account for the effects of --dirtyreload, which breaks their functionality.

MkDocs cannot safely hand off any page to the preview server until the entire site has been built. Navigation is the central obstacle: every page’s template requires the complete site structure to render correctly, so no page can be served in isolation. Beyond navigation, plugins that produce cross-cutting content, such as tag indexes or cross-references, depend on the full collection of pages being available at render time.

Hard-wired inputs and outputs¶

The architecture of MkDocs hard-codes Python Markdown to render Markdown to HTML, and Jinja to render the resulting HTML fragments into complete pages. Files that are not Markdown are just copied to the site_dir as static files. This means that there is no way to support other Markdown dialects, as Python Markdown is hard-wired into MkDocs. Likewise, the templating engine is fixed, and the rendering pipeline always produces HTML.

There are plugins that read other inputs and produce other output files, e.g. Jupyter notebooks, but they need to perform their own I/O operations. Moreover, there are no mechanisms for plugins to collaborate, which means input files may be read and parsed multiple times. For example, a CSV file with data to be displayed in a table and rendered as a chart would be read twice.

A bigger problem, though, is that there is no standard way to further process a plugin’s output. Plugins typically write their output to disk because there is no mechanism to hand data off to other plugins in the MkDocs API. This means that chaining plugins to, for example, optimize a generated image is only possible via implicit plugin callback priorities, and through shared state by attaching to MkDocs’ data types.

MkDocs doesn’t provide an asset pipeline: it simply copies non-Markdown content to the site_dir without giving any guarantee when this will happen. Plugin authors have to rely on knowledge of the implementation, creating a fragile dependency on MkDocs internals.

The optimize plugin, for example, removes images that it will optimize from the Files collection to “steal” them from MkDocs. It does so on the on_env event, which is called just before MkDocs copies static files. There is no reason, in principle, why the plugin could not do the work earlier, except that deferring the operation to the latest possible point gives other plugins a chance to add files to the collection.

No mechanisms for contracts between plugins¶

There are no mechanisms that MkDocs provides to allow plugins to collaborate with each other. Consider, for example, the case of the tags plugin and search. The `tags plugin produces an index of which pages contain each tag. There are different ways in which the information could be made available to the search plugin.

The tags plugin could store its data in an instance variable, accessible to the search plugin via config.plugins["tags"]. But this is loose coupling in name only as the search plugin still assumes a specific plugin is registered under "tags" with a specific internal structure. There is no contract, no interface, no guarantee. It works until it silently doesn’t.

What is more, if both plugins were registering callbacks for the same MkDocs event, the order in which their callbacks are called becomes important. MkDocs defines a priority value to allow plugin authors to influence the order in which callbacks are called. Priority values are a poor substitute for explicit dependencies, though, since they do not express intention.

The table-readers plugin, for example, explicitly checks whether it is listed after the plugins it interacts with. It wants to run before the markdownextradata plugin and after macros.

Developer experience¶

Plugin authors must understand which data are available at which point in the build in order to register callbacks for the correct events. The documentation rarely makes these guarantees explicit, and for anything beyond basic use cases, authors are often forced to read MkDocs’ internal source code to understand what they can rely on.

A good example is page titles: the value changes depending on where in the build you are. Before markdown parsing it is taken from the navigation config or derived from the filename, and only after parsing does it reflect the final page title. There is no way to declare these dependencies explicitly, leaving authors to discover the subtleties themselves.

Page titles are a particularly underspecified corner of MkDocs. The behavior is inconsistent enough that we dedicated a ZAP to untangling all the subtleties of the underlying design.

Conclusions¶

The MkDocs architecture offers a lesson in how a well-intentioned plugin API can become a ceiling rather than a foundation. The callback-and-priority model was appropriate for its time and for the scale it was designed to support, but each of the problems documented above traces back to the same issues: the architecture has no model of data, of how inputs are transformed into outputs, and of the resulting dependencies.

Plugins are given access to shared mutable state at fixed synchronization points, and everything else – ordering, caching, coordination and parallelization – has to be taken care of by plugin developers, which leads to the same problems being solved over and over again, if even possible.

The end result is a system that lacks modularity and has become virtually impossible for the maintainers to evolve. MkDocs was always moving at a slow pace and with extreme conservatism: there have been no major changes in years that meaningfully improve the developer or authoring experience, and the limitations described above remain unaddressed. This is the reason why we are building Zensical, but may also explain the decision of the original maintainer to abandon plugin support entirely in MkDocs 2.0.

Considerations¶

For Zensical, the central lesson is that modules must declare their data dependencies explicitly. When the runtime knows what each module reads and what it produces, a great deal follows automatically. The execution order can be derived from the dependency graph rather than approximated with priority values. The runtime can determine which parts of the build are safe to run concurrently. It can identify which modules are unaffected by a change and skip them, giving us differential builds without involvement of the module author. It can begin serving a page as soon as the modules that contribute to it have completed, rather than waiting for the full build.

The definition of data dependencies must be explicit, by leveraging input types and selectors that can specify the source of data as well as a processing context. This enables the runtime to efficiently schedule operations and cache build artifacts. The implementation of the selector logic must be performant as it will be used a lot for build orchestration.

Modules must avoid working on shared state. MkDocs plugins communicate by writing to shared objects – the page, the Files collection, instance variables – always relying on ordering, which is brittle. The result is implicit, fragile coupling. In Zensical, modules should exchange typed data values through the runtime rather than writing to shared structures. This makes dependencies visible and the system auditable: the runtime knows which module produced a value and which modules consume it. It also makes alternative implementations straightforward: a module depends on data of a given type, not on a specific plugin by name.

By using architectural hoisting to move key concerns into the runtime, modules are much simpler and faster to build and maintain. Much of the complexity in the MkDocs plugin ecosystem exists because the runtime does too little: caching, concurrency, cross-project coordination, and partial builds all have to be re-invented, imperfectly, at the plugin layer. As argued above, these concerns are a better fit for the runtime, which has the global view of the build necessary to implement them correctly and efficiently. The module system should therefore be deliberately thin on the side it exposes to module authors. A module declares its inputs, produces its outputs, and the runtime handles the rest.

Hard-wired components should become modules. Python Markdown and Jinja are not wrong choices, but baking them into the architecture means the ecosystem is locked into them. If the Markdown parser and the templating engine are themselves modules with declared inputs and outputs, they can be replaced or composed with alternatives. This is especially important as Zensical expands beyond the standard documentation use case. Native PDF and EPUB support, for example, require entirely different rendering pipelines with no HTML involved. A modular architecture makes these not special cases to hack around, but first-class configurations of the same underlying system.

All I/O should go through the runtime. This ensures the system remains modular with respect to where data comes from and where results are written. The runtime can be backed by a filesystem, a database, or a content repository without the rest of the system caring. The runtime is also responsible for file watching, triggering exactly the operations that need to re-run when an input changes, and nothing more.