Add plugin system with hooks to override doctype transpilation

docs/configuration.md

@@ -1,20 +1,21 @@
-# Configuration files
+# Configuration
 
-Docstub will automatically look for configuration files in the current directory.
-These files must be named
+Docstub will automatically look for the following configuration files in the current directory:
 
 - `pyproject.toml`
 - `docstub.toml`
 
 Alternatively, config files can also be passed in the command line with the `--config` option.
-In this case, docstub will not look for configuration files in the current directory.
-When multiple configuration files are passed explicitly, their content is merged.
+In this case, docstub will **not** look for configuration files in the current directory.
 
-Out of the box, docstub makes use of an internal configuration files are always loaded.
+When multiple configuration files are found/used, their content is merged in order.
+In case of conflicts, the later file takes precedence.
+
+Docstub also uses internal configuration files.
 One such file is [`numpy_config.toml`](../src/docstub/numpy_config.toml) which provides defaults for NumPy types.
 
 
-## Configuration fields in `[tool.docstub]`
+## Configuration fields
 
 All configuration must be declared inside a `[tool.docstub]` table.
 
@@ -27,7 +28,7 @@ Ignore files and directories matching these [glob-style patterns](https://docs.p
 Patterns that don't start with "/" are interpreted as relative to the
 directory that contains the Python package for which stubs are generated.
 
-Example:
+**Example:**
 
 ```toml
 [tool.docstub]
@@ -39,6 +40,26 @@ ignore_files = [
 - Will ignore any directory anywhere that is named `tests`.
 
 
+(plugin-paths)=
+### `plugin_paths`
+
+[TOML type](https://toml.io/en/latest): array of string(s)
+
+File paths to Python files that implement a [plugin](create_plugin.md).
+Relative paths will be treated as relative to the configuration file's directory.
+
+**Example:**
+
+```toml
+[tool.docstub]
+plugin_paths = [
+    "tools/docstub_extend_syntax.py",
+]
+```
+
+- Will load `docstub_extend_syntax.py` as a plugin.
+
+
 ### `types`
 
 [TOML type](https://toml.io/en/latest): table, mapping string to string
@@ -48,7 +69,7 @@ Docstub can't yet automatically discover where to import types from other packag
 Instead, you can provide this information explicitly.
 Any type on the left side will be associated with the given "module" on the right side.
 
-Example:
+**Example:**
 
 ```toml
 [tool.docstub.types]
@@ -69,7 +90,7 @@ Docstub can't yet automatically discover where to import types from other packag
 Instead, you can provide this information explicitly.
 Any type in a docstring whose prefix matches the name given on the left side, will be associated with the given "module" on the right side.
 
-Example:
+**Example:**
 
 ```toml
 [tool.docstub.type_prefixes]
@@ -87,7 +108,7 @@ plt = "matplotlib.pyplot
 
 Nicknames for types that can be used in docstrings to describe valid Python types or annotations.
 
-Example:
+**Example:**
 
 ```toml
 [tool.docstub.type_nicknames]
@@ -97,3 +118,59 @@ buffer = "collections.abc.Buffer"
 
 - Will map `func` to the `Callable`.
 - Will map `buffer` to `collections.abc.Buffer`.
+
+
+### `doctype_grammar`
+
+[TOML type](https://toml.io/en/latest): string
+
+Override the default grammar with your own.
+You can import, override and extend rules and terminals from "doctype.lark".
+Note that if your grammar results in a modified tree, you need to implement a [custom transformer](#doctype_transformer) as well.
+
+**Example:**
+
+```toml
+[tool.docstub]
+grammar = """
+%import doctype (start, WS_INLINE, ARRAY_NAME)
+
+%extend ARRAY_NAME: "darray"
+
+%ignore WS_INLINE
+"""
+```
+
+- Will allow using [array expressions](#natlang-array) with the term ``darray`` in addition to existing ones.
+
+
+### `doctype_transformer`
+
+Python type: {class}`lark.visitors.Transformer`
+
+Provide your own class to transform the parsed doctype tree into a valid {term}`annotation expression`.
+
+:::{note}
+Only supported in `docstub.py` files!
+:::
+
+**Example:**
+
+```python
+# docstub.py
+
+import lark.visitors
+from docstub._doctype import DoctypeTransformer, Expr
+
+@lark.visitors.v_args(tree=True)
+class MyTransformer(DoctypeTransformer):
+    def union(self, tree: lark.Tree) -> Expr:
+        # Sort items in union alphabetically
+        tree.children = sorted(tree.children, key=str)
+        out = super().union(tree)
+        return out
+
+doctype_transformer = MyTransformer  # Declare the configuration field
+```
+
+- Will sort all members of a union alphabetically, regardless of order in the docstring.

docs/create_plugin.md

@@ -0,0 +1,159 @@
+# Create a plugin
+
+Docstub has experimental support for plugins: standalone Python files that implement a set of hooks to reach directly into Docstub's runtime.
+Plugins are loaded via the configuration field [`plugin_paths`](#plugin-paths).
+
+For a reference of all available hooks see {mod}`docstub.plugins`.
+
+
+## Example: Allow another name in array expressions
+
+We want to expand docstub such that we can use `my_array` inside [array expressions](#natlang-array) like this:
+
+- `my_array of dtype int` → `my_array[int]`
+- `my_array of shape (4, 4)` → `my_array`
+
+First we create the plugin:
+
+```python
+# my_array_plugin.py
+
+from itertools import product
+
+import lark
+from docstub.plugins import hookimpl
+from docstub.doctype import DoctypeTransformer, Expr, Term
+
+my_grammar = """
+%import doctype (start, WS_INLINE, ARRAY_NAME)
+
+%extend ARRAY_NAME: "my_array"
+
+%ignore WS_INLINE
+"""
+
+
+@hookimpl
+def docstub_load_doctype_grammar():
+    return my_grammar
+```
+
+The enable the plugin in the configuration:
+
+```toml
+[tool.docstub]
+plugin_paths = ["path/to/my_array_plugin.py"]
+```
+
+
+## Example: Expansion expression for qualnames
+
+We want to expand docstub such that the following doctype transformations are possible
+
+- `{Iterable, Sequence} of int(s)` → `Iterable[int] | Sequence[int]`
+- `{list, set} of {int, float}` → `list[int] | list[float] | set[int] | set[float]`
+
+First we create the plugin:
+
+```python
+# expansion_plugin.py
+
+from itertools import product
+
+import lark
+from docstub.plugins import hookimpl
+from docstub.doctype import DoctypeTransformer, Expr, Term
+
+my_grammar = """
+%import doctype (start, WS_INLINE, qualname)
+
+%extend qualname: "{" qualname ("," qualname)+ "}" -> expansion
+
+%ignore WS_INLINE
+"""
+
+
+@lark.visitors.v_args(tree=True)
+class ExpansionTransformer(DoctypeTransformer):
+    def start(self, tree):
+        """Transform "start" rule.
+
+        Parameters
+        ----------
+        tree : lark.Tree
+
+        Returns
+        -------
+        expr : Expr
+        """
+        children = map(self._expand_expression, tree.children)
+        return Expr(rule="start", children=tuple(children))
+
+    def expansion(self, tree):
+        """Transform "expansion" rule.
+
+        Actual expansion happens in `start` when the complete expression with
+        all potential expansions can be seen.
+
+        Parameters
+        ----------
+        tree : lark.Tree
+
+        Returns
+        -------
+        expr : Expr
+        """
+        return Expr(rule="expansion", children=tuple(tree.children))
+
+    def _expand_expression(self, expr):
+        """Join combination of all expansions in a new union.
+
+        Parameters
+        ----------
+        expr : Expr
+
+        Returns
+        -------
+        expanded : Expr
+        """
+        expansions = expr.filter(rule="expansion")
+        if not expansions:
+            return expr
+
+        # Create Cartesian product of all expansions where each
+        # value is prefixed by the expansion it will replace
+        expansions = [
+            [(expansion, value) for value in expansion.children]
+            for expansion in expansions
+        ]
+        combinations = list(product(*expansions))
+
+        # Create a new expression for each combination
+        interpolated = []
+        for combination in combinations:
+            new_expr = expr
+            for expansion, value in combination:
+                new_expr = new_expr.replace(expansion, value)
+            interpolated.append(new_expr)
+
+        # Join new expressions as union
+        expanded = self.union(lark.Tree("union", children=interpolated))
+        return expanded
+
+
+@hookimpl
+def docstub_load_doctype_grammar():
+    return my_grammar
+
+
+@hookimpl
+def docstub_load_doctype_transformer():
+    return ExpansionTransformer()
+```
+
+The enable the plugin in the configuration:
+
+```toml
+[tool.docstub]
+plugin_paths = ["path/to/expansion_plugin.py"]
+```

docs/index.md

@@ -22,6 +22,7 @@ It does so by supporting widely used readable conventions such as `array of dtyp
 
 installation
 introduction
+create_plugin
 development/index
 :::
 

docs/installation.md

@@ -19,7 +19,7 @@ In case you want to check out an unreleased version you can install the latest v
 pip install 'docstub @ git+https://github.com/scientific-python/docstub'
 ```
 
-You can append `@COMMIT_SHA` to the repository URL above to intall a specific version other that the `main` branch.
+You can append `@COMMIT_SHA` to the repository URL above to install a specific version other that the `main` branch.
 
 :::{tip}
 If you are want to set up a development environment instead, checkout the [Contribution guide](development/contributing.md).

docs/introduction.md

@@ -139,7 +139,7 @@ Adopting docstub on a large codebase may initially generate many errors.
 Two command line options can help addressing these errors gradually:
 
 * `--group-errors` will group identical errors together.
-  This helps identifying common groups of errors that may be addressed in one go.
+  This helps to identify common groups of errors that may be addressed in one go.
 
 * `--allow-errors` puts an upper limit (["ratchet"](https://qntm.org/ratchet)) on the number of allowed errors.
   This way you can adjust the upper bound of allowed errors as they are addressed.
@@ -154,7 +154,7 @@ Feel welcome to [open an issue](https://github.com/scientific-python/docstub/iss
 ## Dealing with typing problems
 
 For various reasons – missing features in docstub, or limitations of Python's typing system – it may not always be possible to correctly type something in a docstring.
-In those cases, you docstub provides a few approaches to dealing with this.
+In those cases, docstub provides a few approaches to dealing with this.
 
 
 ### Use inline type annotation
@@ -200,6 +200,11 @@ class Foo:
     d: str
 ```
 
+### Create a plugin
+
+Depending on your problem you might be able to deal with it by hooking directly into Docstub's runtime via a plugin. See [](create_plugin.md) for a guide on writing such a plugin.
+
+
 ### Write a manual stub file
 
 If all of the above does not solve your issue, you can fall back to writing a correct stub file by hand.

docs/typing_syntax.md

@@ -69,6 +69,7 @@ In the future, docstub may warn against or disallow nesting these natural langua
 :::
 
 
+(natlang-array)=
 ## Shape and dtype syntax for arrays
 
 This expression allows adding shape and datatype information for data structures like [NumPy arrays](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html).

pyproject.toml

@@ -40,6 +40,7 @@ dependencies = [
     "lark >=1.1.9",
     "black >=24.4.2",
     "isort >=5.13.2",
+    "pluggy",
 ]
 
 [project.urls]