shen-lua

Shen on LuaJIT 2.1: pattern matching, optional sequent types, and Prolog, compiled to Lua. ShenOSKernel 43.1, 154/154 official kernel tests. Embeds in any Lua host. Plain Lua 5.1/5.4/5.5 works too (slower, still correct).

git clone https://github.com/pyrex41/shen-lua && cd shen-lua
bin/shen                            # REPL
bin/shen -e "(+ 1 2)"               # one-liner
bin/shen examples/family.shen       # a program (Shen Prolog in 20 lines)
luajit examples/hello_embed.lua     # embed in Lua, ~25 lines

LuaJIT 2.1 is the primary host (brew install luajit / apt-get install luajit); the source checkout also supports plain Lua as described below. First boot compiles the kernel (including the precompiled standard library, stlib.kl); subsequent boots use the bytecode and standard-library signature caches. Loaded programs are cached fasl-style. Cross-port agreement lives in Bifrost. New to Shen? shenlanguage.org.

How it works

KLambda (the ~46-primitive untyped kernel) is compiled to Lua source; LuaJIT trace-compiles that to machine code. Special forms become native if/return; tail calls are real Lua TCO (and self-tails become loops). type is erased at the Kλ boundary — type checking is the kernel’s own Shen, unchanged.

File	Role
runtime.lua	values, intern, KLambda reader
compiler.lua	KLambda → Lua
prims.lua	primitives, apply/curry, native overrides
boot.lua	kernel load, bytecode + fasl caches
shen.lua	embedding API (require("shen"))
lua_interop.lua	Lua ⇄ Shen
repl.lua	REPL
prolog_engine.lua / prolog_compile.lua / typecheck_native.lua	native Prolog / typecheck

Numbers, strings, and booleans are Lua’s. Symbols are interned (identity ==). () is a unique NIL. Cons is {h,t}; vectors are array tables.

Strings and Unicode

A Shen string is a sequence of Unicode code points, held as a UTF-8 Lua string. pos, tlstr, hdstr, string->n, explode, hash (and anything built on them, such as string.length) count characters, not bytes. (n->string N) returns the UTF-8 encoding of any code point 0..U+10FFFF except the surrogates U+D800..U+DFFF, and raises an error for anything else. The reader decodes source files, read-from-string, and (read Stream) as UTF-8, so "Zürich" reads as six characters and prints back unchanged. pr writes UTF-8. read-byte, write-byte, and read-file-as-bytelist stay raw byte operations. ASCII text stays on a byte-indexed fast path.

Invalid UTF-8 never raises an error and never loses data. Each byte that does not start a well-formed sequence counts as one character on its own: a stray continuation byte, a 0xC0/0xC1/0xF5–0xFF lead byte, a truncated, overlong, or surrogate sequence, or a value above U+10FFFF. pos, tlstr, and explode return that raw byte, so (cn (hdstr S) (tlstr S)) is always S. string->n returns the byte's value (0x80–0xFF), so such a byte in source text reads as the character with that code (Latin-1). One thing does not round-trip: (n->string (string->n B)) gives the valid two-byte encoding of that code, not the original byte.

CLI

bin/shen                       # REPL (multiline, history, Shen backtraces)
bin/shen prog.shen ...         # (load) each file
bin/shen -e "(+ 1 2)"          # eval and print
bin/shen --hush-load prog.shen # run a program; no load echo (use this for golden suites)
bin/shen -q prog.shen          # *hush*: silences load echo *and* (output ...)

--hush-load (or SHEN_HUSH_LOAD=1) is what batch runners want: the program’s own output stays, load chatter does not. -q sets *hush*, which on kernel 42 gates pr itself.

The launcher finds the vendored klambda/ next to the checkout, so bin/shen works from any cwd.

Embed

local shen = require("shen")
shen.boot{ quiet = true }
shen.eval('(define square X -> (* X X))')
print(shen.call("square", 9))         --> 81
local sq = shen.fn("square")          -- ordinary Lua callable
shen.typecheck("[1 2]", "(list number)")

shen.prims / shen.runtime expose F, the reader, and the printer.

Checked integers at external boundaries

LuaJIT's ordinary Shen numbers are IEEE-754 doubles. Beyond the contiguous exact integer range ±(2^53−1), distinct decimal literals can parse to the same number: 9007199254740993 and 9007199254740992 compare equal. The ordinary Shen number model remains unchanged for kernel compatibility.

For integer IDs, counters, or amounts that must not silently round, pass the original decimal string to the opt-in checked path:

local id = shen.checked_integer("9007199254740991") -- accepts ±(2^53−1)
local next_id = shen.checked_add(id, 0)
-- shen.checked_integer("9007199254740993") raises before tonumber can round

In Shen, use (lua.checked-integer "42") and lua.checked-add, lua.checked-sub, or lua.checked-mul for operations whose result must stay in range. They raise trappable Shen errors on invalid input or overflow. The bridge registers string --> number and curried number --> number --> number signatures for typechecked Shen call sites; the runtime checks enforce the narrower safe-integer range. Never pass an already-parsed numeric literal to lua.checked-integer: its original digits may already be lost. The returned value is an ordinary Shen number; subsequent ordinary arithmetic does not carry this guarantee. For values outside ±(2^53−1), retain decimal text or use a separate exact integer representation instead of converting to a Shen number.

From Shen: (lua.call "string.format" ["%s: %d" "answer" 42]). lua.function registers a Lua function as a typed Shen function so (tc +) can prove call sites. lua.map-new / lua.map-get / lua.map-put are hash maps keyed by Shen values under =, hashed in place over cons cells: the fast path for visited sets and memo tables. One gotcha: a plain {} returned from Lua is an empty array and comes back as (); use (lua.table-new) for a table you mean to keep. Details at the top of lua_interop.lua.

Examples

examples/hello_embed.lua	boot, define, call both ways
examples/family.shen	Prolog facts and queries
examples/config_check.lua	typed validation of Lua tables
examples/openresty/	guestbook: one rules.shen on OpenResty and in the browser

More under examples/. A full tour: demo/walkthrough.md.

Tests

make test                      # port specs (test/*_spec.lua)
luajit run-kernel-tests.lua    # official 43.1 suite → 154/154

The kernel suite is vendored in tests/. Port specs cover primitives, REPL, interop, tail-call lowering, boot caches, UTF-8 strings.

Install

luarocks install shen                       # launcher + modules
luarocks make --local shen-scm-1.rockspec   # this tree

LuaJIT required (lua == 5.1). This tree uses ShenOSKernel 43.1; release 0.11.1 uses Tarver's S42 kernel; 0.9.0 was 41.1. Or grab shen-bundle.lua from Releases — one file, require("shen-bundle").

luajit build/make-bundle.lua    # → build/shen-bundle.lua

Performance

workload	time
Kernel load, warm bytecode cache (in-process CPU; earlier run)	~0.03 s
Full CLI startup, warm (wall; 2026-09-27 A/B host)	0.119 s
Full CLI startup, cold (wall; same host)	1.383 s
42 suite, warm fasl (wall; same host)	4.592 s
Einstein solve in bench.lua (in-process CPU; same host)	0.0682 s

These measure different boundaries; the warm kernel-load number excludes standard-library loading and process startup. See the dated run and methodology and benchmark guide; times are host- and workload-dependent.

List builders of the [X | (f ...)] shape compile to a loop (tail recursion modulo cons), so they neither use a stack frame per element nor overflow on long lists; SHEN_TRMC=off restores plain recursion for A/B runs. To read LuaJIT trace logs, run luajit -jv bin/shen ....

Prolog and the typechecker run on a native engine (prolog_engine.lua); the portable kernel predicates that show up on compile and execution paths are overridden in prims.lua. Caches (kernel bytecode, stdlib signatures, user fasl) are content-keyed and safe to delete. The kernel bytecode and stdlib signatures live in ${XDG_CACHE_HOME:-$HOME/.cache}/shen-lua/ (SHEN_KERNEL_CACHE=<path> overrides, =off disables); user fasls in ~/.cache/shen-lua-fasl/ (SHEN_FASL_DIR, SHEN_FASL=off). Internals: doc/PERF-HANDOFF.md, doc/BENCHMARKS.md.

Requirements

LuaJIT 2.1. Kernel sources are in klambda/ (see klambda/PROVENANCE.md). SHEN_KL_DIR can point at another tree.

PUC Lua 5.1/5.4/5.5: same 143/143 (checked on 5.4.7). No FFI → legacy Prolog engine; no bit → caches off. Lua 5.3+ arithmetic is forced to floats so it matches LuaJIT.

Old LuaJIT on aarch64 (2.1.0-beta3): boot-time JIT crash, fixed upstream. Use a current rolling LuaJIT, or SHEN_JIT=off.

Nix

Optional. nix develop or direnv allow for a pinned toolchain. packages.toolchain is what Bifrost composes.