Skip to content

Engine source guide ​

The Rust library compiles a YL module graph into a normalized language and an executable image. Loading that image does not require the source compiler. The public Rust entry points remain documented in the Rust API.

Compilation ​

src/compiler.rs owns the public entry points and the working declaration tables. src/compiler/pipeline.rs gives the order of compilation phases. Phase modules share that per-compilation state; they do not expose another public host interface.

ResponsibilitySource modules
Load graphs and canonical bundled modulescompiler/loading.rs
Register declarations and resolve imports, scopes, and memberscompiler/imports.rs, compiler/resolution.rs
Validate source declarations before expansioncompiler/declarations.rs
Lower concrete grammar, abstract families, and precedencecompiler/grammar.rs, compiler/rules.rs
Expand parameterized grammar, pipe rewrites, and argumentscompiler/rewrites.rs, compiler/arguments.rs, compiler/substitution.rs
Expand authored operations without losing argument effects or call-site identitiescompiler/wrappers.rs
Compile recursive query functions, expressions, actions, and bindingscompiler/query_functions.rs, compiler/query_expressions.rs, compiler/query_actions.rs, compiler/query_arguments.rs
Lower native contracts and scheduling groupscompiler/native_semantics.rs, compiler/semantic_groups.rs, compiler/semantic_definitions.rs
Compile syntax checks and validate conditionscompiler/checks.rs, compiler/condition_validation.rs
Compile unevaluated backend hook argumentscompiler/backend.rs
Preserve highlighting and example templatescompiler/tooling.rs
Validate normalized artifacts, captures, projections, and value shapescompiler/validation.rs, compiler/capture_validation.rs, compiler/capture_shapes.rs, compiler/term_edges.rs

Module IDs, declaration paths, and source spans keep their meaning throughout these phases. Ordered maps preserve deterministic traversal and diagnostic order. GrammarArgument and GrammarBindings describe compile-time grammar arguments; they are distinct from semantic values and host-owned backend values.

The final stage validates the normalized language and compiles image/ tables. See compiled artifacts for the serialized contract. Internal module organization is independent of that artifact format.

YL parsing ​

src/frontend.rs exposes full-file and embedded-expression parsing. Its parser owns one token cursor and a depth budget. The files under src/frontend/ separate declarations, grammar precedence, semantic expressions/actions, structural rewrites, and syntax checks. documentation.rs attaches documentation after parsing, so comments do not become grammar expressions. diagnostic_imports.rs recognizes explicit diagnostic imports before declaration parsing, including forward imports and aliases.

Target-language parsing ​

src/runtime.rs defines AST results and branch-local match state. Its private modules execute borrowed parser-image instructions:

ModuleResponsibility
runtime/session.rsSet up a source parse and publish AST, diagnostics, occurrences, and highlights
runtime/matching.rsMatch rules, regex terminals, trivia, and backtracking continuations
runtime/instructions.rsExecute indexed grammar instructions and structural projections
runtime/precedence.rsExecute recursive abstract families with Pratt binding powers
runtime/occurrences.rsConstruct AST nodes and source-linked semantic occurrences
runtime/checks.rsEvaluate syntax-local checks
runtime/highlighting.rsApply contextual highlighting without leaking failed alternatives

Every attempted match owns its captures, diagnostics, projection markers, and occurrences. Backtracking discards the attempted match as a whole. Shared executable tables stay immutable; per-source bookkeeping belongs to the session.

Tests and changes ​

Compiler and frontend unit tests live beside their domain modules. Public API, artifact reload, negative fixture, and golden tests remain under tests/. Move a rule to its owning phase rather than adding another responsibility to the facade. Preserve existing diagnostics and snapshots when reorganizing implementation.

Run the Rust gates:

sh
cargo test --all-targets
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

For a complete host example, run cargo run --example 14-backend-hooks; the backend tutorial also executes the generated JavaScript and WebAssembly outputs.

Declarative languages. Reusable tools.