feat(core): language lexicons on profiles (M19a)

METAMODEL.md

@@ -384,6 +384,7 @@ implemented* (robustness test of the schema).
 | `kinds` | `Record<kind, {required: TraitName[], optional: TraitName[]}>` | the licit trait compositions per entity kind |
 | `edges` | EdgeKind[] | edge kinds this language can emit |
 | `notes` | string[] | documented static-analysis blind spots (reflection, `Class.forName`, dynamic `require`, macros pre-expansion, …) |
+| `lexicon` | `{keywords, conventional, technical?, fileKinds?, interfacePrefix?}` | the language's words, for the vocabulary (§5.3): reserved words, names the language dictates, plumbing words, kinds whose name is a file path, C#'s interface marker |
 
 **Validation** of an entity against its profile:
 1. `kind` exists in the profile;
@@ -421,6 +422,31 @@ keyed on; and the verdict is per profile, since two profiles may name the same
 kind under different rules. It is a **reader-side** optimisation and requires no
 encoding support — trait sets are deliberately not a wire-level concept.
 
+### 5.3 The lexicon: a language's words
+
+The vocabulary (PLAN.md §20) splits every name into words — `OrderManagerRepository`
+is `order`, `manager`, `repository` — and tags each one. Splitting is
+**universal**: every character that is not a letter, a mark or a digit ends a
+word, and so does a camel-case boundary, so sigils, separators and suffixes
+(`@total`, `valid?`, `Order::Line`) need no data. What a language adds is the
+lexicon, optional on a profile and present on every shipped one:
+
+| Key | Meaning |
+|---|---|
+| `keywords` | the specification's **reserved** words, as the language spells them. A contextual keyword (`record`, `var`, TypeScript's `type`) is a legal identifier, often a domain word, and is not listed |
+| `conventional` | whole identifiers the language or its runtime dictates — overrides, protocol members, operators (`toString`, `__init__`, `<=>`). Their words say nothing about the domain, so such a name contributes none |
+| `technical` | plumbing words of the language beyond the vocabulary's shared list, spelled as the splitter yields them (lowercase letters and digits) |
+| `fileKinds` | kinds whose name is a file path (TypeScript's `module`, Ruby's `file`): the extension is not a word |
+| `interfacePrefix` | `true` in C# alone: `IOrderService` names an interface `OrderService` |
+
+`validateProfile` rejects an entry that could never do what its list says:
+listed twice (`lexicon-duplicate`); a keyword with no letter or a name holding
+whitespace, a technical entry the splitter cannot yield (`lexicon-not-a-word`);
+a conventional name or technical word a keyword already claims
+(`lexicon-keyword-overlap`); a file kind the profile does not define
+(`lexicon-unknown-kind`). The lexicon is not on the wire: `schemas/` is
+unchanged by it.
+
 ---
 
 ## 6. Stub

PLAN.md

@@ -4871,9 +4871,10 @@ Seven principles, locked up front:
    (constructor, lambda) contributes none and is counted. Coverage per
    extractor is a table in this section, filled by the milestone.
 2. **The idiom is the profile's; the algorithm is one.** Keywords,
-   conventional member names, affixes and separators are DATA on the
-   profile (`lexicon`), specifiable without being implemented (invariant 8).
-   The splitter reads them and knows no language.
+   conventional names, technical words and file kinds are DATA on the
+   profile (`lexicon`), specifiable without being implemented (invariant 8);
+   word boundaries are universal and need none. The splitter reads the data
+   and knows no language.
 3. **Classification, not filtering.** Every token is kept and tagged
    `domain | technical | keyword | short`; a consumer filters. Drift needs
    the technical words too, and "not shown" must stay distinguishable from
@@ -4902,38 +4903,54 @@ the universal rules below — present on the eleven shipped:
 
 ```ts
 lexicon?: {
-  keywords: string[];       // reserved words, as the language spells them
-  conventional: string[];   // whole member names the language or its runtime dictates;
+  keywords: string[];       // the specification's RESERVED words, as the language spells them;
+                            // a contextual keyword (record, var, TS's type) is an identifier
+  conventional: string[];   // whole identifiers the language or its runtime dictates;
                             // skipped whole: toString/equals/hashCode/main (Java),
                             // initialize/to_s/inspect/<=> (Ruby), constructor (TS),
                             // __using__/start_link/init/handle_call (Elixir),
                             // Equals/GetHashCode/ToString/Dispose/Main (C#)
-  technical?: string[];     // language-specific noise beyond the shared list:
-                            // impl, bean (Java); concern (Ruby); live, web (Elixir)
-  idioms: {
-    separators: string[];   // "_", "-", ".", "::", "/", "$"
-    prefixes: string[];     // sigils stripped before splitting: "@@", "@", "$", "#", "_"
-    suffixes: string[];     // "?", "!", "=" (Ruby/Elixir), "`N" (C# generic arity)
-    interfacePrefix?: true; // C#: `IOrderService` → order, service (I + Upper + lower)
-    fileModules?: true;     // the module is the file: strip the extension, split the path
-  };
+  technical?: string[];     // language-specific plumbing beyond the shared list, as the
+                            // splitter yields it: bean, jdbc (Java); concern (Ruby); live, web (Elixir)
+  fileKinds?: string[];     // kinds whose name is a file path: TS/JS `module`, Ruby and Elixir
+                            // `file`, PHP `codeFile` — the extension is not a word
+  interfacePrefix?: true;   // C#: `IOrderService` → order, service (I + Upper + lower)
 };
 ```
 
-`ProfileSchema` mirrors it; `codegraph profiles --lang X` prints it in full
-(the command's completeness rule); `schemas/` is unchanged — a profile is not
-on the wire. Tests pin: every shipped lexicon has a non-empty keyword list,
-its entries are distinct, and nothing in `conventional` is also a keyword.
-
-**The split** (`split.ts`): one function, `splitIdentifier(name, lexicon)`,
-in six steps, each pinned by a table of hand cases per idiom:
-
-1. strip one leading prefix and one trailing suffix from the lexicon
-   (`@total` → `total`, `valid?` → `valid`, `save!`, `name=`,
-   `Repository\`1` → `Repository`), then the interface prefix when the flag
-   is set and the rest starts Upper + lower (`IOrderService`, not `IO`);
-2. split on the lexicon's separators; for `fileModules`, drop the
-   extension first (`legacy/acme-report.ts` → `legacy`, `acme`, `report`);
+The first draft carried `idioms.separators`, `prefixes` and `suffixes`; built
+(M19a), they could change no output. The splitter must break on EVERY
+character that is not a letter, a mark or a digit anyway — or `<=>`,
+`[Symbol.iterator]` and `this[]` become tokens — and once it does, `@total`,
+`valid?`, `Order::Line` and `` Repository`1 `` need no data. The draft's
+`fileModules` flag was per PROFILE; file-ness is per KIND (Elixir's `file` is
+a path, its `module` the dotted `AcmeOrder.Registry`, whose `.Registry` a
+profile-wide flag would drop as an extension), so it became `fileKinds`.
+
+`ProfileSchema` mirrors it (a test parses every shipped profile through it
+unchanged); `codegraph profiles --lang X` prints it in full, between the kinds
+and the notes (the command's completeness rule); `schemas/` is unchanged — a
+profile is not on the wire. `validateProfile` rejects an entry that could
+never do what its list says, with four codes: `lexicon-duplicate`,
+`lexicon-not-a-word` (a keyword with no letter, a name holding whitespace, a
+technical entry the splitter cannot yield), `lexicon-keyword-overlap` (a
+conventional name or a technical word a keyword claims first) and
+`lexicon-unknown-kind`. Tests pin, per shipped profile: a non-empty keyword
+list, no contextual keyword listed, `interfacePrefix` in C# alone, and exactly
+the five file kinds above. TypeScript reserves exactly ECMAScript's words, so
+the two profiles read one `ECMASCRIPT_LEXICON`.
+
+**The split** (`split.ts`): one function, `splitIdentifier(name, kind,
+lexicon)`, in six steps, each pinned by a table of hand cases per idiom:
+
+1. for a kind in `fileKinds`, drop the extension of the last path segment
+   (`legacy/acme-report.ts` → `legacy/acme-report`); then the interface
+   prefix when the flag is set and the name starts `I` + Upper + lower
+   (`IOrderService`, not `IO`);
+2. split on every character that is not a Unicode letter, mark or digit —
+   universal, no data: `@total` → `total`, `valid?` → `valid`, `save!`,
+   `name=`, `Order::Line`, `max_lines`, `acme-report`, `Repository\`1` →
+   `repository`, `1`;
 3. camel-case boundaries with the acronym rule — lower→Upper, and an
    Upper run followed by Upper + lower splits before its last letter
    (`HTTPServer` → `http`, `server`; `getURLString` → `get`, `url`,
@@ -5234,10 +5251,10 @@ never looks like a fact.
 
 **M19a — the lexicon, the split, the artifact** (closes the profile half of #19)
 
-- [ ] `Profile.lexicon` in core: type, `ProfileSchema`, METAMODEL §5 row,
+- [x] `Profile.lexicon` in core: type, `ProfileSchema`, METAMODEL §5 row,
       the eleven profiles filled (keywords from each language's
       specification; conventional names from its runtime), `codegraph
-      profiles --lang` printing it; the three lexicon tests; `schemas/`
+      profiles --lang` printing it; the lexicon's validation codes and tests; `schemas/`
       unchanged, proven by the drift check.
 - [ ] `packages/vocabulary/` (`@codegraph/vocabulary`, depends on core and
       analyzer only): `split.ts`, `classify.ts` with `DEFAULT_TECHNICAL`,

docs/cli.md

@@ -314,7 +314,10 @@ facts.
 
 ### `profiles`
 Prints the language profiles core ships — the per-kind required and optional
-trait sets an extractor must respect.
+trait sets an extractor must respect, the edge kinds, the lexicon the
+vocabulary reads (reserved words, conventional names, technical words, file
+kinds), and the notes. `--lang L` prints one profile in full; `--json` prints
+the data as `{kind: "codegraph.profiles/1", profiles[]}`.
 
 ## The Java extractor
 

packages/cli/src/commands/profiles.ts

@@ -1,5 +1,5 @@
 import { compareIds, toJsonString } from "@codegraph/analyzer";
-import { PROFILES, getProfile, type Profile } from "@codegraph/core";
+import { PROFILES, getProfile, type Lexicon, type Profile } from "@codegraph/core";
 import type { ProfilesOptions } from "../args.js";
 import { EXIT, UsageError, type ExitCode } from "../exit.js";
 import { errLine, outLines, type IoSink } from "../io.js";
@@ -11,7 +11,8 @@ import { errLine, outLines, type IoSink } from "../io.js";
  *  - Read `PROFILES` / `getProfile(lang)` from `@codegraph/core`. Profiles are
  *    DATA (CLAUDE.md invariant 8): print them, never synthesize one.
  *  - No `--lang`: list every shipped profile. With `--lang`: print that
- *    profile's kinds (required/optional traits), edge kinds, spaces and notes.
+ *    profile's kinds (required/optional traits), edge kinds, spaces, lexicon
+ *    and notes.
  *  - An unknown `--lang` is a USAGE error (exit 2) naming the languages that do
  *    exist — no model was involved, so it cannot be a finding.
  *  - `--json` prints the profile data on stdout; deterministic key and array
@@ -135,7 +136,7 @@ function summaryLines(profiles: readonly Profile[]): readonly string[] {
 /**
  * The full spec for one language: every kind with its licit trait composition,
  * the licensed edge kinds, the declaration spaces where the profile declares
- * any, and the notes in full.
+ * any, the lexicon, and the notes in full.
  *
  * Kinds are printed in sorted order so the same profile always renders the same
  * bytes and two languages can be diffed against each other; trait lists print
@@ -168,6 +169,9 @@ function specLines(profile: Profile): readonly string[] {
     if (spaces !== undefined) lines.push(`    space     ${list([...spaces])}`);
   }
 
+  lines.push("");
+  lines.push(...lexiconLines(profile.lexicon));
+
   lines.push("");
   lines.push("notes:");
   if (noteCount === 0) {
@@ -179,11 +183,31 @@ function specLines(profile: Profile): readonly string[] {
   return lines;
 }
 
+/**
+ * The words the vocabulary splitter reads (PLAN.md §20.1), every entry, in the
+ * order the profile declares it: a reader asking why `newOrder` gave no `new`
+ * looks for `new` here. Between the kinds and the notes, because the notes stay
+ * last — a reader of this text (the website's page) takes all that follows
+ * `notes:` as notes. The optional rows print only where the profile declares them.
+ */
+function lexiconLines(lexicon: Lexicon | undefined): readonly string[] {
+  if (lexicon === undefined) return ["lexicon:", "  (none)"];
+  const lines = [
+    "lexicon:",
+    `  keywords          ${list([...lexicon.keywords])}`,
+    `  conventional      ${list([...lexicon.conventional])}`,
+  ];
+  if (lexicon.technical !== undefined) lines.push(`  technical         ${list([...lexicon.technical])}`);
+  if (lexicon.fileKinds !== undefined) lines.push(`  file kinds        ${list([...lexicon.fileKinds])}`);
+  if (lexicon.interfacePrefix === true) lines.push("  interface prefix  yes");
+  return lines;
+}
+
 /**
  * The profile as JSON: the same data, with a canonical key and kind order so
  * that identical inputs produce byte-identical stdout (decision 6). Nothing is
- * added, renamed or defaulted — `space` and `notes` stay absent where the
- * profile omits them, because their absence is itself profile information.
+ * added, renamed or defaulted — `space`, `notes` and `lexicon` stay absent where
+ * the profile omits them, because their absence is itself profile information.
  */
 function canonicalProfile(profile: Profile): Record<string, unknown> {
   const kinds: Record<string, unknown> = {};
@@ -208,6 +232,19 @@ function canonicalProfile(profile: Profile): Record<string, unknown> {
     out["space"] = space;
   }
   if (profile.notes !== undefined) out["notes"] = profile.notes;
+  if (profile.lexicon !== undefined) out["lexicon"] = canonicalLexicon(profile.lexicon);
+
+  return out;
+}
 
+/** The lexicon's keys in one fixed order; an absent key stays absent, never `[]` or `false`. */
+function canonicalLexicon(lexicon: Lexicon): Record<string, unknown> {
+  const out: Record<string, unknown> = {
+    keywords: lexicon.keywords,
+    conventional: lexicon.conventional,
+  };
+  if (lexicon.technical !== undefined) out["technical"] = lexicon.technical;
+  if (lexicon.fileKinds !== undefined) out["fileKinds"] = lexicon.fileKinds;
+  if (lexicon.interfacePrefix !== undefined) out["interfacePrefix"] = lexicon.interfacePrefix;
   return out;
 }

packages/cli/test/profiles.test.ts

@@ -100,6 +100,48 @@ describe("one language's full spec", () => {
     );
   });
 
+  /**
+   * The lexicon is what the vocabulary splitter reads (PLAN.md §20.1): someone
+   * asking why `newOrder` gave no `new` reads the keyword list here, so every
+   * entry must be printed, unwrapped, in the order the profile declares it.
+   */
+  it.each(LANGS)("prints %s's lexicon in full, line for line", (lang) => {
+    const lexicon = profileOf(lang).lexicon;
+    if (lexicon === undefined) throw new Error(`${lang} ships no lexicon`);
+    const lines = invoke(["profiles", "--lang", lang]).io.stdoutLines();
+    const start = lines.indexOf("lexicon:");
+    const expected = [
+      "lexicon:",
+      `  keywords          ${lexicon.keywords.join(", ")}`,
+      `  conventional      ${lexicon.conventional.length === 0 ? "(none)" : lexicon.conventional.join(", ")}`,
+      ...(lexicon.technical === undefined ? [] : [`  technical         ${lexicon.technical.join(", ")}`]),
+      ...(lexicon.fileKinds === undefined ? [] : [`  file kinds        ${lexicon.fileKinds.join(", ")}`]),
+      ...(lexicon.interfacePrefix === true ? ["  interface prefix  yes"] : []),
+    ];
+    expect(start, "no lexicon section").toBeGreaterThan(0);
+    expect(lines.slice(start, start + expected.length)).toEqual(expected);
+    // The block ends there: a blank line, then the next section.
+    expect(lines[start + expected.length]).toBe("");
+  });
+
+  // The notes stay last: they are the one section whose entries run to several
+  // sentences, and a reader of the text form (the website's page) takes
+  // everything after `notes:` as notes.
+  it("prints the lexicon after the kinds and before the notes", () => {
+    for (const lang of LANGS) {
+      const lines = invoke(["profiles", "--lang", lang]).io.stdoutLines();
+      expect(lines.indexOf("lexicon:"), lang).toBeGreaterThan(lines.indexOf("kinds:"));
+      expect(lines.indexOf("lexicon:"), lang).toBeLessThan(lines.indexOf("notes:"));
+    }
+  });
+
+  it("prints the interface prefix and the file kinds only where a profile declares them", () => {
+    expect(invoke(["profiles", "--lang", "csharp"]).io.stdout()).toContain("\n  interface prefix  yes\n");
+    expect(invoke(["profiles", "--lang", "java"]).io.stdout()).not.toContain("interface prefix");
+    expect(invoke(["profiles", "--lang", "ts"]).io.stdout()).toContain("\n  file kinds        module\n");
+    expect(invoke(["profiles", "--lang", "java"]).io.stdout()).not.toContain("file kinds");
+  });
+
   it("prints the declaration spaces exactly where a profile declares them", () => {
     const ts = invoke(["profiles", "--lang", "ts"]).io.stdout();
     expect(ts).toContain("space");
@@ -145,6 +187,15 @@ describe("--json prints the profile data as-is", () => {
     );
   });
 
+  it("writes the lexicon's keys in one fixed order, absent keys absent", () => {
+    const keysOf = (lang: string): readonly string[] => {
+      const [json] = payload(["profiles", "--lang", lang]) as readonly Profile[];
+      return Object.keys(json?.lexicon ?? {});
+    };
+    expect(keysOf("csharp")).toEqual(["keywords", "conventional", "technical", "interfacePrefix"]);
+    expect(keysOf("ts")).toEqual(["keywords", "conventional", "technical", "fileKinds"]);
+  });
+
   it("says the same thing as the text form (decision 8)", () => {
     const text = invoke(["profiles", "--lang", "go"]).io.stdout();
     const [json] = payload(["profiles", "--lang", "go"]) as readonly Profile[];