Add an offline law index and archive-parsing commands over the open data packages #17

Summary

Now that Get-LovdataPublicDataset / Save-LovdataPublicDataset (PR #16, fixes #9) can download the open data packages, the natural next layer is turning the downloaded archives into something queryable: an offline law index for fast lookup, plus commands that parse the XML documents inside the archives into structured objects. This stays entirely on the key-free surface — no account or key needed.

These findings come from the Storhaug-ting/S62 session (PR Storhaug-ting/S62#2), which built a working generator against gjeldende-lover.tar.bz2 and worked out the archive internals. Capturing them here so the implementation in this module doesn't have to rediscover them.

Proposed capability

• Find-LovdataLaw <term> — resolve a name/short-title/legacy id to a law id (e.g. Find-LovdataLaw veglova -> 1963-06-21-23) without a network round-trip. Build it the same way this org's font modules ship FontsData.json: regex legacyID / title / titleShort out of the first ~4 KB of each XML file and check in the result as a small JSON index (S62 reports ~163 KB for 759 laws, ~66 ms lookups, index build ~2 s). See the GoogleFonts / NerdFonts FontsData.json pattern for prior art.
• A parse command that turns an archive XML document into a structured object (metadata + document body).

Archive internals (verified by S62)

• gjeldende-lover.tar.bz2 extracts to nl/ with 759 XML files (~50.9 MB unpacked). gjeldende-sentrale-forskrifter uses a different subfolder — do not hardcode nl.
• Filename convention: nl-YYYYMMDD-NNN.xml. Laws with no running number use 000 (e.g. servituttlova LOV-1968-11-29 -> nl-19681129-000.xml).
• Files are XHTML-compatible and parse directly with [System.Xml.XmlDocument] after stripping the leading <!DOCTYPE html>. (The public web pages do not — they contain undeclared entities like  . A concrete reason to prefer the API/packages over scraping.)

Metadata keys — use the class attribute, not the visible label

Metadata sits in dl.data-document-key-info as dt/dd pairs. The visible label varies by target form (I kraft frå / I kraft fra); the class name is stable across all laws. Key on the class:

<dt class="dateInForce">I kraft frå</dt><dd class="dateInForce">1967-01-01</dd>

Stable class keys: legacyID, dokid, ministry, dateInForce, lastChangeInForce, lastChangedBy, legalArea, lastupdated, titleShort, title, miscInformation, refid, changesToDocuments, dateOfPublication, table-of-contents. ministry and legalArea may hold multiple values (<li> lists). legacyID is LOV-<id>, so the law id is the value with the prefix removed.

Document structure

Semantic and compact: main.documentBody > section.section (chapter, may nest) > article.legalArticle (paragraph) whose h3|h4.legalArticleHeader contains span.legalArticleValue (e.g. § 1-1) and span.legalArticleTitle. Clauses are article.legalP or article.numberedLegalP (the latter carries (1) inline in the text). Lists are ol|ul.defaultList with the marker in li[data-name]. Change notes are article.changesToParent; footnotes are footer.footnotes > article.footnote with span.footnoteLabel.

Design note — bundle metadata only, not law text

S62 deliberately bundled only metadata, not the statute text: Lovdata refreshes nightly while module releases do not, so checked-in law text goes stale between releases. A stale font list is harmless; a law excerpt that looks authoritative but cites a repealed provision is not. If we ever consider shipping data, keep it to the index/metadata.

Gotcha — .gitattributes for generated data

Any generated/checked-in data file should be marked text eol=lf in .gitattributes. With core.autocrlf=true the generator writes LF while git puts CRLF in the working copy, so a -Verify self-check reports a mismatch in a fresh clone even though the content is identical.

Reference implementation

Working code in Veiforeningen/scripts/kilder/Lovdata.psm1 (Storhaug-ting/S62 PR #2); it is slated to move into this module / Storhaug-ting/Kilden.

Notes

• Keep this on the key-free surface; the authenticated document/content endpoints remain deferred to Add the authenticated Lovdata API surface in a later release #15 / Add document metadata and content retrieval commands #6.
• Likely wants default display views (Add default display views for the Lovdata object types #11) once the object types exist.

Scope boundary (clarified with Storhaug-ting/S62)

The reference implementation currently merges two concerns; when it moves, it should split along this line so nobody builds it twice:

Markdown conversion does not belong in this module. Kilden already owns that convention (kilde.psd1 with source URL/checksum/conversion rules, the original file, the generated markdown, and an Update-Source.ps1 that verifies in CI). This module returns objects; Kilden renders them. So the parse command here should emit structured objects only — no markdown.

Index command refinements

• Naming: prefer -Refresh over -Force for the regenerate-from-a-fresh-package switch. In PowerShell -Force implies "override a safeguard", whereas this is just "fetch again".
• S62 has this running in practice: checked-in JSON by default (~66 ms lookup, no download), rebuild from a fresh package on -Refresh (~40 s: 5.6 MB + unpack).
• Why a checked-in index is safe here (the failure mode is benign): the index holds only id, short title, and title, fields that almost never change. Staleness means a brand-new law is missing, i.e. a miss you notice immediately, which is qualitatively different from stale statute text that fails silently while looking authoritative.

Implementation caveats to carry over

• ConvertTo-LovdataFilename deserves its own test. Format is nl-{date:yyyyMMdd}-{nr:000}.xml; the zero-padding on both fields is easy to get wrong. Boundary cases: 1968-11-29 (no number -> 000), 2013-06-21-100 (three-digit number), 1966-12-09-1 (single digit -> 001).
• section.section nests (e.g. jordskiftelova has sub-chapters), so chapter level must be derived from actual nesting depth or from the h2/h3/h4 tag, never assumed flat. This was the only structural surprise across the laws S62 reviewed.

Default display views (relates to #11)

The lookup objects have natural default columns: LovId, Korttittel, Tittel. Tittel is long enough to break table rendering, so it should be omitted from the default table view.

Regression fixtures available (offered by Storhaug-ting/S62)

S62 has nine current laws that regenerate byte-identical and are verified from a fresh clone, so the parse layer can be tested against a known-good corpus without building the expected output first. jordskiftelova is the strongest single test case: nested section.section, 132 articles, and both footnotes and change notes. Coordinate with the S62 session (Storhaug-ting/S62 PR #2) when the parse command lands.