Skip to content

CI: Dependabot PRs fail on version-pinning tests and browser console noise, not on regressions #150

Closed

Description

@oto-macenauer-absa

Describe the bug

Dependabot PRs fail CI for reasons that have nothing to do with whether the knowledge base still works. Three of the seven open Dependabot PRs are red, and none of the failures points at a regression:

PR Bump Failing check Cause
#137 postcss 8.5.28 → 8.5.32 E2E package.json must pin postcss exactly as actions/ does
#110 htmlparser2 8.0.2 → 12.0.0 E2E package.json must pin htmlparser2 exactly as actions/ does
#149 dev-dependencies group (Playwright 1.65, Astro 7.4, …) E2E 6 Mermaid tests: Permissions policy violation: unload is not allowed in this document.
#149 — PR Requirements printf: write error: Broken pipe in check-pr-requirements v0.2.0 on a large PR body

1. Tests that pin versions instead of testing behaviour. tests/artifact-checks.spec.js asserts that the root package.json pins htmlparser2, postcss and mermaid at exactly the versions actions/package.json pins. Dependabot only watches /, so every parser bump is red by construction, and actions/ (which ships to every docs repo) is never bumped at all. Mermaid is in the Dependabot ignore list for the same reason. The intent of the test is "the build and the action parse the same way", but a version string is a proxy for that, not a check of it.

2. A console-error assertion that fails on browser noise. The Mermaid suite in tests/web-fragment.spec.js fails on any console.error. Newer Chromium (bundled with Playwright 1.65) reports unload listeners as a permissions-policy violation, and reframed (web-fragments 0.8.2) registers one on its iframe window: iframeWindow.addEventListener("unload", …). The diagrams still render, and the same tests assert that. Only the message is new.

3. PR Requirements SIGPIPE. check-pr-requirements v0.2.0 runs printf … | grep -q under pipefail, so a body bigger than the pipe buffer fails the check. Fixed upstream in v0.3.0 (#148). Merging #148 fixes this, so this issue does not cover it.

Area

CI / tests

Steps to Reproduce

  1. Bump postcss in the root package.json only (what Dependabot does).
  2. npm test: artifact-checks.spec.js fails although the build and the checker work.
  3. Run the Mermaid tests with Playwright ≥ 1.65: they fail on the unload console message although the diagram renders.

Expected state

  • A dependency bump fails CI only when it changes behaviour.
  • Parser parity is tested by behaviour. The checker self-test (actions/lib/check.selftest.js, which pins every rule's findings) also runs against the parsers the build resolves, as part of the root suite. A root htmlparser2/postcss that parses differently fails that test. One that does not is free to move. The version-equality assertions are removed.
  • The Mermaid fixture is tested by rendering. The E2E tests already prove the vendored bundle draws the diagram, so the version-equality assertion goes, and mermaid leaves the Dependabot ignore list.
  • Dependabot also watches /actions, so the dependencies we ship to docs repos get security and minor bumps too (grouped minor/patch, like the root).
  • Console-error assertions ignore known browser deprecation noise from third-party code, with the reason next to it. Uncaught page errors stay fatal, and so does every other console error.
  • Comments and CLAUDE.md no longer say the two package files must be pinned in step.

Impact / Severity

Medium. Every parser or Playwright bump needs manual work, actions/ dependencies drift unpatched, and a red Dependabot PR no longer means anything.

Related / References

#110, #137, #148, #149

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions