Skip to content

fix(importer): add lookback window and pre-fetch check for REST imports - #6310

Draft
another-rex wants to merge 2 commits into
google:masterfrom
another-rex:fix-rest-import-lookback
Draft

another-rex wants to merge 2 commits into
google:masterfrom
another-rex:fix-rest-import-lookback

Conversation

@another-rex

Copy link
Copy Markdown
Contributor

Note

Stacked on top of #6306.

Summary

Incremental REST imports (handleImportREST) previously skipped any index entry with modified <= LastUpdated. Because LastUpdated advances to maxModified on each run, an advisory update could be permanently missed by the regular importer whenever:

  1. Upstream published advisories to the REST index out of modified timestamp order (e.g., an advisory generated slightly earlier than another entry that already advanced LastUpdated), or
  2. The index (all.json) already listed the new modified timestamp on run N, advancing LastUpdated, while the individual <id>.json endpoint briefly served the previous version during CDN/storage propagation.

Changes

  • Apply a rolling 12h lookback window (restImportLookbackWindow) against LastUpdated when scanning REST index entries in handleImportREST (while still persisting the true LastUpdated watermark).
  • Attach KnownID and KnownModified from the REST index entry onto WorkItem and check VulnerabilityStore.GetSourceModified at whole-second granularity before opening SourceRecord in processUpdate, avoiding redundant HTTP <id>.json fetches and Pub/Sub tasks for advisories within the lookback window that are already up to date.

agy

@another-rex
another-rex force-pushed the fix-rest-import-lookback branch from 8b4582b to db53e09 Compare November 4, 2026 23:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant