Skip to content

Add DomainWatcherAction for watched TLD alerts - #3286

Open
CydeWeys wants to merge 1 commit into
google:masterfrom
CydeWeys:domain-watcher-action
Open

CydeWeys wants to merge 1 commit into
google:masterfrom
CydeWeys:domain-watcher-action

Conversation

@CydeWeys

@CydeWeys CydeWeys commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Adds a Nomulus-native domain watcher action to monitor configured TLDs (such as closed/brand TLDs) for domain creations and deletions and send daily email notifications grouped by TLD.

Summary of Changes

  • Cursor & Schema: Added global CursorType.DOMAIN_WATCHER to Cursor.java and regenerated db-schema.sql.generated (no Flyway migration required since Cursor.type is an unconstrained text column).
  • Configuration: Added misc.domainWatcherTlds (defaults to [], null-safe, strips leading ., normalizes to canonical lowercase ASCII/punycode) to default-config.yaml, RegistryConfigSettings, and RegistryConfig.
  • Action (DomainWatcherAction):
    • Endpoint: POST /_dr/task/domainWatcher (Action.Service.BACKEND, Auth.AUTH_ADMIN).
    • Uses a 15-minute global LockHandler lock (logging a warning and returning HTTP 204 on contention) and a 15-minute cooldown buffer so in-flight EPP commits and Cloud SQL read-replica lag settle before the cursor advances.
    • First-run safe: initializes the cursor without sending historical spam.
    • Warns if any configured TLD is not present in Tlds.getTlds().
    • Queries replicaTm() (TRANSACTION_REPEATABLE_READ) with a per-section cap of 500 domains (plus truncation notice) for:
      1. Newly created domains (Domain.creationTime in window)
      2. Pending domain deletions (DomainHistory DOMAIN_DELETE with modificationTime in window and deletionTime >= endTime, alerting immediately on non-AGP deletes that drop DNS)
      3. Finalized domain deletions (Domain.deletionTime in window)
    • Sends plain-text email grouped by TLD via GmailClient to newAlertRecipientEmailAddress outside any DB transaction, intentionally advancing the cursor afterward (at-least-once delivery).
  • Wiring & Schedule: Registered in RequestComponent.java and routing.txt, and scheduled daily at 0 12 * * * UTC in cloud-scheduler-tasks-production.xml and cloud-scheduler-tasks-sandbox.xml via /_dr/cron/fanout?queue=retryable-cron-tasks.
  • Tests: Added comprehensive unit tests in DomainWatcherActionTest and RegistryConfigTest.

BUG=http://b/200681440


This change is Reviewable

@CydeWeys CydeWeys left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding DomainWatcherAction! The core design is clean: the half-open [startTime, endTime) window avoids boundary off-by-one duplicates/gaps, Cursor.type is unconstrained text (so no Flyway migration is needed), DB transaction boundaries keep GmailClient.sendEmail outside transactions, and the cron/fanout/routing wiring looks good.

Left inline comments below covering a few important operational/semantic items (P1) and smaller improvements (P2).

Additional notes (not on diff lines):

  • DB indexes (P2): "Domain" currently has single-column B-tree indexes on (tld), (creation_time), and (deletion_time), with no composite (tld, creation_time) or (tld, deletion_time) index. For a normal 24-hour window or small closed/brand TLDs Postgres can use the timestamp index or a BitmapAnd cleanly, though if a high-volume TLD is ever watched across a long outage window, consider bounding the query window or adding composite indexes.
  • Scheduler XMLs (P2): Omitting domainWatcher from alpha/crash/qa is consistent with exportDropList and exportDomainLists (production + sandbox only).

private static final String LOCK_NAME = "DomainWatcherAction";

// Cooldown buffer before now so in-flight EPP transactions commit before the cursor advances.
static final Duration COOLDOWN_DURATION = Duration.ofMinutes(5);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Replica lag > 5 minutes can silently drop events permanently. Because queryDomainChanges queries replicaTm() and then advances the cursor on the primary (tm()), any primary transaction committed before now - 5m that has not yet replicated to the read replica will be permanently skipped.

Consider increasing COOLDOWN_DURATION (e.g., 15–30 minutes, which is negligible for a daily 12:00 UTC report) and/or checking replica replay lag before advancing the cursor (or querying tm() if watched TLDs remain small).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — increased COOLDOWN_DURATION from 5 minutes to 15 minutes to absorb both in-flight EPP commits and Cloud SQL read-replica lag.

AND deletionTime >= :startTime
AND deletionTime < :endTime
ORDER BY tld ASC, deletionTime ASC, domainName ASC
""";

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Non-AGP deletes are delayed by 35 days, and restored domains are never reported. Only Add-Grace-Period deletes set deletionTime = now; standard deletes in DomainDeleteFlow set deletionTime = now + 30d RGP + 5d pendingDelete while immediately removing the domain from DNS, and DomainRestoreRequestFlow resets deletionTime = END_INSTANT.

If prompt alerting when a delete is initiated on closed/brand TLDs is desired, consider querying DomainHistory for DOMAIN_DELETE (and optionally DOMAIN_RESTORE) by modification time (or adding a pending-delete query). If reporting only finalized purges is intentional, please document this in the class Javadoc and clarify the email section heading (e.g., "Finalized domain deletions").

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added a DomainHistory (DOMAIN_DELETE, modificationTime in [startTime, endTime), resource.deletionTime >= endTime) query using existing history_modification_time and history_type indexes so non-AGP deletes alert immediately under "Pending domain deletions", while finalized purges report under "Finalized domain deletions". Documented all three categories in class Javadoc.

GmailClient gmailClient,
Response response,
@Config("domainWatcherTlds") ImmutableSet<String> watchedTlds,
@Config("newAlertRecipientEmailAddress") InternetAddress alertRecipientAddress) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Please confirm (per b/200681440) whether newAlertRecipientEmailAddress is the intended destination for domain watcher reports, or whether a dedicated config property (e.g., domainWatcherRecipientEmailAddress) would be preferable so watcher alerts can be routed independently of general operational alerts.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — reusing misc.newAlertRecipientEmailAddress (domain-registry-notify+prod@google.com in prod) is intentional so watcher reports follow the standard Nomulus alert routing.

// Return 204 so Cloud Tasks does not retry when another instance already holds the lock.
response.setStatus(SC_NO_CONTENT);
response.setPayload("Could not acquire lock; already running?");
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Consider shortening the lock lease from Duration.ofHours(1) to ~10–15 minutes (so a crashed pod doesn't lock out retries for an hour) and logging at WARNING when lock acquisition fails (LockHandlerImpl only logs at INFO).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — shortened the lock lease to 15 minutes (LOCK_LEASE_DURATION) and added WARNING-level logging on lock contention.

gmailClient.sendEmail(EmailMessage.create(subject, body, alertRecipientAddress));
}

tm().transact(() -> tm().put(Cursor.createGlobal(DOMAIN_WATCHER, searchEndTime)));

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Sending the email before advancing the cursor means a transient DB error here will cause Cloud Tasks to retry and send a duplicate email. That at-least-once trade-off (preferring a duplicate over a missed alert) is the right one for a watcher, but it's worth a brief inline comment documenting the intent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added an inline comment documenting the intentional at-least-once delivery trade-off.

domain.domainName(), domain.registrarId(), domain.creationTime()));
if (domain.deletionTime().isBefore(END_INSTANT)) {
body.append(String.format(", deleted: %s", domain.deletionTime()));
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: If a domain is created and put into PENDING_DELETE within the same window, deletionTime.isBefore(END_INSTANT) is true even though deletionTime is in the future (> endTime), which will print deleted: <future timestamp> under "Newly created domains". Consider labeling it scheduled deletion: %s when domain.deletionTime().isAfter(endTime).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — appendSection now labels deletionTime > endTime as scheduled deletion: <timestamp>.

@Config("domainWatcherTlds")
public static ImmutableSet<String> provideDomainWatcherTlds(RegistryConfigSettings config) {
return ImmutableSet.copyOf(config.misc.domainWatcherTlds);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Null-safety, TLD canonicalization, and config test coverage:

  • If an environment override YAML specifies domainWatcherTlds: with an empty/null value, YamlUtils.mergeMaps replaces default-config.yaml's [] with null, causing ImmutableSet.copyOf(config.misc.domainWatcherTlds) to throw NullPointerException. Please guard against null.
  • Domain.tld is stored in canonical lowercase punycode without a leading dot. Uppercase, leading-dot, or Unicode IDN entries in YAML will silently match 0 rows in WHERE tld IN :tlds. Recommend normalizing entries (stripping any leading . and passing through DomainNameUtils.canonicalizeHostname), documenting canonical ASCII/punycode in default-config.yaml, and optionally logging a warning in DomainWatcherAction if a configured TLD is not in Tlds.getTlds().
  • Please also add unit test coverage in RegistryConfigTest for provideDomainWatcherTlds (empty default, null safety, and normalization/punycode).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — guarded against null, stripped leading ., normalized entries via canonicalizeHostname(...), documented canonical ASCII/punycode in default-config.yaml, added a WARNING log for unknown TLDs in DomainWatcherAction, and added unit tests in RegistryConfigTest.


@Test
void testFirstRun_initializesCursorWithoutSendingEmail() {
persistResource(newDomain("existing.google"));

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: persistResource(newDomain("existing.google")) creates the domain at clock.now() (INITIAL_TIME), which is inside the 5-minute cooldown window (> INITIAL_TIME - 5m) and would have been excluded even if a cursor already existed (and would be picked up on the next run). Set creationTime before INITIAL_TIME.minus(COOLDOWN_DURATION) (e.g., -1d) to genuinely verify historical domains are skipped.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — updated testFirstRun_initializesCursorAndSkipsHistoricalDomains to set creationTime to INITIAL_TIME - 1d (well before cooldown).

newDomain("too-recent.google")
.asBuilder()
.setCreationTimeForTest(INITIAL_TIME.minus(Duration.ofMinutes(2)))
.build());

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Missing exact boundary and historical-deletion tests: too-recent.google is 3 minutes past searchEndTime, so it wouldn't catch < vs <= off-by-one bugs. Please add tests covering:

  • creationTime == searchStartTime (included) and creationTime == searchEndTime (excluded on run 1, included on run 2).
  • deletionTime == searchStartTime (included) and deletionTime == searchEndTime (excluded on run 1, included on run 2).
  • deletionTime < searchStartTime (previously deleted domains are not re-reported).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added testExactWindowBoundaries_halfOpenIntervalForCreationsAndDeletions covering == startTime, == endTime (excluded on run 1, included on run 2), and deletionTime < startTime.

.youtube (1):
- ephemeral.youtube (registrar: TheRegistrar, created: 2026-04-30T13:55:00Z, \
deleted: 2026-04-30T14:55:00Z)
""");

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 / P2: Grouping, ordering, and additional edge-case coverage:

  • P1: Each watched TLD currently has only 1 created and 1 deleted domain, persisted in alphabetical TLD order (google then youtube) matching watchedTlds, so this test would still pass if ORDER BY tld ASC, ... ASC, domainName ASC were broken. Add a test with watchedTlds = ImmutableSet.of("youtube", "google"), multiple domains per TLD persisted out of order, and at least two domains sharing the same timestamp to verify TLD grouping counts, timestamp ordering, and domainName ASC tie-breaking.
  • P2: Consider also covering unwatched TLD deletions, a transferred-then-deleted domain (verifying currentSponsorRegistrarId vs creationRegistrarId), searchEndTime < cursorTime (not just ==), a newly created domain with a future pending deletionTime, and asserting cursor advancement on the second run (L241–L252). Splitting this 104-line multi-scenario test into smaller focused tests would also improve failure diagnostics.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — split into focused tests covering out-of-order TLD grouping (youtube, google), timestamp and domainName ASC tie-breaking, pending deletions and scheduled deletion: formatting, transferred-then-deleted currentSponsorRegistrarId, unwatched TLD deletions, searchEndTime < cursorTime, truncation, and second-run cursor advancement.

@CydeWeys
CydeWeys force-pushed the domain-watcher-action branch from 43b5325 to 8201c2a Compare November 20, 2026 19:37
@CydeWeys
CydeWeys requested a review from weiminyu November 20, 2026 22:04
AND modificationTime < :endTime
AND resource.tld IN :tlds
AND resource.deletionTime >= :endTime
ORDER BY resource.tld ASC, modificationTime ASC, resource.domainName ASC

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the second sort is modification time, that's so fine-grained that there's no point in then also sorting by domain name, as it's extremely unlikely that two domain names will ever be deleted down to the exact same millisecond (or microsecond or whatever the granularity is here).

WHERE tld IN :tlds
AND creationTime.creationTime >= :startTime
AND creationTime.creationTime < :endTime
ORDER BY tld ASC, creationTime.creationTime ASC, domainName ASC

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment below -- if sorting by creation time second, then sorting by domain name third seems pointless.

WHERE tld IN :tlds
AND deletionTime >= :startTime
AND deletionTime < :endTime
ORDER BY tld ASC, deletionTime ASC, domainName ASC

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above re: sorting by domain name after deletion time.

Add a Nomulus-native domain watcher action to monitor configured TLDs
(such as closed/brand TLDs) for domain creations and deletions and send
daily email notifications grouped by TLD.

Changes:
- Add global `CursorType.DOMAIN_WATCHER` to `Cursor` and regenerate
  `db-schema.sql.generated` (no Flyway migration needed since
  `Cursor.type` is an unconstrained `text` column).
- Add `misc.domainWatcherTlds` (default empty list) to
  `default-config.yaml`, `RegistryConfigSettings`, and `RegistryConfig`
  (null-safe, strips leading dots, and normalizes to canonical lowercase
  ASCII/punycode).
- Implement `DomainWatcherAction` (`/_dr/task/domainWatcher`) using a
  15-minute global `LockHandler` lock (warning on contention, HTTP 204),
  a 15-minute cooldown buffer (absorbing in-flight EPP commits and read
  replica lag), first-run cursor initialization without historical spam,
  bounded read-only `replicaTm()` queries (capped at 500 domains per
  section with truncation notice) for newly created domains, pending
  deletions (`DomainHistory` `DOMAIN_DELETE`), and finalized deletions,
  and `GmailClient` delivery outside any DB transaction (advancing the
  cursor after email delivery for intentional at-least-once semantics).
- Wire `DomainWatcherAction` into `RequestComponent`, `routing.txt`, and
  daily (`0 12 * * *` UTC) Cloud Scheduler tasks in `production` and
  `sandbox` via `/_dr/cron/fanout?queue=retryable-cron-tasks`.
- Add unit tests in `DomainWatcherActionTest` and `RegistryConfigTest`
  covering config normalization/null-safety, empty config, lock
  contention, first-run historical skipping, exact half-open boundaries,
  TLD grouping and timestamp ordering, pending vs finalized deletions,
  transferred-then-deleted registrar attribution, truncation, and email
  failure retry safety.

BUG=http://b/200681440
@CydeWeys
CydeWeys force-pushed the domain-watcher-action branch from 8201c2a to 2bf8b69 Compare November 20, 2026 22:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant