Repository navigation
Conversation
CydeWeys
left a comment
There was a problem hiding this comment.
Thanks for adding DomainWatcherAction! The core design is clean: the half-open [startTime, endTime) window avoids boundary off-by-one duplicates/gaps, Cursor.type is unconstrained text (so no Flyway migration is needed), DB transaction boundaries keep GmailClient.sendEmail outside transactions, and the cron/fanout/routing wiring looks good.
Left inline comments below covering a few important operational/semantic items (P1) and smaller improvements (P2).
Additional notes (not on diff lines):
- DB indexes (P2):
"Domain"currently has single-column B-tree indexes on(tld),(creation_time), and(deletion_time), with no composite(tld, creation_time)or(tld, deletion_time)index. For a normal 24-hour window or small closed/brand TLDs Postgres can use the timestamp index or a BitmapAnd cleanly, though if a high-volume TLD is ever watched across a long outage window, consider bounding the query window or adding composite indexes. - Scheduler XMLs (P2): Omitting
domainWatcherfromalpha/crash/qais consistent withexportDropListandexportDomainLists(production + sandbox only).
| private static final String LOCK_NAME = "DomainWatcherAction"; | ||
|
|
||
| // Cooldown buffer before now so in-flight EPP transactions commit before the cursor advances. | ||
| static final Duration COOLDOWN_DURATION = Duration.ofMinutes(5); |
There was a problem hiding this comment.
P1: Replica lag > 5 minutes can silently drop events permanently. Because queryDomainChanges queries replicaTm() and then advances the cursor on the primary (tm()), any primary transaction committed before now - 5m that has not yet replicated to the read replica will be permanently skipped.
Consider increasing COOLDOWN_DURATION (e.g., 15–30 minutes, which is negligible for a daily 12:00 UTC report) and/or checking replica replay lag before advancing the cursor (or querying tm() if watched TLDs remain small).
There was a problem hiding this comment.
Done — increased COOLDOWN_DURATION from 5 minutes to 15 minutes to absorb both in-flight EPP commits and Cloud SQL read-replica lag.
| AND deletionTime >= :startTime | ||
| AND deletionTime < :endTime | ||
| ORDER BY tld ASC, deletionTime ASC, domainName ASC | ||
| """; |
There was a problem hiding this comment.
P1: Non-AGP deletes are delayed by 35 days, and restored domains are never reported. Only Add-Grace-Period deletes set deletionTime = now; standard deletes in DomainDeleteFlow set deletionTime = now + 30d RGP + 5d pendingDelete while immediately removing the domain from DNS, and DomainRestoreRequestFlow resets deletionTime = END_INSTANT.
If prompt alerting when a delete is initiated on closed/brand TLDs is desired, consider querying DomainHistory for DOMAIN_DELETE (and optionally DOMAIN_RESTORE) by modification time (or adding a pending-delete query). If reporting only finalized purges is intentional, please document this in the class Javadoc and clarify the email section heading (e.g., "Finalized domain deletions").
There was a problem hiding this comment.
Done — added a DomainHistory (DOMAIN_DELETE, modificationTime in [startTime, endTime), resource.deletionTime >= endTime) query using existing history_modification_time and history_type indexes so non-AGP deletes alert immediately under "Pending domain deletions", while finalized purges report under "Finalized domain deletions". Documented all three categories in class Javadoc.
| GmailClient gmailClient, | ||
| Response response, | ||
| @Config("domainWatcherTlds") ImmutableSet<String> watchedTlds, | ||
| @Config("newAlertRecipientEmailAddress") InternetAddress alertRecipientAddress) { |
There was a problem hiding this comment.
P1: Please confirm (per b/200681440) whether newAlertRecipientEmailAddress is the intended destination for domain watcher reports, or whether a dedicated config property (e.g., domainWatcherRecipientEmailAddress) would be preferable so watcher alerts can be routed independently of general operational alerts.
There was a problem hiding this comment.
Confirmed — reusing misc.newAlertRecipientEmailAddress (domain-registry-notify+prod@google.com in prod) is intentional so watcher reports follow the standard Nomulus alert routing.
| // Return 204 so Cloud Tasks does not retry when another instance already holds the lock. | ||
| response.setStatus(SC_NO_CONTENT); | ||
| response.setPayload("Could not acquire lock; already running?"); | ||
| } |
There was a problem hiding this comment.
P2: Consider shortening the lock lease from Duration.ofHours(1) to ~10–15 minutes (so a crashed pod doesn't lock out retries for an hour) and logging at WARNING when lock acquisition fails (LockHandlerImpl only logs at INFO).
There was a problem hiding this comment.
Done — shortened the lock lease to 15 minutes (LOCK_LEASE_DURATION) and added WARNING-level logging on lock contention.
| gmailClient.sendEmail(EmailMessage.create(subject, body, alertRecipientAddress)); | ||
| } | ||
|
|
||
| tm().transact(() -> tm().put(Cursor.createGlobal(DOMAIN_WATCHER, searchEndTime))); |
There was a problem hiding this comment.
P2: Sending the email before advancing the cursor means a transient DB error here will cause Cloud Tasks to retry and send a duplicate email. That at-least-once trade-off (preferring a duplicate over a missed alert) is the right one for a watcher, but it's worth a brief inline comment documenting the intent.
There was a problem hiding this comment.
Done — added an inline comment documenting the intentional at-least-once delivery trade-off.
| domain.domainName(), domain.registrarId(), domain.creationTime())); | ||
| if (domain.deletionTime().isBefore(END_INSTANT)) { | ||
| body.append(String.format(", deleted: %s", domain.deletionTime())); | ||
| } |
There was a problem hiding this comment.
P2: If a domain is created and put into PENDING_DELETE within the same window, deletionTime.isBefore(END_INSTANT) is true even though deletionTime is in the future (> endTime), which will print deleted: <future timestamp> under "Newly created domains". Consider labeling it scheduled deletion: %s when domain.deletionTime().isAfter(endTime).
There was a problem hiding this comment.
Done — appendSection now labels deletionTime > endTime as scheduled deletion: <timestamp>.
| @Config("domainWatcherTlds") | ||
| public static ImmutableSet<String> provideDomainWatcherTlds(RegistryConfigSettings config) { | ||
| return ImmutableSet.copyOf(config.misc.domainWatcherTlds); | ||
| } |
There was a problem hiding this comment.
P1: Null-safety, TLD canonicalization, and config test coverage:
- If an environment override YAML specifies
domainWatcherTlds:with an empty/null value,YamlUtils.mergeMapsreplacesdefault-config.yaml's[]withnull, causingImmutableSet.copyOf(config.misc.domainWatcherTlds)to throwNullPointerException. Please guard againstnull. Domain.tldis stored in canonical lowercase punycode without a leading dot. Uppercase, leading-dot, or Unicode IDN entries in YAML will silently match 0 rows inWHERE tld IN :tlds. Recommend normalizing entries (stripping any leading.and passing throughDomainNameUtils.canonicalizeHostname), documenting canonical ASCII/punycode indefault-config.yaml, and optionally logging a warning inDomainWatcherActionif a configured TLD is not inTlds.getTlds().- Please also add unit test coverage in
RegistryConfigTestforprovideDomainWatcherTlds(empty default, null safety, and normalization/punycode).
There was a problem hiding this comment.
Done — guarded against null, stripped leading ., normalized entries via canonicalizeHostname(...), documented canonical ASCII/punycode in default-config.yaml, added a WARNING log for unknown TLDs in DomainWatcherAction, and added unit tests in RegistryConfigTest.
|
|
||
| @Test | ||
| void testFirstRun_initializesCursorWithoutSendingEmail() { | ||
| persistResource(newDomain("existing.google")); |
There was a problem hiding this comment.
P1: persistResource(newDomain("existing.google")) creates the domain at clock.now() (INITIAL_TIME), which is inside the 5-minute cooldown window (> INITIAL_TIME - 5m) and would have been excluded even if a cursor already existed (and would be picked up on the next run). Set creationTime before INITIAL_TIME.minus(COOLDOWN_DURATION) (e.g., -1d) to genuinely verify historical domains are skipped.
There was a problem hiding this comment.
Done — updated testFirstRun_initializesCursorAndSkipsHistoricalDomains to set creationTime to INITIAL_TIME - 1d (well before cooldown).
| newDomain("too-recent.google") | ||
| .asBuilder() | ||
| .setCreationTimeForTest(INITIAL_TIME.minus(Duration.ofMinutes(2))) | ||
| .build()); |
There was a problem hiding this comment.
P1: Missing exact boundary and historical-deletion tests: too-recent.google is 3 minutes past searchEndTime, so it wouldn't catch < vs <= off-by-one bugs. Please add tests covering:
creationTime == searchStartTime(included) andcreationTime == searchEndTime(excluded on run 1, included on run 2).deletionTime == searchStartTime(included) anddeletionTime == searchEndTime(excluded on run 1, included on run 2).deletionTime < searchStartTime(previously deleted domains are not re-reported).
There was a problem hiding this comment.
Done — added testExactWindowBoundaries_halfOpenIntervalForCreationsAndDeletions covering == startTime, == endTime (excluded on run 1, included on run 2), and deletionTime < startTime.
| .youtube (1): | ||
| - ephemeral.youtube (registrar: TheRegistrar, created: 2026-04-30T13:55:00Z, \ | ||
| deleted: 2026-04-30T14:55:00Z) | ||
| """); |
There was a problem hiding this comment.
P1 / P2: Grouping, ordering, and additional edge-case coverage:
- P1: Each watched TLD currently has only 1 created and 1 deleted domain, persisted in alphabetical TLD order (
googlethenyoutube) matchingwatchedTlds, so this test would still pass ifORDER BY tld ASC, ... ASC, domainName ASCwere broken. Add a test withwatchedTlds = ImmutableSet.of("youtube", "google"), multiple domains per TLD persisted out of order, and at least two domains sharing the same timestamp to verify TLD grouping counts, timestamp ordering, anddomainName ASCtie-breaking. - P2: Consider also covering unwatched TLD deletions, a transferred-then-deleted domain (verifying
currentSponsorRegistrarIdvscreationRegistrarId),searchEndTime < cursorTime(not just==), a newly created domain with a future pendingdeletionTime, and asserting cursor advancement on the second run (L241–L252). Splitting this 104-line multi-scenario test into smaller focused tests would also improve failure diagnostics.
There was a problem hiding this comment.
Done — split into focused tests covering out-of-order TLD grouping (youtube, google), timestamp and domainName ASC tie-breaking, pending deletions and scheduled deletion: formatting, transferred-then-deleted currentSponsorRegistrarId, unwatched TLD deletions, searchEndTime < cursorTime, truncation, and second-run cursor advancement.
43b5325 to
8201c2a
Compare
| AND modificationTime < :endTime | ||
| AND resource.tld IN :tlds | ||
| AND resource.deletionTime >= :endTime | ||
| ORDER BY resource.tld ASC, modificationTime ASC, resource.domainName ASC |
There was a problem hiding this comment.
If the second sort is modification time, that's so fine-grained that there's no point in then also sorting by domain name, as it's extremely unlikely that two domain names will ever be deleted down to the exact same millisecond (or microsecond or whatever the granularity is here).
| WHERE tld IN :tlds | ||
| AND creationTime.creationTime >= :startTime | ||
| AND creationTime.creationTime < :endTime | ||
| ORDER BY tld ASC, creationTime.creationTime ASC, domainName ASC |
There was a problem hiding this comment.
See comment below -- if sorting by creation time second, then sorting by domain name third seems pointless.
| WHERE tld IN :tlds | ||
| AND deletionTime >= :startTime | ||
| AND deletionTime < :endTime | ||
| ORDER BY tld ASC, deletionTime ASC, domainName ASC |
There was a problem hiding this comment.
See above re: sorting by domain name after deletion time.
Add a Nomulus-native domain watcher action to monitor configured TLDs (such as closed/brand TLDs) for domain creations and deletions and send daily email notifications grouped by TLD. Changes: - Add global `CursorType.DOMAIN_WATCHER` to `Cursor` and regenerate `db-schema.sql.generated` (no Flyway migration needed since `Cursor.type` is an unconstrained `text` column). - Add `misc.domainWatcherTlds` (default empty list) to `default-config.yaml`, `RegistryConfigSettings`, and `RegistryConfig` (null-safe, strips leading dots, and normalizes to canonical lowercase ASCII/punycode). - Implement `DomainWatcherAction` (`/_dr/task/domainWatcher`) using a 15-minute global `LockHandler` lock (warning on contention, HTTP 204), a 15-minute cooldown buffer (absorbing in-flight EPP commits and read replica lag), first-run cursor initialization without historical spam, bounded read-only `replicaTm()` queries (capped at 500 domains per section with truncation notice) for newly created domains, pending deletions (`DomainHistory` `DOMAIN_DELETE`), and finalized deletions, and `GmailClient` delivery outside any DB transaction (advancing the cursor after email delivery for intentional at-least-once semantics). - Wire `DomainWatcherAction` into `RequestComponent`, `routing.txt`, and daily (`0 12 * * *` UTC) Cloud Scheduler tasks in `production` and `sandbox` via `/_dr/cron/fanout?queue=retryable-cron-tasks`. - Add unit tests in `DomainWatcherActionTest` and `RegistryConfigTest` covering config normalization/null-safety, empty config, lock contention, first-run historical skipping, exact half-open boundaries, TLD grouping and timestamp ordering, pending vs finalized deletions, transferred-then-deleted registrar attribution, truncation, and email failure retry safety. BUG=http://b/200681440
8201c2a to
2bf8b69
Compare
Adds a Nomulus-native domain watcher action to monitor configured TLDs (such as closed/brand TLDs) for domain creations and deletions and send daily email notifications grouped by TLD.
Summary of Changes
CursorType.DOMAIN_WATCHERtoCursor.javaand regenerateddb-schema.sql.generated(no Flyway migration required sinceCursor.typeis an unconstrainedtextcolumn).misc.domainWatcherTlds(defaults to[], null-safe, strips leading., normalizes to canonical lowercase ASCII/punycode) todefault-config.yaml,RegistryConfigSettings, andRegistryConfig.DomainWatcherAction):POST /_dr/task/domainWatcher(Action.Service.BACKEND,Auth.AUTH_ADMIN).LockHandlerlock (logging a warning and returning HTTP204on contention) and a 15-minute cooldown buffer so in-flight EPP commits and Cloud SQL read-replica lag settle before the cursor advances.Tlds.getTlds().replicaTm()(TRANSACTION_REPEATABLE_READ) with a per-section cap of 500 domains (plus truncation notice) for:Domain.creationTimein window)DomainHistoryDOMAIN_DELETEwithmodificationTimein window anddeletionTime >= endTime, alerting immediately on non-AGP deletes that drop DNS)Domain.deletionTimein window)GmailClienttonewAlertRecipientEmailAddressoutside any DB transaction, intentionally advancing the cursor afterward (at-least-once delivery).RequestComponent.javaandrouting.txt, and scheduled daily at0 12 * * *UTC incloud-scheduler-tasks-production.xmlandcloud-scheduler-tasks-sandbox.xmlvia/_dr/cron/fanout?queue=retryable-cron-tasks.DomainWatcherActionTestandRegistryConfigTest.BUG=http://b/200681440
This change is