Skip to content

fix: upsert null-channel activity relations on their own unique index - #5349

Merged
themarolt merged 3 commits into
mainfrom
fix/issue-1114-null-channel-upsert
Nov 15, 2026
Merged

themarolt merged 3 commits into
mainfrom
fix/issue-1114-null-channel-upsert

Conversation

@themarolt

@themarolt themarolt commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ix_unique_activity_relations_dedup_key treats NULL channels as distinct, so two writes of the same NULL-channel activity both inserted a row. The nightly meetings export still created 0–7 such duplicates a day, and cvent created 68 in one sync on 2026-11-12. This PR adds a partial unique index on the dedup key WHERE channel IS NULL and makes createOrUpdateRelations upsert NULL-channel rows on it, so a racing second write updates the existing row. It also writes the relation before the activity content goes to Kafka, so the racing write's content is sent under the stored activityId instead of an id no relation points to.

Changes

  • Migration V1794682748__null-channel-dedup-unique-index.sql: CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS ix_unique_activity_relations_dedup_key_null_channel ON "activityRelations" ("timestamp", platform, type, "sourceId", "segmentId") WHERE channel IS NULL.
    • It's already built on production. We built it by hand on 2026-11-14, after removing the remaining duplicates, so on production this migration is a no-op. The comment in the file says to check pg_index.indisvalid anywhere else it runs: a leftover duplicate leaves an INVALID index that IF NOT EXISTS would then skip.
  • createOrUpdateRelations: with conflictOn = 'dedupKey', rows with a NULL channel go into a second INSERT … ON CONFLICT ("timestamp", "platform", "type", "sourceId", "segmentId") WHERE "channel" IS NULL DO UPDATE. The WHERE matches the partial index, which Postgres requires to use it as the conflict target. Rows with a channel and the conflictOn = 'activityId' path are unchanged.
    • An empty value list is now skipped instead of sending an INSERT with no rows.
  • createOrUpdateRelations returns the stored activityId for rows that hit an existing dedup key (RETURNING on the upsert, matched back to the input by key), for rows with and without a channel.
  • processActivities writes the relations before insertActivities and sends each activity under the id its relation was stored with. The Kafka-first order dated from when the activity was the primary row; nothing depended on it. A failure between the two writes now leaves a relation without content until the retry, which finds the relation and sends the content under the same id. Before, it left activity content no relation pointed to.
  • Docs: data-flow.md and activity-pipeline.md describe the new order.
  • Comments: the lookup comment in findActivityRelationsByDedupKeys is updated; the new comment in processActivities states why relations go first.

Deploy order: the index has to exist before this code runs, because ON CONFLICT … WHERE channel IS NULL fails without a matching index. On production it already exists. Until this is deployed, a racing NULL-channel insert from the old code fails with a unique violation and the data-sink result is retried; after the deploy, it upserts instead.

Two writes in one batch: processActivities already keeps one activity per dedup key in a batch (NULL channel counted as <empty>), so one INSERT never holds the same NULL-channel key twice. Two copies would make an ON CONFLICT DO UPDATE fail with "cannot affect row a second time".

Tested against a local Postgres 14 with both indexes, calling createOrUpdateRelations:

  1. A mixed batch inserts one NULL-channel row and one row with a channel.
  2. The same keys again with new activityIds update the existing rows, with no new rows, and the upsert returns the stored ids (NULL channel and with a channel).
  3. The activityId path still updates by id.
  4. A different NULL-channel key inserts a new row.

The index build script and the ON CONFLICT clause were also checked against a local Postgres 14 before the production build.

End-to-end on a local stack (Postgres, Kafka, Redis, Temporal) with 3 normal and 3 high data-sink-worker pods on 4 partitions, main (4d72a97, with the new index) against this PR:

  • Normal ingestion: 39,659 real meetings activities (all NULL channel), ingested, re-sent and retried. activityRelations is identical between the two runs (39,637 rows). The other differences are the usual multi-worker member-merge noise, and this PR's steps (createOrUpdateRelations, insertActivities) take the same time.

  • Forced race: 400 new keys, each sent twice at once, 200 with a NULL channel and 200 with a channel.

    Relation rows per key Activity ids in Kafka with no relation
    main, NULL channel 1 250 (153 of 200 keys)
    main, with a channel 1 29 (17 keys)
    this PR, NULL channel 1 0
    this PR, with a channel 1 0

    On main with the index, a racing NULL-channel insert fails the whole batch's relation INSERT after the activities were already sent to Kafka, so every activity in that batch is left without a relation and the retry gives them new ids. This is what production does until this PR is deployed.

  • Kafka send failing after the relation write (failure injected locally): 3 batches failed, the worker retried them, and all 60 keys ended with one relation and their content under the same activityId.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Performance improvement
  • Chore / dependency update
  • Documentation

Related issue

Closes linuxfoundation/lfx-cdp-insights-ops#1114

ix_unique_activity_relations_dedup_key treats NULL channels as distinct, so two
writes of the same NULL-channel activity both inserted. A partial unique index on
the dedup key WHERE channel IS NULL now holds one row per key, and
createOrUpdateRelations upserts NULL-channel rows on it.

Refs linuxfoundation/lfx-cdp-insights-ops#1114

Signed-off-by: Uroš Marolt <uros@marolt.me>
Copilot AI balanced review requested due to automatic review settings November 14, 2026 21:05
@cursor

cursor Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core data-sink persistence order and NULL-channel upsert semantics; production requires the new index before deploy to avoid conflict-target errors.

Overview
Adds a partial unique index on activityRelations for dedup keys when channel IS NULL, so racing inserts for the same NULL-channel activity upsert one row instead of creating daily duplicates.

createOrUpdateRelations now routes NULL-channel rows through a separate INSERT … ON CONFLICT target matching that index, returns input activityId → stored activityId when a dedup key collides, and skips empty insert batches. processActivities writes relations before insertActivities, then repoints payloads to the stored activityId so Kafka/Tinybird content and Postgres stay aligned on retries and concurrent writes.

Docs for the activity pipeline now describe the Postgres-first write order and the updated partial-failure behavior (relation without body until retry).

Reviewed by Cursor Bugbot for commit c8538f2. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge with the documented requirement that the unique index exists before the worker runs.

Summary

The PR adds a unique index for NULL-channel activity relations and uses it for repeat writes. The latest changes save relations before publishing content and reuse the stored activity ID.

  • Activities with no channel now share one relation key.
  • Activity content now uses the ID stored with its relation.

Diagram

sequenceDiagram
    participant DS as data_sink_worker
    participant PG as Postgres
    participant K as Kafka
    DS->>PG: Upsert relation on its matching unique index
    PG-->>DS: Return stored activityId
    DS->>DS: Replace a racing write's generated ID
    DS->>K: Publish content under stored activityId
Loading

Reviews (3) · Last reviewed commit: "fix: write activity relations before sen..." · Reviewed by Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent first writes for the same NULL-channel key can report success while retaining a different activity ID, leaving the incoming title, body, and attributes unreachable through Tinybird’s ID-based join.

1 open issue
What changed in this PR

This PR prevents duplicate NULL-channel activity relations in CDP by adding a dedicated unique index and routing their upserts to it.

Changes:

  • Adds a concurrent partial unique index for NULL-channel deduplication keys.
  • Splits relation upserts by channel presence and skips empty batches, preserving the activity-ID conflict path.
  • Updates comments to explain the new deduplication behavior.
File Description
services/​libs/​data-access-layer/​src/​activityRelations/​index.ts Updates the lookup comment to describe partial-index uniqueness.
services/​libs/​data-access-layer/​src/​activities/​sql.ts Routes NULL-channel upserts to the partial index and skips empty batches.
services/​apps/​data_sink_worker/​src/​service/​activity.service.ts Updates comments explaining existing- and new-relation upserts.
backend/​src/​database/​migrations/​V1794682748__null-channel-dedup-unique-index.sql Adds the partial unique index and documents rollout requirements.

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 14, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Concurrent first writes for NULL-channel keys can report success while silently losing the later write’s content because the upsert retains a different activity ID from the one used to publish that content.

1 open issue

Review effort: Balanced

Racing writes of the same new activity both sent their content under their own
activityId, and the relation upsert kept only one. createOrUpdateRelations now
returns the stored activityId for rows that hit an existing dedup key, and
data-sink-worker writes the relations first and sends the content under that id.
A failed send is retried under the same id instead of leaving an orphaned row.

Refs linuxfoundation/lfx-cdp-insights-ops#1114

Signed-off-by: Uroš Marolt <uros@marolt.me>
Copilot AI balanced review requested due to automatic review settings November 14, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The PostgreSQL/Kafka commit-order change requires human validation of concurrent writes and partial-send recovery, plus verification of partial unique index validity and cleanup/deployment ordering in each target environment.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@themarolt

Copy link
Copy Markdown
Contributor Author

Re: Copilot review · "needs a closer look" checklist

  • Concurrent writes: covered by the forced race in the e2e (400 keys, each sent twice at once, across 6 workers). Main leaves 250 (NULL channel) and 29 (with a channel) activity ids in Kafka with no relation; this PR leaves 0.
  • Partial-send recovery: covered by a Kafka send failure injected after the relation write. The worker retried, found the relation and sent the content under the same activityId: 60/60 keys correct.
  • Partial unique index validity: on production the index was built and checked on 2026-11-14 (indisvalid = t, 840 MB) after the remaining duplicates were removed. Locally the migration ran clean. Staging hasn't been checked yet.
  • Cleanup and deploy order per environment: production is ready (duplicates removed, index valid), so data-sink-worker can deploy. Staging needs the same order before this code reaches it: check for NULL-channel duplicate keys and remove them with cleanup-null-channel-duplicates, run the migration and check indisvalid, then deploy data-sink-worker. Without a valid index, ON CONFLICT … WHERE channel IS NULL has no arbiter and NULL-channel relation writes fail.

@themarolt
themarolt merged commit 0a95e0a into main Nov 15, 2026
17 checks passed
@themarolt
themarolt deleted the fix/issue-1114-null-channel-upsert branch November 15, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants