Repository navigation
fix: upsert null-channel activity relations on their own unique index - #5349
Conversation
ix_unique_activity_relations_dedup_key treats NULL channels as distinct, so two writes of the same NULL-channel activity both inserted. A partial unique index on the dedup key WHERE channel IS NULL now holds one row per key, and createOrUpdateRelations upserts NULL-channel rows on it. Refs linuxfoundation/lfx-cdp-insights-ops#1114 Signed-off-by: Uroš Marolt <uros@marolt.me>
PR SummaryMedium Risk Overview
Docs for the activity pipeline now describe the Postgres-first write order and the updated partial-failure behavior (relation without body until retry). Reviewed by Cursor Bugbot for commit c8538f2. Bugbot is set up for automated code reviews on this repo. Configure here. |
[Medium impact] The PR appears safe to merge with the documented requirement that the unique index exists before the worker runs. SummaryThe PR adds a unique index for NULL-channel activity relations and uses it for repeat writes. The latest changes save relations before publishing content and reuse the stored activity ID.
|
Refs linuxfoundation/lfx-cdp-insights-ops#1114 Signed-off-by: Uroš Marolt <uros@marolt.me>
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent first writes for the same NULL-channel key can report success while retaining a different activity ID, leaving the incoming title, body, and attributes unreachable through Tinybird’s ID-based join.
1 open issue
What changed in this PR
This PR prevents duplicate NULL-channel activity relations in CDP by adding a dedicated unique index and routing their upserts to it.
Changes:
- Adds a concurrent partial unique index for NULL-channel deduplication keys.
- Splits relation upserts by channel presence and skips empty batches, preserving the activity-ID conflict path.
- Updates comments to explain the new deduplication behavior.
| File | Description |
|---|---|
services/libs/data-access-layer/src/activityRelations/index.ts |
Updates the lookup comment to describe partial-index uniqueness. |
services/libs/data-access-layer/src/activities/sql.ts |
Routes NULL-channel upserts to the partial index and skips empty batches. |
services/apps/data_sink_worker/src/service/activity.service.ts |
Updates comments explaining existing- and new-relation upserts. |
backend/src/database/migrations/V1794682748__null-channel-dedup-unique-index.sql |
Adds the partial unique index and documents rollout requirements. |
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Concurrent first writes for NULL-channel keys can report success while silently losing the later write’s content because the upsert retains a different activity ID from the one used to publish that content.
Review effort: Balanced
Racing writes of the same new activity both sent their content under their own activityId, and the relation upsert kept only one. createOrUpdateRelations now returns the stored activityId for rows that hit an existing dedup key, and data-sink-worker writes the relations first and sends the content under that id. A failed send is retried under the same id instead of leaving an orphaned row. Refs linuxfoundation/lfx-cdp-insights-ops#1114 Signed-off-by: Uroš Marolt <uros@marolt.me>
There was a problem hiding this comment.
🔵 Needs a closer look
The PostgreSQL/Kafka commit-order change requires human validation of concurrent writes and partial-send recovery, plus verification of partial unique index validity and cleanup/deployment ordering in each target environment.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
|
Re: Copilot review · "needs a closer look" checklist
|

Summary
ix_unique_activity_relations_dedup_keytreats NULL channels as distinct, so two writes of the same NULL-channel activity both inserted a row. The nightly meetings export still created 0–7 such duplicates a day, and cvent created 68 in one sync on 2026-11-12. This PR adds a partial unique index on the dedup keyWHERE channel IS NULLand makescreateOrUpdateRelationsupsert NULL-channel rows on it, so a racing second write updates the existing row. It also writes the relation before the activity content goes to Kafka, so the racing write's content is sent under the storedactivityIdinstead of an id no relation points to.Changes
V1794682748__null-channel-dedup-unique-index.sql:CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS ix_unique_activity_relations_dedup_key_null_channel ON "activityRelations" ("timestamp", platform, type, "sourceId", "segmentId") WHERE channel IS NULL.pg_index.indisvalidanywhere else it runs: a leftover duplicate leaves an INVALID index thatIF NOT EXISTSwould then skip.createOrUpdateRelations: withconflictOn = 'dedupKey', rows with a NULL channel go into a secondINSERT … ON CONFLICT ("timestamp", "platform", "type", "sourceId", "segmentId") WHERE "channel" IS NULL DO UPDATE. TheWHEREmatches the partial index, which Postgres requires to use it as the conflict target. Rows with a channel and theconflictOn = 'activityId'path are unchanged.INSERTwith no rows.createOrUpdateRelationsreturns the storedactivityIdfor rows that hit an existing dedup key (RETURNINGon the upsert, matched back to the input by key), for rows with and without a channel.processActivitieswrites the relations beforeinsertActivitiesand sends each activity under the id its relation was stored with. The Kafka-first order dated from when the activity was the primary row; nothing depended on it. A failure between the two writes now leaves a relation without content until the retry, which finds the relation and sends the content under the same id. Before, it left activity content no relation pointed to.data-flow.mdandactivity-pipeline.mddescribe the new order.findActivityRelationsByDedupKeysis updated; the new comment inprocessActivitiesstates why relations go first.Deploy order: the index has to exist before this code runs, because
ON CONFLICT … WHERE channel IS NULLfails without a matching index. On production it already exists. Until this is deployed, a racing NULL-channel insert from the old code fails with a unique violation and the data-sink result is retried; after the deploy, it upserts instead.Two writes in one batch:
processActivitiesalready keeps one activity per dedup key in a batch (NULL channel counted as<empty>), so oneINSERTnever holds the same NULL-channel key twice. Two copies would make anON CONFLICT DO UPDATEfail with "cannot affect row a second time".Tested against a local Postgres 14 with both indexes, calling
createOrUpdateRelations:activityIds update the existing rows, with no new rows, and the upsert returns the stored ids (NULL channel and with a channel).activityIdpath still updates by id.The index build script and the
ON CONFLICTclause were also checked against a local Postgres 14 before the production build.End-to-end on a local stack (Postgres, Kafka, Redis, Temporal) with 3 normal and 3 high data-sink-worker pods on 4 partitions, main (
4d72a97, with the new index) against this PR:Normal ingestion: 39,659 real meetings activities (all NULL channel), ingested, re-sent and retried.
activityRelationsis identical between the two runs (39,637 rows). The other differences are the usual multi-worker member-merge noise, and this PR's steps (createOrUpdateRelations,insertActivities) take the same time.Forced race: 400 new keys, each sent twice at once, 200 with a NULL channel and 200 with a channel.
On main with the index, a racing NULL-channel insert fails the whole batch's relation
INSERTafter the activities were already sent to Kafka, so every activity in that batch is left without a relation and the retry gives them new ids. This is what production does until this PR is deployed.Kafka send failing after the relation write (failure injected locally): 3 batches failed, the worker retried them, and all 60 keys ended with one relation and their content under the same
activityId.Type of change
Related issue
Closes linuxfoundation/lfx-cdp-insights-ops#1114