Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
-- Must exist before createOrUpdateRelations upserts with ON CONFLICT ... WHERE channel IS NULL.
-- A leftover NULL-channel duplicate leaves it INVALID, which IF NOT EXISTS skips: check indisvalid.
create unique index concurrently if not exists ix_unique_activity_relations_dedup_key_null_channel
on "activityRelations" ("timestamp", platform, type, "sourceId", "segmentId")
where channel is null;
9 changes: 5 additions & 4 deletions docs/design/activity-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,10 @@ re-queues the pending result IDs.
## Inside the data sink

For each batch, `data_sink_worker` resolves members and organizations before it writes anything,
then writes the full activity to Kafka before it writes the relation row to Postgres. A failure
between those two writes leaves an activity in Tinybird's `activities` with no relation row in
Postgres yet.
then writes the relation row to Postgres before it writes the full activity to Kafka, under the
`activityId` the relation was stored with. A failure between those two writes leaves a relation row
without its content in Tinybird until the result is retried, and the retry reuses that
`activityId`.

```mermaid
sequenceDiagram
Expand All @@ -151,8 +152,8 @@ sequenceDiagram
Q->>DS: result
DS->>PG: resolve member (create or merge)
DS->>PG: findAffiliation
DS->>K: insertActivities to "activities" topic
DS->>PG: createOrUpdateRelations into activityRelations
DS->>K: insertActivities to "activities" topic
DS->>SS: triggerMembersSync
```

Expand Down
12 changes: 7 additions & 5 deletions docs/knowledge/data-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,16 @@ Every path ends at the `data_sink_worker` Kafka queue, except where noted.
1. resolves the member from the activity's identities, creating or merging as needed (see
[`member-identity.md`](member-identity.md));
2. resolves the organization to credit, with `findAffiliation`;
3. sends the **full activity**, body and attributes included, through `insertActivities` to the
Kafka topic `activities` (`services/libs/data-access-layer/src/activities/insert.ts`);
4. writes the **relation row** through `createOrUpdateRelations` into Postgres `activityRelations`
3. writes the **relation row** through `createOrUpdateRelations` into Postgres `activityRelations`
(`services/libs/data-access-layer/src/activities/sql.ts`);
4. sends the **full activity**, body and attributes included, through `insertActivities` to the
Kafka topic `activities` (`services/libs/data-access-layer/src/activities/insert.ts`);
5. queues an OpenSearch sync through `search_sync_worker`.

Because the Kafka send comes first, a failure between steps 3 and 4 can leave an activity in
Tinybird's `activities` with no relation row in Postgres yet.
The relation is written first. When two writes of the same new activity race, the second upsert
lands on the stored row, and its content is sent under that row's `activityId`. A failure between
steps 3 and 4 leaves a relation row without its content in Tinybird until the result is retried;
the retry finds the relation and sends the content under the same `activityId`.

The relation row is not just foreign keys. Besides `memberId`, `organizationId`, `segmentId` and
the other links, it keeps the activity's `type`, `timestamp`, `platform`, `channel`, `sourceId`,
Expand Down
39 changes: 27 additions & 12 deletions services/apps/data_sink_worker/src/service/activity.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1930,16 +1930,6 @@ export default class ActivityService extends LoggerBase {
}
}

const toUpsert = preparedForUpsert.map((a) => a.payload)
if (toUpsert.length > 0) {
this.log.trace(`[ACTIVITY] Upserting ${toUpsert.length} activities!`)
await logExecutionTimeV2(
async () => insertActivities(this.client, toUpsert),
this.log,
'activityService -> processActivities -> insertActivities',
)
}

// Filter out relations that don't need updates to avoid unnecessary database writes
let skippedCount = 0
const relationsToUpsert = preparedForUpsert
Expand Down Expand Up @@ -1986,11 +1976,11 @@ export default class ActivityService extends LoggerBase {
})
.map((d) => d.relation)

let storedActivityIds = new Map<string, string>()
if (relationsToUpsert.length > 0) {
this.log.trace(
`[ACTIVITY] Upserting ${relationsToUpsert.length} activity relations (filtered from ${preparedForUpsert.length}, skipped ${skippedCount})`,
)
// A NULL channel never hits ON CONFLICT on the dedup key, so found rows are updated by id.
await logExecutionTimeV2(
async () =>
createOrUpdateRelations(
Expand All @@ -2002,7 +1992,7 @@ export default class ActivityService extends LoggerBase {
this.log,
'activityService -> processActivities -> createOrUpdateRelations (existing)',
)
await logExecutionTimeV2(
storedActivityIds = await logExecutionTimeV2(
async () =>
createOrUpdateRelations(
this.pgQx,
Expand All @@ -2018,6 +2008,31 @@ export default class ActivityService extends LoggerBase {
)
}

// Relations are written first so a racing duplicate's content lands under the stored id,
// and a failed send is retried under the same id instead of leaving an orphaned activity.
for (const prepared of preparedForUpsert) {
const storedId = storedActivityIds.get(prepared.payload.id)
if (storedId) {
prepared.payload.id = storedId
}
}
if (storedActivityIds.size > 0) {
this.log.info(
{ count: storedActivityIds.size },
'New activities matched relations written concurrently, sending them under the stored ids',
)
}

const toUpsert = preparedForUpsert.map((a) => a.payload)
if (toUpsert.length > 0) {
this.log.trace(`[ACTIVITY] Upserting ${toUpsert.length} activities!`)
await logExecutionTimeV2(
async () => insertActivities(this.client, toUpsert),
this.log,
'activityService -> processActivities -> insertActivities',
)
}

const createdTypes = new Set<string>()
const createdChannels = new Set<string>()
for (const prepared of preparedForUpsert) {
Expand Down
112 changes: 97 additions & 15 deletions services/libs/data-access-layer/src/activities/sql.ts
Original file line number Diff line number Diff line change
Expand Up @@ -424,17 +424,39 @@ export async function getLastActivitiesForMembers(
return activities.rows
}

function relationDedupKey(relation: {
timestamp: string | Date
platform: string
type: string | null
sourceId: string | null
channel: string | null
segmentId: string
}): string {
return JSON.stringify([
new Date(relation.timestamp).toISOString(),
relation.platform,
relation.type ?? null,
relation.sourceId ?? null,
relation.channel ?? null,
relation.segmentId,
])
}

// Returns input activityId -> stored activityId for rows that hit an existing dedup key: the
// caller must publish the activity content under the stored id.
export async function createOrUpdateRelations(
qe: QueryExecutor,
relations: IActivityRelationCreateOrUpdateData[],
skipChecks = false,
conflictOn: 'dedupKey' | 'activityId' = 'dedupKey',
): Promise<void> {
): Promise<Map<string, string>> {
const storedActivityIds = new Map<string, string>()
if (relations.length === 0) {
return
return storedActivityIds
}

const params: Record<string, unknown> = {}
const writtenKeys: { activityId: string; key: string }[] = []
let index = 0

const activityIds = new Set<string>()
Expand All @@ -444,6 +466,9 @@ export async function createOrUpdateRelations(
const checkedSegments = new Map<string, boolean>() // id -> exists
const checkedConversations = new Map<string, boolean>() // id -> exists
const valueList: string[] = []
// ix_unique_activity_relations_dedup_key treats NULL channels as distinct, so a NULL-channel row
// never conflicts there. These rows conflict on the partial index for NULL channels instead.
const nullChannelValueList: string[] = []
for (const data of relations) {
if (data.username === undefined || data.username === null) {
continue
Expand Down Expand Up @@ -695,7 +720,23 @@ export async function createOrUpdateRelations(
params[isContributionParam] = false
params[pullRequestReviewStateParam] = data.pullRequestReviewState ?? null

valueList.push(
if (conflictOn === 'dedupKey') {
writtenKeys.push({
activityId: data.activityId,
key: relationDedupKey({
timestamp: data.timestamp,
platform: data.platform,
type: data.type ?? null,
sourceId: data.sourceId ?? null,
channel: data.channel ?? null,
segmentId: data.segmentId,
}),
})
}

const values =
conflictOn === 'dedupKey' && params[channelParam] === null ? nullChannelValueList : valueList
values.push(
`
(
$(${activityIdParam}),
Expand Down Expand Up @@ -725,8 +766,36 @@ export async function createOrUpdateRelations(
)
}

await qe.result(
`
const statements: [string[], string][] = [
[
valueList,
conflictOn === 'activityId'
? '("activityId")'
: '("timestamp", "platform", "type", "sourceId", "channel", "segmentId")',
],
// matches the partial index ix_unique_activity_relations_dedup_key_null_channel
[
nullChannelValueList,
'("timestamp", "platform", "type", "sourceId", "segmentId") WHERE "channel" IS NULL',
Comment thread
themarolt marked this conversation as resolved.
Resolved
Comment thread
themarolt marked this conversation as resolved.
Resolved
],
]

const storedIdsByKey = new Map<string, string>()
for (const [values, conflictTarget] of statements) {
if (values.length === 0) {
continue
Comment thread
themarolt marked this conversation as resolved.
Resolved
}

const stored: {
activityId: string
timestamp: Date
platform: string
type: string | null
sourceId: string | null
channel: string | null
segmentId: string
}[] = await qe.select(
`
INSERT INTO "activityRelations" (
"activityId",
"memberId",
Expand All @@ -751,14 +820,10 @@ export async function createOrUpdateRelations(
"pullRequestReviewState",
"createdAt",
"updatedAt")
VALUES ${valueList.join(',')}
VALUES ${values.join(',')}

ON CONFLICT ${
conflictOn === 'activityId'
? '("activityId")'
: '("timestamp", "platform", "type", "sourceId", "channel", "segmentId")'
}
DO UPDATE
ON CONFLICT ${conflictTarget}
DO UPDATE
SET
"updatedAt" = EXCLUDED."updatedAt",
"memberId" = EXCLUDED."memberId",
Expand All @@ -777,10 +842,27 @@ export async function createOrUpdateRelations(
"gitDeletions" = COALESCE(EXCLUDED."gitDeletions", "activityRelations"."gitDeletions"),
"score" = EXCLUDED."score",
"isContribution" = EXCLUDED."isContribution",
"pullRequestReviewState" = EXCLUDED."pullRequestReviewState";
"pullRequestReviewState" = EXCLUDED."pullRequestReviewState"
RETURNING "activityId", "timestamp", "platform", "type", "sourceId", "channel", "segmentId";
`,
params,
)
params,
)

if (conflictOn === 'dedupKey') {
for (const row of stored) {
storedIdsByKey.set(relationDedupKey(row), row.activityId)
}
}
}

for (const { activityId, key } of writtenKeys) {
const storedId = storedIdsByKey.get(key)
if (storedId && storedId !== activityId) {
storedActivityIds.set(activityId, storedId)
}
}

return storedActivityIds
}

export async function updateActivityRelationsById(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,8 @@ export async function findActivityRelationsByDedupKeys(

const columns = ALL_ACTIVITY_RELATION_COLUMNS.map((c) => `ar."${c}"`).join(', ')

// The dedup index treats NULL channels as distinct, so a key with a NULL channel can match
// several copies. Return one row per key: the latest copy.
// A NULL channel is matched with "is not distinct from". The partial unique index on NULL-channel
// keys keeps one row per key; distinct on still returns the latest copy if two ever exist.
return qx.select(
`
select distinct on (k.ts, k.platform, k.type, k."sourceId", k.channel, k."segmentId") ${columns}
Expand Down
Loading