Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,14 @@ commands:
checkout-suite:
steps:
- run:
name: Verify deployment suite v2.0.5
name: Verify deployment suite v2.0.10
command: |
set -euo pipefail
git init --quiet /tmp/tc-deploy-scripts
git -C /tmp/tc-deploy-scripts remote add origin https://github.com/topcoder-platform/tc-deploy-scripts.git
git -C /tmp/tc-deploy-scripts fetch --quiet --depth 1 origin refs/tags/v2.0.5
git -C /tmp/tc-deploy-scripts fetch --quiet --depth 1 origin refs/tags/v2.0.10
git -C /tmp/tc-deploy-scripts checkout --quiet --detach FETCH_HEAD
test "$(git -C /tmp/tc-deploy-scripts rev-parse HEAD)" = bff574ab5c0b46773e25d2af1eff76675fba2ba5
test "$(git -C /tmp/tc-deploy-scripts rev-parse HEAD)" = e81a6772a33819c1b57c88a95d04494e59c559b8
jobs:
build:
executor: reviewed
Expand Down
4 changes: 2 additions & 2 deletions deploy/MIGRATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# OIDC deployment migration

This change replaces the Auth0 credential broker and SSM-to-shell deployment path with [tc-deploy-scripts v2.0.5](https://github.com/topcoder-platform/tc-deploy-scripts/releases/tag/v2.0.5), fetched by tag and verified against commit `bff574ab5c0b46773e25d2af1eff76675fba2ba5`. Build and deployment run in separate jobs. The deployer cannot read Parameter Store, Secrets Manager or decrypt KMS data, including through another assumed role. ECS continues resolving the existing runtime secret references using the execution role.
This change replaces the Auth0 credential broker and SSM-to-shell deployment path with [tc-deploy-scripts v2.0.10](https://github.com/topcoder-platform/tc-deploy-scripts/releases/tag/v2.0.10), fetched by tag and verified against commit `e81a6772a33819c1b57c88a95d04494e59c559b8`. Build and deployment run in separate jobs. The deployer cannot read Parameter Store, Secrets Manager or decrypt KMS data, including through another assumed role. ECS continues resolving the existing runtime secret references using the execution role.

## Production target

Expand Down Expand Up @@ -39,6 +39,6 @@ IAM templates and detailed AWS baselines are maintained in the private [v6-servi

## Dev branch refresh

This PR targets `develop` and preserves its current application/dependency fixes. The shared suite is v2.0.5 (82 offline tests); service-specific adapters and their tests remain part of review. Keep this PR in draft until its documented infrastructure and integration prerequisites are met. Updating these files does not provision AWS or CircleCI resources.
This PR targets `develop` and preserves its current application/dependency fixes. The shared suite is v2.0.10 (82 offline tests); service-specific adapters and their tests remain part of review. Keep this PR in draft until its documented infrastructure and integration prerequisites are met. Updating these files does not provision AWS or CircleCI resources.

Source checkout uses the public HTTPS repository and verifies the exact CircleCI commit, avoiding dependency on failing SSH checkout keys. No GitHub token is added to build or deployment contexts.
6 changes: 3 additions & 3 deletions deploy/release.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,9 @@ def release(aws, m, publisher, release_tag):
response = aws.call('ecs', 'describe-task-definition', taskDefinition=previous['taskDefinition'], include=['TAGS'])
task, tags = response['taskDefinition'], response.get('tags', [])
migration_request(m, task, tags, 'preflight@sha256:' + '0' * 64)
deadline = time.monotonic() + 900
runtime = publisher.publish('forms-api-v6', 'forms-api-v6:candidate', release_tag, deadline)
migration = publisher.publish('forms-api-v6', 'forms-api-v6:migrate-candidate', release_tag + '-migrate', deadline)
# Each image gets its own scan budget after upload and digest verification.
runtime = publisher.publish('forms-api-v6', 'forms-api-v6:candidate', release_tag)
migration = publisher.publish('forms-api-v6', 'forms-api-v6:migrate-candidate', release_tag + '-migrate')
d.require(initial_stable(read_service(aws, m), previous['taskDefinition']), 'Service changed during image publication')
migration_task = migrate(aws, m, previous, task, tags, migration)
definition = promote(aws, m, previous, task, tags, runtime)
Expand Down
7 changes: 7 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,13 @@ reports `classification: "UNKNOWN"` with null details. Both accept an optional
`UNKNOWN` also matches unclassified submissions. The filter is ignored for forms
without classification. See forms-processor-v6 ADR 0001.

**SendGrid message IDs.** For a form with a `sendgrid-email` processor flow (Let's
talk, SpaceTech), both routes also append `sendgrid_message_ids` (label "SendGrid
logs"), last: the SendGrid message IDs email-service-v6 reported for the submission's
deliveries, oldest send first, or `[]` before any email was sent. The CSV writes the
list as a JSON array. The reports portal links each ID to
`https://app.sendgrid.com/email_logs/<id>/details`.

Dates are optional **inclusive UTC calendar dates** in `YYYY-MM-DD`; an end date
includes timestamps through 23:59:59.999. Invalid calendar dates and reversed
ranges return 400. Existing version-specific JSON/CSV endpoints accept these same
Expand Down
6 changes: 6 additions & 0 deletions docs/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,12 @@ the recipients and template from `SPACE_TECH_CHALLENGE_RECIPIENTS` and
`SPACE_TECH_CHALLENGE_TEMPLATE_ID`. Enable it in SQL after both are set and the processor
is redeployed.

Migration `20261104010000_processor_delivery_sendgrid_message` adds
`ProcessorDelivery.sendgridMessageId` and `emailSentAt`. forms-processor-v6 fills them
from email-service-v6's `external.action.email.sent` reports, matched by the email's
`<submissionId>:<flowId>` correlation ID; the latest non-null ID wins. They are deleted
with the delivery, and the submissions report lists them as `sendgrid_message_ids`.

Processor events contain answers and must be included in personal-data retention
and erasure procedures separately from `Submission`. Deleting a ProcessorEvent
cascades its delivery records and removes deduplication protection, so retain it
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
-- SendGrid message IDs reported by email-service-v6 (external.action.email.sent) for
-- processor email deliveries; forms-processor-v6 matches them by <submissionId>:<flowId>.
ALTER TABLE forms."ProcessorDelivery"
ADD COLUMN "sendgridMessageId" VARCHAR(255),
ADD COLUMN "emailSentAt" TIMESTAMPTZ(3);
22 changes: 13 additions & 9 deletions prisma/schema.prisma
Original file line number Diff line number Diff line change
Expand Up @@ -247,15 +247,19 @@ model ProcessorEvent {

/// Per-flow acceptance receipt and retry schedule; Bus API acceptance is not recipient delivery confirmation.
model ProcessorDelivery {
submissionId String @db.Uuid
flowId String @db.VarChar(100)
attempts Int @default(0)
nextAttemptAt DateTime @default(now()) @db.Timestamptz(3)
lastError String? @db.VarChar(100)
deliveredAt DateTime? @db.Timestamptz(3)
createdAt DateTime @default(now()) @db.Timestamptz(3)
event ProcessorEvent @relation(fields: [submissionId], references: [submissionId], onDelete: Cascade, onUpdate: NoAction)
flow ProcessorFlow @relation(fields: [flowId], references: [id], onDelete: Restrict, onUpdate: NoAction)
submissionId String @db.Uuid
flowId String @db.VarChar(100)
attempts Int @default(0)
nextAttemptAt DateTime @default(now()) @db.Timestamptz(3)
lastError String? @db.VarChar(100)
deliveredAt DateTime? @db.Timestamptz(3)
createdAt DateTime @default(now()) @db.Timestamptz(3)
/// SendGrid X-Message-Id reported by email-service-v6 (external.action.email.sent); the latest non-null report wins.
sendgridMessageId String? @db.VarChar(255)
/// When SendGrid accepted the delivery's email, per the latest report.
emailSentAt DateTime? @db.Timestamptz(3)
event ProcessorEvent @relation(fields: [submissionId], references: [submissionId], onDelete: Cascade, onUpdate: NoAction)
flow ProcessorFlow @relation(fields: [flowId], references: [id], onDelete: Restrict, onUpdate: NoAction)

@@id([submissionId, flowId])
@@index([deliveredAt, nextAttemptAt])
Expand Down
55 changes: 54 additions & 1 deletion src/forms/forms.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ type Definition = Prisma.FormVersionGetPayload<{
const CLASSIFY_ACTION = 'ai-classify';
const UNKNOWN_LABEL = 'UNKNOWN';

/** Processor action that emails through email-service-v6, which reports SendGrid message IDs back. */
const EMAIL_ACTION = 'sendgrid-email';
/** Report column listing each submission's SendGrid message IDs, after any classification columns. */
const SENDGRID_MESSAGES_COLUMN = 'sendgrid_message_ids';

/** Report columns appended for forms with AI classification, after the answer columns. */
const CLASSIFICATION_COLUMNS = [
'classification',
Expand Down Expand Up @@ -210,6 +215,11 @@ export class FormsService {
(await this.db.processorFlow.count({
where: { formKey: key, action: CLASSIFY_ACTION },
})) > 0;
// Forms emailed by the processor report the SendGrid message IDs of their deliveries.
const emailed =
(await this.db.processorFlow.count({
where: { formKey: key, action: EMAIL_ACTION },
})) > 0;
const where: Prisma.SubmissionWhereInput = {
versionId: { in: form.versions.map((version) => version.id) },
createdAt: { ...dates, ...(snapshot ? { lte: snapshot } : {}) },
Expand Down Expand Up @@ -261,8 +271,13 @@ export class FormsService {
'form_version',
...fields.map((field) => field.key),
...(classified ? CLASSIFICATION_COLUMNS : []),
...(emailed ? [SENDGRID_MESSAGES_COLUMN] : []),
];
const data = submissions.slice(0, page.limit).map((submission) => {
const pageSubmissions = submissions.slice(0, page.limit);
const messageIds = emailed
? await this.sendgridMessageIds(pageSubmissions.map(({ id }) => id))
: new Map<string, string[]>();
const data = pageSubmissions.map((submission) => {
const row: Record<string, unknown> = Object.fromEntries(
columns.map((column) => [column, null]),
);
Expand All @@ -289,6 +304,8 @@ export class FormsService {
}
if (classified)
Object.assign(row, classificationCells(submission.classification));
if (emailed)
row[SENDGRID_MESSAGES_COLUMN] = messageIds.get(submission.id) ?? [];
return row;
});
return {
Expand All @@ -297,6 +314,7 @@ export class FormsService {
labels: {
...Object.fromEntries(fields.map((field) => [field.key, field.label])),
...(classified ? CLASSIFICATION_LABELS : {}),
...(emailed ? { [SENDGRID_MESSAGES_COLUMN]: 'SendGrid logs' } : {}),
},
data,
total,
Expand All @@ -305,6 +323,41 @@ export class FormsService {
};
}

/**
* Reads the SendGrid message IDs email-service-v6 reported for these submissions' processor deliveries.
* Submission has no relation to the processor inbox, so deliveries are matched by submission ID.
* @param submissionIds One report page of submissions.
* @returns IDs per submission in send order; submissions without a reported send are absent.
* @throws Database errors on failure.
*/
private async sendgridMessageIds(
submissionIds: string[],
): Promise<Map<string, string[]>> {
const deliveries = submissionIds.length
? await this.db.processorDelivery.findMany({
where: {
submissionId: { in: submissionIds },
sendgridMessageId: { not: null },
// Only email deliveries report SendGrid sends.
flow: { action: EMAIL_ACTION },
},
// PostgreSQL sorts NULLs first ascending; sends without a reported time go last.
orderBy: [
{ emailSentAt: { sort: 'asc', nulls: 'last' } },
{ createdAt: 'asc' },
],
select: { submissionId: true, sendgridMessageId: true },
})
: [];
const ids = new Map<string, string[]>();
for (const { submissionId, sendgridMessageId } of deliveries) {
const list = ids.get(submissionId);
if (list) list.push(sendgridMessageId!);
else ids.set(submissionId, [sendgridMessageId!]);
}
return ids;
}

/**
* Prepares a complete CSV export, validating the request before HTTP headers are sent.
* @param key Form key. @param dates Inclusive date filters, without table pagination.
Expand Down
102 changes: 102 additions & 0 deletions test/forms.integration.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,108 @@ describe('forms API with real PostgreSQL', () => {
});
});

it('reports SendGrid message IDs per submission only for forms with an email flow', async () => {
const key = await create({
...definition(),
fields: definition().fields.map((field) => ({
...field,
required: false,
})),
});
const version = await db.formVersion.findFirstOrThrow({
where: { form: { key } },
});
const reportPath = `/v6/forms/${key}/submissions`;
const report = async () =>
(
await request(app.getHttpServer())
.get(reportPath)
.auth(reporter, { type: 'bearer' })
.expect(200)
).body as {
columns: string[];
labels: Record<string, string>;
data: Record<string, unknown>[];
};
// No email flow yet: the report is unchanged.
expect((await report()).columns).not.toContain('sendgrid_message_ids');

await db.processorFlow.createMany({
data: [
{ id: `${key}-internal`, formKey: key, action: 'sendgrid-email' },
{ id: `${key}-reply`, formKey: key, action: 'sendgrid-email' },
{ id: `${key}-copy`, formKey: key, action: 'sendgrid-email' },
{ id: `${key}-classify`, formKey: key, action: 'ai-classify' },
],
});
const [emailed, pending] = [randomUUID(), randomUUID()];
await db.submission.createMany({
data: [emailed, pending].map((id, index) => ({
id,
versionId: version.id,
idempotencyKey: randomUUID(),
requestHash: 'a'.repeat(64),
createdAt: new Date(Date.UTC(2020, 9, 1 + index)),
})),
});
await db.processorEvent.createMany({
data: [emailed, pending].map((submissionId) => ({
submissionId,
formKey: key,
payload: {},
})),
});
await db.processorDelivery.createMany({
data: [
{
submissionId: emailed,
flowId: `${key}-reply`,
sendgridMessageId: 'sg-reply',
emailSentAt: new Date(Date.UTC(2020, 9, 1, 0, 2)),
},
{
submissionId: emailed,
flowId: `${key}-internal`,
sendgridMessageId: 'sg-internal',
emailSentAt: new Date(Date.UTC(2020, 9, 1, 0, 1)),
},
// An ID without a reported send time sorts after the dated sends.
{
submissionId: emailed,
flowId: `${key}-copy`,
sendgridMessageId: 'sg-undated',
},
// Only email deliveries count, whatever another flow's row holds.
{
submissionId: emailed,
flowId: `${key}-classify`,
sendgridMessageId: 'sg-not-email',
},
// Queued, or sent without a reported ID: no link.
{ submissionId: pending, flowId: `${key}-internal` },
],
});

const body = await report();
expect(body.columns.at(-1)).toBe('sendgrid_message_ids');
expect(body.labels.sendgrid_message_ids).toBe('SendGrid logs');
const byId = Object.fromEntries(
body.data.map((row) => [row.submission_id, row.sendgrid_message_ids]),
);
expect(byId).toEqual({
[emailed]: ['sg-internal', 'sg-reply', 'sg-undated'],
[pending]: [],
});
const csv = await request(app.getHttpServer())
.get(`${reportPath}/export`)
.auth(reporter, { type: 'bearer' })
.expect(200);
expect(csv.text.split('\r\n')[0]).toContain('"sendgrid_message_ids"');
expect(csv.text).toContain(
'"[""sg-internal"",""sg-reply"",""sg-undated""]"',
);
});

it('reports AI classification columns and filters by label only for forms with a classify flow', async () => {
const key = await create({
...definition(),
Expand Down
Loading