Skip to content

Repository navigation

compliance.md

Latest commit

 

History

History
222 lines (171 loc) · 52.2 KB

compliance.md

File metadata and controls

222 lines (171 loc) · 52.2 KB
Raw

Compliance Posture

Status: active

When to use this runbook: planning a deployment that must satisfy regulatory requirements (GDPR, CCPA, HIPAA, PCI DSS, SOC 2), responding to a security questionnaire from a prospective customer, or evidencing compliance during an audit.

This document describes what Powernode-the-software supports out of the box, what it leaves to operator configuration, and what it explicitly does not claim. Powernode-the-software is not certified for any specific compliance regime. Certification is operator-owned — this doc tells you what the platform makes easy versus hard.

Contents

Data classification

Powernode stores the following classes of data. Treat each according to your regulatory scope.

Class Examples Default storage Notes
Account/user PII name, email, password hash, last login Postgres users, accounts Email is the primary identifier; minimize collection if your regime requires data minimization
Auth secrets password hashes (bcrypt), JWT secrets, API keys Postgres + Vault Passwords never logged. Vault stores rotation-capable secrets
Agent/conversation content user prompts, AI responses, tool call payloads Postgres ai_conversations, ai_messages May contain customer-supplied PII inside prompts. Audit before retaining indefinitely
Audit logs every state change with actor + resource + metadata Postgres audit_logs Append-only; never edited. Required for most compliance regimes
Knowledge entries platform knowledge contributions, learnings, embeddings Postgres ai_shared_knowledges, ai_compound_learnings May contain operational/customer details depending on what agents recorded
Vector embeddings 1536-dim OpenAI, 768-dim Ollama-default Postgres pgvector columns Re-derivable from source text
Payment data tokens, last4, brand Stripe/PayPal vault — NOT in Postgres Powernode stores only payment-method tokens (provider-side opaque IDs). Full card data never enters the platform
Generated artifacts PDF/CSV reports, file uploads Worker filesystem (worker/storage/) Treat the storage volume as containing whatever data the report queries returned

Retention

Data Default retention Configured via Compliance note
Audit logs indefinite none — append-only Most regimes require 1-7 years. Default is fine. Add manual archival for cost.
Conversations indefinite none Consider auto-expiry for GDPR data-minimization. No built-in TTL.
Generated reports none built-in manual — files live on the worker filesystem (worker/storage/) Regenerable from data; snapshot/prune the storage volume per your regime
Background job dead set 14 days (Sidekiq default) worker/config/sidekiq.yml Operational only — no customer data
Loki logs 7 days configs/logging/loki-config.yml retention_period PCI requires 1 year minimum; adjust before claiming PCI
Postgres backups 30 days local Maintenance::BackupCleanupJob (BACKUP_RETENTION_DAYS) Match to longest applicable retention; off-host/S3 copies are operator-managed

GDPR / CCPA

What the platform supports:

  • Right to erasure: Compliance::AccountTerminationJob and Compliance::DataDeletionJob (in worker/) are each a sequence of internal API calls, not a single atomic transaction. As of IMP-b33a3ecca331 (server-side params contract, gained during review: response-body key handling, an index-filter gap, a status-transition guard) both jobs reliably reach a terminal state (completed/grace_period-for-retry, or failed) with the effects they claim actually applied.
    • The jobs literally never ran correctly before this fix, independent of everything else below. BackendApiClient#handle_response (worker/app/services/backend_api_client.rb) returns the parsed JSON body verbatim — STRING keys — on a 2xx response and raises on any non-2xx; it is not a {success:, data:} symbol-keyed envelope. Every response read in both jobs used symbol keys (response[:success], response[:data]) against that string-keyed hash, so those reads were always nil — DataDeletionJob raised "Failed to fetch" on every single run, and AccountTerminationJob's return unless response[:success] made it a permanent no-op. Worker specs never caught this because they stub symbol-keyed doubles. Fixed by reading responses the way the rest of the worker's compliance-adjacent jobs do (string keys, e.g. Ai::ApprovalExpiryJob) — applied to every response read in both files, including ones that predate this task. Api::V1::Internal::DataDeletionRequestsController#show also nests its payload one level deeper (data.data_deletion_request, not data) — the job now unwraps both levels.
    • The account-termination job's index query used to ignore its own filter. Api::V1::Internal::AccountTerminationsController#index used to return every active (pending/grace_period/processing) termination regardless of the status/grace_period_expired params the job sent — once the key bug above was fixed, the next 6-hourly run would have destructively processed (anonymize users, delete API keys/webhooks, cancel the account) every termination still pending (never confirmed by the account owner) and every grace_period termination whose 30-day window hadn't actually elapsed. Fixed by honoring those params, reusing the model's own Account::Termination.ready_for_processing scope for the job's exact "ready to process" query rather than reimplementing its semantics.
    • Account termination — per-user step: works. Consents/terms-acceptances/password-history are deleted, and the user record is anonymized in place (email/name/credentials/2FA/preferences cleared, password history purged, status set inactive — see Api::V1::Internal::UsersController#anonymize) rather than the row being deleted.
    • Account termination — account-level step: works. The account-status update goes through one narrow, purpose-built action — PATCH /api/v1/internal/accounts/:id/terminate (Api::V1::Internal::AccountsController#terminate) — instead of an unrouted, payload-driven PATCH accounts/:id. Operator decision on what "terminated" means for the account row: status is set to cancelled, the existing valid_account_status enum value closest to "this account is done" (no migration, no new status value, no new timestamp column — Account has no fitting one). The action takes no payload and is idempotent (calling it on an already-cancelled account succeeds without a duplicate write or audit row). It is called BEFORE the termination record is marked completed (a completed termination is excluded from every re-fetch this job makes, so a failure after marking it complete would have left the termination "done" while the account was never actually cancelled, with no automatic retry). Account::Termination#completed_at remains the authoritative WHEN/WHY record for the termination itself. Reviewed for side effects: no Account model callback reacts to its own status changing (only after_create_commit hooks exist, irrelevant to an update); account.active? gates login (Authentication concern, Auth::SessionsController) and AI/discovery processing (several internal controllers/services) — both correctly stop once the account is cancelled, with no destructive or billing-side action triggered automatically. The billing extension's Account decorator adds associations and export helpers only, no callbacks. Separately, Account::Termination#handle_status_change (an after_update callback on the TERMINATION record, not Account) sets account.status = "suspended" the moment a termination transitions TO grace_period — this fires on this job's own error-revert path (a failed run reverts the termination to grace_period for retry), so a partially-processed account is automatically locked out (login blocked) while the retry is pending, not left fully active.
    • Account termination — the log used to be wiped on every successful run, and a first fix still stripped some of it. process_termination started from termination_log = [], and the PATCH update REPLACED the stored jsonb array rather than appending — so a successful run erased whatever history the model itself had written (requested, confirmed, reminder-sent entries) as well as any prior failed attempt's error entry. A first fix had the job seed its local array from the termination's already-fetched termination_log and send the whole thing back — but the permitted key list only covered keys the JOB itself writes (event/user_id/count/error/reason/at), so model-authored entries (confirm!/cancel!/complete!/schedule_reminders, which write by/days_before/scheduled_for) were silently stripped on every worker write regardless, and a whole-array replace remained a lost-update race against any concurrent writer. The actual fix: AccountTerminationsController#update accepts a termination_log_append param — ONLY the job's own new entries for this run — and merges it onto the CURRENT stored log server-side, inside a with_lock (pessimistic row lock, read-reload-append-write). The whole-array termination_log param is removed entirely (no legacy path); the job no longer seeds or re-sends fetched history at all. The permitted key list also dropped by/days_before/scheduled_for (only the model's own writers use them, never the job), and every appended entry's event VALUE is checked against the exact set the job builds — key-permitting alone still let a forged event NAME through (event/at were always permitted keys), so {event: 'confirmed', at: ...} would otherwise inject a fabricated model-lifecycle entry into the audit trail unchallenged.
    • Account termination — the raw status-write endpoint used to accept ANY transition, with no audit trail. AccountTerminationsController#update (the only endpoint Compliance::AccountTerminationJob PATCHes) had no transition guard at all: a worker principal could jump a termination straight to completed (destructive — cancels the account) without ever having gone through processing, or restart an un-expired grace_period termination early, with no audit row recording it. Fixed with an explicit guard mirroring the transitions the job actually makes — grace_period→processing only when can_start_processing? holds (the grace period has actually elapsed, mirroring the model's own guard on start_processing!), processing→completed/grace_period — checked twice (once fail-fast, once inside the with_lock against a freshly-reloaded row) so a concurrent writer can't race past it. A status-less log-only write (the reminder-sent path) is allowed only while the termination is still grace_period/processing; once completed/cancelled its history is frozen. Every allowed status transition writes an account_termination.status_transition audit row with from_status/to_status.
    • Account termination — subscription anonymization: when the billing extension is loaded, subscription data was never anonymized (the job called an unrouted /subscription/anonymize endpoint). Core has no generic seam for this today (Powernode::BillingBridge registers subscription/payment/plan models and a provisioning quota/meter handler, but no anonymize handler) — rather than build a new bridge seam or call an unrouted endpoint, the job now explicitly skips this step and records {event: 'subscription_anonymize_skipped', reason: 'no_billing_extension_provider'} in the termination log. This is a real, currently-open gap: a terminated account's subscription/billing record is not anonymized. Tracked as a follow-up.
    • Data deletion: DataManagement::DeletionRequest gained a failed status value (it previously had none, so the job's own status: 'failed' writes were themselves rejected by model validation on top of the params-contract bug). The "anonymize"/"full"/"partial" deletion types now route the profile, audit_logs, payments, and consents data types at real, already-routed internal actions (there was never a /api/v1/internal/data_deletion/:type route) — those actions now also return the record count the job logs (delete_consents/delete_files used to be message-only, so that count was always read as 0). A request stuck in processing (e.g. the worker crashed mid-run) now resumes on retry instead of being silently skipped; an unrecoverable failure now ends in status: 'failed' (terminal) instead of a dangling processing record with no signal. failed has no retry path and none was added: approve_request requires pending? and execute_request requires approved?, so no admin action_type transitions a failed request back to something this job would ever process again — a user whose deletion failed must file a NEW request (Api::V1::PrivacyController#request_deletion; its DataManagement::DeletionRequest.active guard already excludes failed, so the dead request does not block a fresh one). A per-data-type erasure failure now raises a dedicated Compliance::DataDeletionJob::PartialDeletionFailure (not a bare string) — the outer rescue recognizes it and skips its own redundant status: 'failed' write, since that path already wrote the terminal status (with the full deletion_log/retention_log detail the outer write doesn't have) before raising, and a second write would now 422 under the transition guard (failed→failed isn't an allowed transition).
    • Data deletion — approving a request used to omit its own grace period. DataDeletionRequestsController#approve_request never set grace_period_ends_at (the model's own #approve! method does, but nothing calls it — this controller action is the only approval path that exists). Compliance::DataDeletionJob#execute reads that field unconditionally once a request is approved, and the Time.zone.parse(...) call sits BEFORE the job's begin/rescue block entirely (deliberately — the grace-period check has to run before the request is ever marked processing), so the resulting Time.zone.parse(nil) TypeError was never caught by anything: no failed write happened, no error was recorded anywhere, the request stayed approved forever, and every Sidekiq retry hit the exact same nil value and crashed the exact same way — a permanent, invisible crash-loop, not a retryable failure, with no terminal state to even alert on. Fixed on both sides: approve_request now sets grace_period_ends_at (reusing the model's GRACE_PERIOD_DAYS constant), and the job independently guards against a missing value regardless — a nil/blank grace_period_ends_at now writes a terminal status: 'failed' and returns cleanly instead of raising (defense in depth against any other path that reaches approved/processing without the field set).
    • Data deletion — the generic status-write endpoint used to accept ANY transition, with a lost-update race and an under-scoped status-less write. The branch Compliance::DataDeletionJob PATCHes (no action_type) had no transition guard at all: any mTLS-enrolled worker principal could set any valid model status — including completed on a request that was never approved — bypassing complete_request's own guard, its audit row, and its user notification, effectively forging a GDPR completion record. Fixed with an explicit allowlist (approved→processing, processing→processing/completed/failed; anything else is rejected with 422 INVALID_STATUS_TRANSITION) plus ONE narrow conditional exception — approved→failed is allowed ONLY when grace_period_ends_at is actually blank (the exact data-integrity defect the nil-guard above exists for, not a general escape hatch from approved) — checked twice (fail-fast, then again inside a with_lock against a freshly-reloaded row so a concurrent writer can't race past it), with its own audit action (data_deletion.status_transition, carrying both from_status and to_status, captured from the freshly-locked row so it can't name a stale status) on every allowed transition. A status-less write (no status key) carrying progress fields (completed_at/deletion_log/retention_log/error_message/metadata/processing_started_at) is now permitted only while the request is actively processing — previously unrestricted for the first four fields, and metadata/processing_started_at weren't guarded at all — a request that's pending/approved has no run in flight to report progress for, and no real caller ever sends these status-less outside processing anyway.
    • Data deletion — data-type coverage (IMP-bf52b4da135b). DELETABLE_DATA_TYPES previously named 9 categories, five of which (files, activity, settings, communications, analytics) were recorded as {action: 'skipped', reason: 'no_backing_data_model'} — an honest log of a promise the platform could not keep. That skip reason is now no_erasure_path: the old value claimed there was no data model, which was never true for files (FileManagement::Object exists; what is absent is a safe erasure path). The rename is forward-only, and the old value was misleading for older rows: before this change files, settings and communications were on the skip list too, so any request that named them and completed recorded skipped / no_backing_data_model while that data was not erased (settings and communications were reached only as a side effect of #anonymize on a request that also processed profile, and EmailDelivery never). Nothing re-processes those requests; a data subject whose completed request named one of those types must file a new one. Only activity and analytics rows are accurately described by the old string. The constant is the platform's advertisement of what a data subject can have erased, so every remaining entry now has a real erasure backend and the rest are withdrawn from the offer. Withdrawal means the platform stops offering a category it has no safe erasure path for; it is not a claim that the underlying data does not exist.

      • settings → the users.preferences / users.notification_preferences columns (DELETE /api/v1/internal/users/:user_id/settings).

      • communications → two models: Notification and EmailDelivery (DELETE /api/v1/internal/users/:user_id/communications). EmailDelivery was erased by nothing at all, so a completed deletion logged communications → deleted while the subject's plaintext recipient_email and the fully rendered body_html/body_text of every email sent to them survived verbatim.

        Known residue — unattributed deliveries survive. email_deliveries.user_id is nullable, and erasure matches on user_id only. A delivery row whose user_id is null is therefore not erased, even when its recipient_email is the data subject's address — so an email sent to them before (or outside) user attribution can persist after a completed deletion, carrying their address and the rendered message body. This is deliberate: matching on recipient_email instead would reach other people's deliveries that happen to share an address, which is a worse failure than the one it fixes. Operators who need those rows gone must handle them out of band. Narrowing this properly means attributing deliveries at write time, which is a separate piece of work.

      • Scope of the settings/communications win — read this before assuming a general improvement. On the default full path these two endpoints are not load-bearing: profile is processed first and #anonymize already clears both preference columns and deletes notifications inside its own transaction, so the new endpoints run second and find those rows gone. They matter for exactly two cases — a partial request naming settings/communications, and a full request that retains profile. The genuinely new erasure on every path is EmailDelivery, which #anonymize never touched.

      • files → FileManagement::Object rows and their blobs, through FileManagement::Erasure (DELETE /api/v1/internal/users/:user_id/files for a deletion request; DELETE /api/v1/internal/accounts/:account_id/files for an account termination). IMP-bf52b4da135b had withdrawn it: five tables reference file_objects with no inverse association and no on_delete (chat_message_attachments, system_disk_image_publications ×2, system_node_architectures ×3), so destroying a chat-attached file raised PG::ForeignKeyViolation … "fk_rails_ca093e583a" mid-iteration after shares had already been deleted; after_destroy :remove_from_storage swallowed a blob-removal failure; and the scope held non-personal platform artifacts. IMP-d97f6e3bbc2b is the erasure path, and each of those is resolved by construction rather than by care:

        Restrict foreign keys go through a seam, not a list. FileManagement::ErasureReferentRegistry is a core-owned handler registry (the Powernode::HandlerRegistry shape, like Devops::ContainerLifecycleRegistry); core never names an extension. Before a batch, every handler is asked which ids it holds; before each destroy, every handler may release its own references. Core's chat_message_attachments handler releases (nullifies file_object_id — the attachment's pointer is a convenience, belongs_to … optional: true, and the attachment row stays). Known residue — the attachment row survives. That row carries filename, transcription, metadata and storage_url; it is chat data, and no DELETABLE_DATA_TYPES category erases chat_message_attachments (or chat messages) today, so after a completed files erasure the attachment metadata of a chat-attached file persists with its pointer nulled. Deciding which chat records are a data subject's for Article 17 is the same unanswered policy question as activity below. The system extension's handler holds all three of its boot-image pointers (system_disk_image_publications current/prior, and the unconstrained system_node_platforms.disk_image_file_object_id): nullifying any of them would corrupt a boot-image pointer or a rollback history underneath a running fleet, so the file is refused with a reason naming the holder (held_by_system_disk_image_publication, …) and reported as a gap. A restrict FK nobody registered is still safe: the violation fires inside the per-file transaction and is caught, and the file is reported as held / referenced_by_restrict_fk.

        Each file is erased in its own transaction, with the blob removal inside it. The referents are released, the shares, versions and processing jobs deleted and the row destroyed in one transaction; the model's after_destroy removes the blob after the row DELETE and before the storage-counter write, still inside that transaction, in a strict mode where a provider returning false (every provider rescues internally and returns false rather than raising) raises and rolls the whole file back — row, shares, versions, processing jobs, tags and the destroy's own audit row. A failed blob delete is therefore never counted as erased. The destroy's audit row redacts the personal content fields (filename, storage_key, metadata, exif_data, …), and the versions and processing jobs are removed with delete_all so their own Auditable rows (which would archive a version's storage_key and change_description, a job's parameters and results) are never written; the erasure does not re-archive what it erased. Residual window: the provider has removed the blob and the transaction then fails to commit (the storage-counter write, or the commit itself). The row survives, the blob is gone, the file is reported as failed; a retry re-selects it and completes, because every provider's delete_file treats a missing blob on a reachable store as success. Until then the row is a dangling pointer, not retained content. Reachability of a path-backed store (local root, NFS, SMB) — positive evidence only. An unmounted share's mount point is an empty directory in which every blob is "missing", so "missing" alone proves nothing. The providers' delete_file therefore deletes an existing blob whatever the mount state, and counts a missing one as removed only with positive evidence that the store is live: a liveness marker (.powernode_store) at the store's base, written by initialize_storage — i.e. by a successful write to the store — holding a nonce that is also kept on the FileManagement::Storage row (metadata.liveness_marker). The marker must be readable and match the row's current nonce; without that the blob is refused (false, reported storage_removal_failed for a stale or missing marker and store_not_initialized for a row with no nonce — both retryable), never "removed". The erasure decision reads no mount table, device or server address; /proc/self/mountinfo is consulted only by the NFS/SMB initialize_storage, to refuse writing the marker onto a base that is not on a share of the right type. The providers' legacy mounted? still serves initialize_storage / test_connection / health_check but never the erasure decision. The same probe (store_live?) is what the erasure uses to tell a dead store from one bad object (below). Fail direction per setup:

        Setup, blob missing Marker Result
        Share mounted at mount_path, blob genuinely gone (retry after a partial commit) present, matches removed — correct
        NFS/SMB share unmounted; mount point is an empty directory absent refused (closed) — correct
        Local root on a dedicated volume that is unmounted; empty mount-point directory absent refused (closed) — correct
        autofs direct map, server down (the lookup triggers an automount that fails) unreadable refused (closed) — correct
        btrfs subvolume as the mount point, nothing mounted absent refused (closed) — correct
        mount_path nested under the real mount (/mnt/nfs/storage-a on a share at /mnt/nfs, one share serving several storages) present at the nested base removed — correct; this configuration is supported
        Same-device bind mount of an empty directory onto the mount point (a container bind-mounting a host directory whose share is unmounted on the host) absent refused (closed) — correct
        mount_path: / — local root only present at / removed — a nonsense configuration, reads as live. On NFS/SMB, initialize refuses (/ is never on an nfs/cifs mount): the operator creates a nested base directory on the share and points mount_path at it before initializing
        Store created before the marker existed (no nonce on the row) n/a refused (closed), reason store_not_initialized, until an operator re-runs initialize (below); the store is not treated as dead, so its other files are still attempted
        initialize_storage run on an NFS/SMB store while the share is unmounted not written initialize refuses (false, the error names the share and the filesystem it found) — the marker is written only onto a base that /proc/self/mountinfo shows on an nfs/nfs4 (NFS) or cifs/smb3 (SMB) mount; a marker left there by an older build is shadowed once the share mounts, and the next initialize on the live share writes a new nonce it no longer matches

        Liveness is three-valued. No nonce on the row (a store that predates the marker) is no evidence: a missing blob is refused with the reason store_not_initialized — the erasure's failure reason, the deletion request's error_message and the termination log all say to initialize the store — but the store is not probed as dead, so the files beside it, whose blobs may well exist, are still attempted. A nonce with a missing or mismatched marker is dead: refused as storage_removal_failed, and the storage's remaining files in the batch are short-circuited (below). Existing stores that predate the marker therefore need an operator to re-run initialize_storage for them (Admin → Storage → test / initialize, or FileManagement::Storage#storage_provider.initialize_storage from a console) with the share mounted — on NFS and SMB the initialize itself refuses otherwise. New stores get the marker on creation. S3, GCS and Azure have no mount: the service's own answer is the evidence (a delete of a missing key succeeds), so they need no marker and are never short-circuited. Concurrent duplicate: the worker's HTTP client retries a timed-out DELETE, so the same batch can be in flight twice; each file is row-locked at the top of its transaction, and the loser finds no row and reports nothing — no second destroy, storage-counter decrement or audit row. Latent, not live: processed derivatives would be written under processed/<file_object_id>/… storage keys with no row of their own, and nothing in the erasure removes them; no provider implements the upload that would create them today (worker_files_controller), so none exist — when one does, the erasure must remove that prefix too.

        Only personal artifacts are erased. FileManagement::Erasure::PERSONAL_CATEGORIES (user_upload, workflow_output, ai_generated, temp, import, page_content) is an allowlist; a disk_image, sbom_export, attestation_proof, supply_chain_scan_report, vendor_* or system file in a terminated account is left and counted as retained_platform_artifacts. A category added later defaults to retained-and-reported, not destroyed. Soft-deleted personal files are included: the platform still holds them. Open operator question — the allowlist on the user path. import, workflow_output and page_content look organisational rather than personal: a departing employee's deletion request erases the account's data imports and workflow outputs they happened to upload, and page_content has no producer today. The same allowlist serves both paths; if those three should be the account's to keep, the user path (DELETE /api/v1/internal/users/:user_id/files) needs a narrower list (user_upload, ai_generated, temp) while the termination path keeps all six. Unchanged pending a ruling.

        Bounded per request. One request erases at most batch_size files (default 50, maximum 200, sized against the worker's per-request client timeout, API_TIMEOUT, 120 s by default) and answers with remaining and a cursor (UUIDv7 ids are time-ordered); the worker loops on the cursor, which always advances past a failed file, and gives up after 10,000 batches. A dead storage does not starve the rest, and one undeletable blob does not starve its batch-mates: within one request, after a storage's first failed blob removal the server runs that storage's liveness probe (store_live?, the same marker test as above) once; only a positive unreachability result (no evidence the store is live) short-circuits — the storage's remaining files in the batch are reported storage_unreachable without a provider call — while a failure on a live store stays per-file (an object under Object Lock or legal hold, a denied ACL, a blob the provider refused) and the files beside it are still attempted. So a dead store costs one attempt plus one probe per batch, the cursor moves past its files, and files on a healthy store behind them are still erased; the walk is bounded because remaining strictly decreases. S3, GCS and Azure report live always (no mount to lose), so every failure there is per-file and the cost is bounded by the batch size times the SDK's own timeout. The deletion-request path erases the files the subject uploaded in their account (account_id and uploaded_by_id — a co-member's uploads are the co-member's personal data, and file_objects.account_id is independent of the uploader's account); the termination path erases every personal file in the account.

        Honest outcomes on both paths, with held and operational failures kept apart. Compliance::DataDeletionJob records files → deleted with the summed count. A file held by a referent is a policy outcome no retry clears: the category is recorded as a failed erasure and the request ends failed with the reason on record — the subject's file is still there. A file whose blob the provider could not remove, a storage that hangs (an NFS hard mount with the server down, a blackholed object-store endpoint — surfaced as the worker's 120-second timeout, an open circuit, or a 408/5xx from the server), and a server that becomes unavailable mid-walk are all operational: the job raises RetryableErasureFailure, writes error_message plus the partial deletion_log / retention_log (the types already settled) on the still-processing request, and Sidekiq's retry resumes it; a 4xx other than a routing-miss 404 (a malformed cursor, a missing user) is not operational and ends the request failed like any other per-type error; a resumed run carries the persisted settled entries forward instead of re-running those types to a count of 0, re-runs only the unsettled ones, and the completing write clears error_message. A 404 that arrives on a later batch, after files were already erased, is never a skip — it is the 4xx failure it is; only a routing-miss 404 on the first request (no code: "NOT_FOUND" envelope — an old server) is a skip. An interrupted attempt's erased count is persisted as a failed files entry with records_affected, and the resumed run adds it to its own count, so the final files entry reports the whole erasure even though the cursor restarts. The retry window is short and unattended: BaseJob gives 3 retries, and RetryableErasureFailure takes its non-ApiError backoff (count⁴ + 15 + rand(30)·(count+1)): roughly 15–45 s, 16–76 s and 31–121 s — about four minutes end to end — nothing sweeps processing deletion requests afterwards, and nothing alerts. A real storage outage therefore outlasts the retries, and the operator must act: once the retries are exhausted the job lands in the Sidekiq dead set and the request stays processing with error_message set (it never becomes failed, because failed has no re-arm path and the anonymized subject cannot re-file). Operator recovery: restore the storage backend (re-mount the share, restore the root, fix credentials — FileManagement::Storage#test_connection tells you which), then re-run the job: retry it from the Sidekiq dead set, or enqueue Compliance::DataDeletionJob.perform_async(<deletion_request_id>); the job resumes any processing row past its grace period. Compliance::AccountTerminationJob records deleted_files with the count; held files become a files_erasure_held entry (ids and reasons, never filenames) and the termination proceeds, since no retry clears a policy hold; an operational failure raises with a bounded sample of ids plus a count, the termination reverts to grace_period, and the next sweep retries — a permanent provider fault retries every sweep, loudly, rather than completing a termination that left personal data in object storage. An older server build that still answers erased: false on the accounts route is recorded as files_erasure_skipped, exactly as before; an older server that has no users/:id/files route at all answers 404, which the deletion job records as skipped / no_erasure_path.

        Rollout constraint: the worker module must not lag the backend module for this change. A new worker against an old server is safe in both paths: the deletion job records files as skipped on the 404, and the termination job records files_erasure_skipped on erased: false, exactly the pre-erasure behaviour. An old worker against a new server under-erases silently and records a completed termination with files left: on termination it makes one DELETE, reads only count and ignores remaining, so an account with 300 personal files erases 50 and completes; on a full deletion request it does not walk files at all (the type is not in its list), so the request completes with the files untouched and nothing in the log. Deploy hub-worker before (or together with) hub-backend; never let it lag. After the backend lands, re-initialize every local, NFS and SMB file storage with its share mounted (Admin → Storage → test / initialize): until a store carries the liveness marker, every missing blob on it is refused as store_not_initialized and the deletion request parks in retry with that reason in error_message. Azure note: a 404 from a blob delete counts as "already removed" only for BlobNotFound; ContainerNotFound (the store itself is gone) fails closed.

        Other residues, beside the sibling categories' own. Referents are asked per file, inside its transaction, after the row lock, so a platform pointer set on a file after the batch was selected is still caught — including on the one column no foreign key backs — and the audit row's referents_consulted lists the handlers that answered, so "no handler registered" is distinguishable from "nothing held". Uploads that land after the final remaining count is read are not part of that run; the request completes with them in place, the same way a notification sent after the communications erasure survives it. Storage-provider logs no longer name the storage key (which embeds the filename) on the erasure path — the providers' delete_file log lines carry the file object id — but logs written before this change, and any other provider operation's lines, may still hold keys; treat log retention as its own regime.

      • activity and analytics are withdrawn: no category-level erasure path was ever built for either, and no model in core is scoped to either category. This is not "no user-associated activity data exists" — it plainly does (Devops::DockerActivity has belongs_to :triggered_by, class_name: "User"; McpToolExecution, Ai::AgentExecution, Ai::Conversation, Ai::Message, Ai::RagQuery, Ai::ContextAccessLog and KnowledgeBase::ArticleView are all user-associated), and nothing erases any of it. Deciding which of those constitute a data subject's "activity" for Article 17 is an unanswered policy question, so the category must not be offered as though it had been answered.

      • The model now rejects every withdrawn type on create, so a request for a category the platform cannot erase is refused up front (422) instead of accepted and later logged as skipped. The validation is create-only, so the worker's in-flight status writes on pre-existing rows are unaffected; those legacy rows still take the no_erasure_path skip path.

    • Data deletion — data_types_to_retain is now binding for profile. Compliance::DataDeletionJob#process_full_deletion ran an unconditional anonymize_user after its per-type loop, so retaining profile produced a retention_log entry saying the profile was retained while the user row was anonymized in place anyway — the deletion record contradicted what actually happened to the data subject. (It also ran the same anonymize twice on every other request.) The loop's own profile branch is now the only anonymize call. A full deletion now honours data_types_to_retain for every DELETABLE_DATA_TYPES member (closed by IMP-ca3551dd27be). audit_logs and payments used to ignore retention by a different mechanism: they were missing from the job's list and were anonymized unconditionally after the loop, so retaining either produced no retention_log entry and the anonymize ran anyway. The job now walks all six members through the one per-type loop. A retained type gets a retention_log entry with its reason and its endpoint is never called. A type that is not retained is erased or anonymized once and recorded in the deletion_log, and a failure there is a per-type failed entry. Two gaps remain. The anonymize deletion type still ignores data_types_to_retain and anonymizes the profile, audit logs and payments unconditionally; this is tracked as a separate follow-up offer. The default retain list (financial_records, tax_documents, legal_agreements), which every request filed by a data subject carries, names no erasable category, so on its own it retains nothing.
  • Right of access: data export endpoint (/api/v1/users/me/data_export — verify present in your release) bundles the user's personal data into a JSON download. Compliance::DataExportJob (in worker/) assembles the archive from the Api::V1::Internal::DataExportsController endpoints, one per data type.
    • Files (IMP-8aab38f3ad62): GET /api/v1/internal/accounts/:account_id/export/files?user_id= exports the FileManagement::Object rows the data subject uploaded in their account: keyed on account_id AND uploaded_by_id. It is not every file in the account, because a co-member's uploads are that co-member's personal data. Soft-deleted rows are included with deleted_at, since the platform still holds them. The export carries file metadata, not file content: filename, content type, size, category, visibility, version, timestamps, and exif_data, which can hold GPS/location and is the subject's personal data. Internal locators such as storage_key are left out. Providing copies of the file content itself (Art. 15(3)) is an open item under follow-up offer 01a0d741-fca8-7751-8fc8-91dc2cb76a20. user_id is required, and a user who is not a member of the account gets 404. Before this fix, the endpoint was gated on account.respond_to?(:files), which is always false, so every export's files section was empty while the endpoint reported success.
    • Activity — withdrawn (IMP-8aab38f3ad62): activity is no longer in DataManagement::ExportRequest::EXPORTABLE_DATA_TYPES, and GET /api/v1/internal/users/:user_id/export/activity is removed. It was gated on user.respond_to?(:activities), which is also always false, so it answered an always-empty success. The per-user trail it could have meant, AuditLog, is already exported as audit_logs. As on the erasure side, withdrawal is not a claim that no other activity-like data exists (for example AI conversations or MCP tool executions); which of those belong in an Article 15 export is still an open policy question.
    • Billing sections (payments, invoices, subscriptions) come from the billing extension. In core mode the endpoints answer data: [] with meta: { available: false, reason: "no_export_provider_installed" }, so the absence is reported instead of reading as "this account has no records".
    • The archive manifest — what the job writes when it runs (nothing in the product queues it today; see the known gap below). export_info.manifest records, per requested type, either exported with the real record count, or why the section is absent: unavailable (no provider installed), not_exportable (no export path exists — currently settings, communications, and activity on a request filed before its withdrawal), or failed. The job's completion log line lists the per-type counts (completed: profile=1 files=3 …), and a separate warning names every absent section; it previously logged "completed successfully" whether the archive held the subject's data or nothing.
    • Known gap — the export pipeline still cannot deliver an archive to a data subject. The endpoints above answer correctly, and IMP-0310a1351dab fixed the job itself: it reads the nested data.data_export_request response and drives status through the controller's action_type dispatch, so a request that is queued does reach completed. What is still missing is everything around it (tracked as follow-up offer 01a0d741-fca8-7751-8fc8-91dc2cb76a20):
      1. A data subject's own request is never enqueued. Api::V1::PrivacyController#request_export creates the DataManagement::ExportRequest and returns; it does not queue Compliance::DataExportJob.
      2. Nothing creates a termination export either. Account::Termination.initiate is the code that requests and queues a termination's export, and it has no caller: no controller, job or extension invokes it, so a termination exists only if it is created out of band.
      3. The archive cannot be served, and nobody is told where it is. The job writes to Dir.tmpdir on the worker host; PrivacyController#download_export serves only Rails.root/tmp/data_exports on the server. The server generates its own download token and the job cannot read it back, so the "export ready" notification carries download_token: nil. A completed export is therefore not downloadable, and a termination waits out the 7-day window instead.
    • Archive residue on the worker host. The archive is the subject's full personal-data export and lives in the worker's tmp directory, which the server cannot reach. The worker now removes it: Compliance::AccountTerminationJob deletes the archive paths the server reports for the export rows it deleted (best effort, and only for a regular file whose resolved location is inside the export directory: a symlink is refused, and nothing is deleted at all if that directory is not a real directory owned by the worker's user that no group or other user can write into), and DataExportJob deletes an archive written by a run that then failed. Two cases are not covered: an archive written on a different worker host than the one that runs the termination sweep, and an archive from a request nothing ever deletes. Both stay until the host's tmp is cleared. The server's own cleanup is best effort too: a file it cannot remove is logged and the row is still deleted.
  • Audit log of access: AuditLog.where(user_id: X) evidences who touched what data and when. The read actions are logged for sensitive resources.
  • Privacy by default: new accounts have minimum visibility; the operator opts into broader sharing.

What the operator must add:

  • Data Processing Agreement (DPA) template with your customers.
  • Subject access request (SAR) workflow — Powernode provides the data, you provide the workflow.
  • Cookie consent banner for the frontend if collecting analytics cookies — not shipped.
  • PII minimization review per your jurisdiction.

HIPAA

Powernode is not HIPAA-ready out of the box. The platform can technically store PHI in any text column, but lacks:

  • BAA-compliant infrastructure agreements (operator-owned)
  • Designated HIPAA-trained admin role separate from super_admin
  • Encryption-at-rest on every persistent volume by default (operator-configured)
  • Automatic PHI tagging in audit logs (would require schema additions)

If you must run on Powernode in a HIPAA context, plan for a custom audit + additional controls. The platform doesn't actively prevent HIPAA-grade use, but it doesn't help.

PCI DSS

server/config/initializers/pci_compliance.rb scaffolds:

  • Parameter filtering for card_number, cvv, expiry, etc. in Rails request logs
  • HSTS + secure-cookie defaults
  • Rate limiting on payment endpoints

What's NOT shipped:

  • Network segmentation for the cardholder-data environment (operator-owned)
  • Quarterly external ASV scans (third-party)
  • Penetration testing artifacts (third-party)
  • Annual SAQ-D or ROC

Critically, Powernode does not store full card data — Stripe/PayPal handle that. The platform's PCI scope is "merchant of record using a PCI-compliant processor" (SAQ-A territory) unless you've done something custom.

SOC 2

Powernode supports SOC 2 Type II evidence collection via the audit log + access control system. Trust services criteria evidence map:

Criterion Where to find evidence in Powernode
Security: access control `AuditLog.where(action: ~/login
Security: change management git log (code), AuditLog.where(action: ~/admin_settings_update/) (config), migration history
Availability Service metrics in Prometheus + status checks (see observability.md)
Confidentiality: encryption TLS via reverse proxy (operator), at-rest via Postgres TDE (operator), secret storage via Vault
Processing integrity AuditLog, request/response logs in Loki, per-job result records in Sidekiq
Privacy See GDPR / CCPA section

SOC 2 reporting itself requires an external auditor and 12+ months of evidence. Powernode generates the evidence; you contract the auditor.

Encryption posture

Layer Default Notes
TLS in flight Operator-configured at reverse proxy (Traefik default — see project_reverse_proxy_state memory) Verify HSTS enabled
Postgres at-rest Off (operator may enable LUKS/filesystem encryption / Cloud SQL CMEK / etc.) Required for HIPAA, encouraged for PII
Vault transit secrets AES-256-GCM by default Vault handles this internally
Backup files Plaintext .sql.gz by default Add GPG or use SSE-KMS on S3 for encryption-at-rest
Worker file storage Plaintext on local filesystem Generated reports may contain sensitive data — encrypt the volume

Access controls

  • Permissions, not roles: every protected operation checks has_permission?('name'). Roles bundle permissions but the actual gate is per-permission.
  • Worker JWTs are short-lived (5 min) with a 4-min cache window. Compromise impact bounded.
  • Vault-stored credentials rotate independently of code deploys.
  • Kill switch: any administrator can halt all AI activity globally (see incident-response.md#the-kill-switch).
  • Account scoping: API endpoints default to current_user.account.scope for data isolation. Cross-account access requires explicit analytics.global-style permissions.

Audit log evidence

The audit_logs table is the platform's primary compliance evidence store. Schema:

Column Use
action What happened (from AuditActions::ALL_ACTIONS allowlist; rejections fail validation)
user_id Who did it (nullable for system/worker actions)
account_id Scope of the action
resource_type + resource_id What was touched
old_values + new_values State diff (subset of changed columns)
metadata Free-form context
severity + risk_level Triage hints
source Origin: user, system, worker, api
ip_address + user_agent Request provenance
created_at Timestamp

Adding a new action: extend the appropriate *_ACTIONS constant in server/app/models/concerns/audit_actions.rb and include it in ALL_ACTIONS. Forgetting this causes silent log-write failures, which surface as missing evidence during audits — see the recent REPORT_REQUEST_ACTIONS addition for the pattern.

Where to look during an audit

Auditor asks for... Locate via
"Show me who accessed X" AuditLog.where(resource_id: 'X')
"Show me all admin actions in Q1" AuditLog.joins(:user).where(action: ADMIN_PATTERNS, created_at: Q1).order(:created_at)
"Show me the change-management trail for production deploys" git log --since=... + CI pipeline records (see docs/operations/production-deployment.md)
"Show me failed login attempts" AuditLog.where(action: 'login_failed', created_at: <window>)
"Show me data exports" AuditLog.where(action: 'gdpr.data_export', created_at: <window>) (verify action present in your release)
"Show me kill switch activations" Ai::KillSwitchEvent.where(event_type: 'halt') (or account.ai_kill_switch_events.halts) on the ai_kill_switch_events table
"Prove this user's data was deleted" Compliance::AccountTerminationJob records; AuditLog.where(action: 'gdpr.account_terminated', user_id: X)

See also

  • incident-response.md — security event response
  • postgres-backup.md — backup policy + retention
  • observability.md — log retention configuration
  • server/app/models/concerns/audit_actions.rb — canonical action list
  • server/config/initializers/pci_compliance.rb — PCI parameter filtering

Last verified: 2026-09-18