Skip to content

feat: GSM Secrets - Read secret inputs through the store at launch - #475

Open
yuechao-qin wants to merge 5 commits into
gsm-t1-store-hookfrom
gsm-t2-read-at-launch
Open

yuechao-qin wants to merge 5 commits into
gsm-t1-store-hookfrom
gsm-t2-read-at-launch

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

A launch now finds a secret input in the secret store when it is no longer in the database: give OrchestratorService_Sql a secret_store, and the launch reads the database first, then the store. Without a store, nothing changes.

In the database (secret table)? In the secret store? The task gets
✅ not checked the database value
❌ ✅ the secret store value
❌ ❌ today's "User alice does not have secret TOKEN" error
❌ ⚠️ can't be reached a system error carrying the secret store's error

With no secret_store given, only the first and third rows can happen, exactly as today.

secret = session.get(bts.Secret, (user_id, secret_name))
secret_value = secret.secret_value if secret else None
if secret_value is None and self._secret_store:
    secret_value = self._secret_store.get(user_id=user_id, secret_name=secret_name)

Why

With #474, the secret routes move each row into the store on first use, so a launch can no longer rely on the table alone.

A launch only reads: get takes no session and never moves a row, so the launch transaction stays as it is today, with no commit and no row lock. The row comes first because a row that still exists holds the same value as any copy the store has: a move saves to the store before it deletes the row, in one commit.

Verification

  • tests/test_secret_store.py, launching a one-task pipeline through OrchestratorService_Sql with a mock launcher: a row reaches the task with or without a store and stays in the table, with the store left empty; a secret only in the store is read from it; a missing secret is today's user error with or without a store, and nothing is launched; a failing get fails the launch with the store's error.
  • Mutation-checked: never asking the store, reading it with a session, ignoring the row, and not failing on a missing secret each turn tests red.
  • Full suite: 664 passed, 22 failed — the same 22 in tests/instrumentation/test_execution_tracing.py fail on master. Also passes on Python 3.14.

# Conflicts:
#	cloud_pipelines_backend/secret_value_store.py
#	tests/test_secret_value_store.py
…ore, at launch

A launch only reads: get takes no session, so nothing moves or commits during a launch.
@yuechao-qin
yuechao-qin requested a review from a team November 19, 2026 04:27
# Conflicts:
#	tests/test_secret_store.py
sleep_seconds_between_queue_sweeps: float = 1.0,
output_data_purge_duration: datetime.timedelta = None,
*,
secret_store: secret_store.SecretStore | None = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rename to internal _secret_store for a bit of time. It's trivial to rename to public secret_store in any time in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants