Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/howto/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ A collection of how-to guides.
define-schemas
resolve-keyed-by
use-fetches
use-secrets
docker
create-actions
send-notifications
Expand Down
98 changes: 98 additions & 0 deletions docs/howto/use-secrets.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
Use Secrets
===========

Tasks often need secrets such as API tokens or private keys. Taskgraph can
have ``run-task`` fetch these automatically.

Using Secrets
-------------

Secrets are implemented as part of the :mod:`~taskgraph.transforms.run`
transforms, and are fetched by ``run-task``. So make sure your task uses those
transforms and a ``run-task`` based ``run.using``:

.. code-block:: yaml

loader: taskgraph.loader.transform:loader

transforms:
- taskgraph.transforms.run:transforms
- taskgraph.transforms.task:transforms

Then list the secrets in the task's ``secrets`` key:

.. code-block:: yaml

tasks:
deploy:
run:
using: run-task
command: ./deploy.sh
secrets:
- type: github-repo-token
app: my-github-app
env: GITHUB_TOKEN
- type: secret
name: project/myproject/level-3/deploy
key: privkey
file: ~/.ssh/id_rsa

Here ``deploy.sh`` will find a GitHub token in the ``GITHUB_TOKEN`` env and a
``~/.ssh/id_rsa`` file containing a private ssh key.

The transforms add the scopes each secret needs and enable the
``taskclusterProxy`` feature, then pass the list to ``run-task`` in the
``RUN_TASK_SECRETS`` environment variable. No two secrets can be exported to
the same variable or file, and a secret can't be exported to a variable that is
also set in ``worker.env``.

Substituting Values
-------------------

Secret names often depend on the level or project. Use the
:ref:`task_context <task_context transforms>` transforms, before the run
transforms, to substitute parameters into them:

.. code-block:: yaml

transforms:
- taskgraph.transforms.task_context
- taskgraph.transforms.run:transforms
- taskgraph.transforms.task:transforms

tasks:
deploy:
task-context:
from-parameters:
level: level
project: project
substitution-fields: [secrets]
run:
using: run-task
command: ./deploy.sh
secrets:
- type: secret
name: project/{project}/level-{level}/deploy
env: DEPLOY_TOKEN

Every string in ``secrets`` is formatted, so write a literal ``{`` or ``}`` as
``{{`` or ``}}``. ``secrets`` can also be keyed by a value as a whole, but not
entry by entry.

Secret Types
------------

``github-repo-token``
An installation token of the GitHub App ``app``. ``repo`` defaults to the
``base_repository`` parameter, ``permissions`` to ``{"contents": "read"}``
and ``env`` to ``GITHUB_TOKEN``. Adds an
``auth:github-repo-token:<app>/<owner>/<repo>:<permission>:<level>`` scope
for each permission.

``secret``
A secret from Taskcluster's secrets service, exported to ``env`` or written
to ``file``. ``key`` exports a single key of the secret instead, using dots
for nested keys such as ``foo.bar``. Adds a ``secrets:get:<name>`` scope.

See :ref:`run-task-secrets` for the details of each type, including what
happens when a secret can't be fetched.
52 changes: 45 additions & 7 deletions docs/reference/run-task.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,24 @@ the task can both use them. List them as JSON in ``RUN_TASK_SECRETS``:
- type: github-repo-token
app: read
repo: https://github.com/myorg/myrepo
- type: secret
name: project/myproject/level-3/deploy
key: token
env: DEPLOY_TOKEN

The only type so far is ``github-repo-token``. It asks Taskcluster's auth service
for an installation token of the GitHub App ``app``, scoped to ``repo``. Its
keys are:
Each entry has a ``type``, which decides its other keys. An unknown ``type`` or
key, or invalid JSON, fails the task. A variable that is already set is never
replaced.

Rather than setting ``RUN_TASK_SECRETS`` by hand, tasks using the
:mod:`~taskgraph.transforms.run` transforms can list secrets in their
``secrets`` key. See :doc:`/howto/use-secrets`.

github-repo-token
~~~~~~~~~~~~~~~~~

Asks Taskcluster's auth service for an installation token of the GitHub App
``app``, scoped to ``repo``. Its keys are:

``app`` (required)
The name of the GitHub App registered with Taskcluster.
Expand All @@ -37,17 +51,41 @@ keys are:
The token's permissions. Defaults to ``{"contents": "read"}``.
``env``
The environment variable to export the token in. Defaults to
``GITHUB_TOKEN``. A variable that is already set is never replaced.
``GITHUB_TOKEN``.

``run-task`` also keeps the token for its own GitHub requests. The task needs
the ``taskclusterProxy`` feature and an
``auth:github-repo-token:<app>/<owner>/<repo>:<permission>:<level>`` scope for
each permission. If a request fails, for example because a pull request from a
fork lacks the scope, ``run-task`` logs a warning and carries on without the
token. The same happens when the auth service rejects the request, such as for
an unknown ``app`` or invalid ``permissions``. Invalid JSON, an unknown ``type``
or key, or a ``repo`` that isn't on GitHub fails the task. Installation tokens
expire after an hour.
an unknown ``app`` or invalid ``permissions``. A ``repo`` that isn't on GitHub
fails the task. Installation tokens expire after an hour.

secret
~~~~~~

Fetches a secret from Taskcluster's secrets service. Its keys are:

``name`` (required)
The name of the secret.
``env``
The environment variable to export the secret in.
Comment thread
ahal marked this conversation as resolved.
Resolved
``file``
The file to write the secret to. Relative paths are relative to the
task's working directory. Missing parent directories are created, and on

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a potential conflict here?
If the secret is fetched into the same path as the checkout, and happens before checkout, would the checkout fail? Should we add a check for this? (Seems corner case AND self-inflicted, but still)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking for this adds a moderate amount of complexity, I don't think it's worth it tbh.. Just.. don't do that :p

POSIX systems only the task's user can read the file. An existing file is
never replaced.
``key``
A key of the secret to export, rather than the whole secret. Dots select
keys of nested objects, so ``foo.bar`` exports ``secret["foo"]["bar"]``.

Exactly one of ``env`` or ``file`` is required. A string value is written as
is, and any other value as JSON. If the variable is already set or the file
already exists, the secret isn't fetched. The task needs
the ``taskclusterProxy`` feature and a ``secrets:get:<name>`` scope. Unlike
``github-repo-token``, a failure to fetch the secret, or a missing ``key``,
fails the task.

.. _run-task-base-revisions:

Expand Down
151 changes: 119 additions & 32 deletions src/taskgraph/run-task/run-task
Original file line number Diff line number Diff line change
Expand Up @@ -1195,6 +1195,122 @@ def fetch_github_token(app, owner, repo, permissions):
return None


def fetch_taskcluster_secret(name):
"""Return the value of a secret from Taskcluster's secrets service."""
try:
with urllib.request.urlopen(SECRET_BASEURL_TPL.format(name), timeout=10) as res:
return json.load(res)["secret"]
except (urllib.error.URLError, OSError, ValueError, KeyError, TypeError) as e:
raise RuntimeError(
f"RUN_TASK_SECRETS: unable to fetch secret {name!r}: {e}"
) from None


def _export_secret(envvar, value):
if envvar in os.environ:
print_line(b"secrets", f"{envvar} is already set; not replacing it\n".encode())
else:
os.environ[envvar] = value


def _check_secret_keys(entry, allowed):
unknown = set(entry) - {"type"} - allowed
if unknown:
raise RuntimeError(f"RUN_TASK_SECRETS: unknown keys {sorted(unknown)}")


def _fetch_github_repo_token(entry):
_check_secret_keys(entry, {"app", "repo", "permissions", "env"})

repo = entry.get("repo")
github_repo = isinstance(repo, str) and _github_owner_repo(repo)
if not github_repo:
raise RuntimeError("RUN_TASK_SECRETS: 'repo' must be a GitHub repository")

owner, name = github_repo
token = fetch_github_token(
entry.get("app"),
owner,
name,
entry.get("permissions", {"contents": "read"}),
)
if not token:
return
GITHUB_TOKENS.setdefault((owner.lower(), name.lower()), token)
print_line(b"secrets", f"fetched github-repo-token for {owner}/{name}\n".encode())
_export_secret(entry.get("env", "GITHUB_TOKEN"), token)


def _write_secret_file(path, value):
"""Write a secret to a new file only the task user can read (on POSIX)."""
path.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(fd, "w") as fh:
fh.write(value)
except BaseException:
# Don't leave a partially written secret behind.
path.unlink(missing_ok=True)
raise


def _fetch_secret(entry):
_check_secret_keys(entry, {"name", "key", "env", "file"})

name = entry.get("name")
envvar = entry.get("env")
path = entry.get("file")
key = entry.get("key")
if not isinstance(name, str) or not name:
raise RuntimeError("RUN_TASK_SECRETS: 'name' must be a secret name")
if (envvar is None) == (path is None):
raise RuntimeError("RUN_TASK_SECRETS: exactly one of 'env' or 'file' is required")
if envvar is not None and (not isinstance(envvar, str) or not envvar):
raise RuntimeError("RUN_TASK_SECRETS: 'env' must be a variable name")
if path is not None and (not isinstance(path, str) or not path):
raise RuntimeError("RUN_TASK_SECRETS: 'file' must be a path")
if key is not None and not isinstance(key, str):
raise RuntimeError("RUN_TASK_SECRETS: 'key' must be a string")

# Relative paths are relative to the task's working directory.
if path is not None:
path = Path(os.path.abspath(os.path.expanduser(path)))

# Don't fetch a value that wouldn't be used.
if envvar is not None and envvar in os.environ:
print_line(
b"secrets", f"{envvar} is already set; not fetching {name}\n".encode()
)
return
if path is not None and path.exists():
print_line(
b"secrets", f"{path} already exists; not fetching {name}\n".encode()
)
return

value = fetch_taskcluster_secret(name)
if key is not None:
# Dots separate the keys of nested objects, e.g. "foo.bar".
for part in key.split("."):
if not isinstance(value, dict) or part not in value:
raise RuntimeError(
f"RUN_TASK_SECRETS: secret {name!r} has no key {key!r}"
)
value = value[part]
print_line(b"secrets", f"fetched secret {name}\n".encode())
value = value if isinstance(value, str) else json.dumps(value)
if path is not None:
_write_secret_file(path, value)
Comment thread
ahal marked this conversation as resolved.
Resolved
else:
_export_secret(envvar, value)


SECRET_FETCHERS = {
"github-repo-token": _fetch_github_repo_token,
"secret": _fetch_secret,
}


def fetch_secrets():
"""Fetch the secrets listed in RUN_TASK_SECRETS and export them."""
if not (raw := os.environ.get("RUN_TASK_SECRETS")):
Expand All @@ -1207,41 +1323,12 @@ def fetch_secrets():
raise RuntimeError("RUN_TASK_SECRETS must be a JSON list of objects")

for entry in entries:
if entry.get("type") == "github-repo-token":
unknown = set(entry) - {"type", "app", "repo", "permissions", "env"}
if unknown:
raise RuntimeError(f"RUN_TASK_SECRETS: unknown keys {sorted(unknown)}")

repo = entry.get("repo")
github_repo = isinstance(repo, str) and _github_owner_repo(repo)
if not github_repo:
raise RuntimeError("RUN_TASK_SECRETS: 'repo' must be a GitHub repository")

owner, name = github_repo
envvar = entry.get("env", "GITHUB_TOKEN")
token = fetch_github_token(
entry.get("app"),
owner,
name,
entry.get("permissions", {"contents": "read"}),
)
if not token:
continue
GITHUB_TOKENS.setdefault((owner.lower(), name.lower()), token)
print_line(
b"secrets", f"fetched github-repo-token for {owner}/{name}\n".encode()
)
if envvar in os.environ:
print_line(
b"secrets",
f"{envvar} is already set; not replacing it\n".encode(),
)
else:
os.environ[envvar] = token
else:
fetcher = SECRET_FETCHERS.get(entry.get("type"))
if not fetcher:
raise RuntimeError(
f"RUN_TASK_SECRETS: unsupported type {entry.get('type')!r}"
)
fetcher(entry)


def github_compare(owner, repo, base, head):
Expand Down
Loading
Loading