Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,47 @@ result, execution of the graph is performed as a parallelised breadth first sear
This package contains some creative APIs which work around limitations in Golang generics; chief
of these is the `ID` type used in place of slices of keys because Golang slices are invariant.

## Overriding tasks

A graph's tasks often create things for themselves: clients of other services, say. That makes a
whole graph hard to test, as there is no way to give it a fake client. `RunWithOverrides` replaces the
task which provides a key:

```go
tg.RunWithOverrides(ctx, graph, inputs, []tg.Binding{keyClient.Bind(fakeClient)})
```

The task providing `keyClient` is not run, and the tasks which depend on it see the fake. This makes
a graph's dependencies a matter of which keys its tasks provide: anything which a test may want to
replace should be a key of its own.

It is meant for tests: `Graph.Run` has no way to replace tasks, so code which runs a graph and needs to
be tested can take a function for running graphs, and be handed `Overrider.Run` by its tests.

* An input for a key which a task provides is still an error, so a key is only ever replaced on
purpose. The same goes for overriding one key twice, or overriding a key which is also an input.
* If a task provides several keys they must all be overridden, or none (`ErrPartialOverride`).
Otherwise the task would still have to run, with all of its effects, with some of its results
thrown away.
* Overriding a key which no task provides is an error (`ErrUnknownOverride`), which catches
overrides left behind by a task which has since been renamed or removed. `CheckWithOverrides`
finds these without running the graph.
* The override may be absent: `keyClient.BindError(err)` replaces the task with a failure, to see how
the tasks which depend on it cope.
* The overrides are included in the `Binder` returned, as the results of the tasks they replace would
have been.
* Inputs which only a replaced task needed are not required.
* Tasks which only produced inputs for a replaced task still run, as the graph can't know that its
caller doesn't want their results.
* Overrides apply to the graph they are given. A graph run as a task with `AsTask` is replaced as a
whole, by overriding every key it exposes, but its tasks can't be overridden individually.
* An `Overrider` holds one set of overrides for something which runs several graphs, such as an
operator with a graph to create and a graph to delete. Each graph is run with the overrides for the
keys it provides, and `Overrider.Check` makes sure that every override is for some graph.

`taskgraphtest.Test` and `taskgraphtest.Suite` take `Overrides`, those of a `Test` replacing those
of its `Suite` for the same key.

## Limitations and sharp edges

* Taskgraph has no deadlock detection. In theory the fact that the graph is acyclic should prevent
Expand Down
144 changes: 144 additions & 0 deletions override.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
package taskgraph

import (
"context"
"errors"
"fmt"
"sort"
"strings"
)

var (
// ErrUnknownOverride is returned by RunWithOverrides and CheckWithOverrides if an override is
// given for a key which no task in the graph provides. This is almost always a stale override,
// left over from a task which has since been renamed or removed.
ErrUnknownOverride = errors.New("override of a key which no task provides")

// ErrPartialOverride is returned by RunWithOverrides and CheckWithOverrides if overrides cover
// some, but not all, of the keys which a task provides. The task would still have to run, with
// all of its effects, only to have some of its results discarded, which is very unlikely to be
// what was intended. Override all of a task's keys, or none of them.
ErrPartialOverride = errors.New("override of only some of the keys which a task provides")
)

// RunWithOverrides runs the graph, as Graph.Run does with the inputs, with the tasks which provide
// the keys of the overrides replaced by them. It is the way to substitute what the tasks of a graph
// would otherwise create for themselves, such as clients of other services, in a test:
//
// taskgraph.RunWithOverrides(ctx, graph, inputs, []taskgraph.Binding{keyClient.Bind(fakeClient)})
//
// The task providing the key is not run. If the task provides other keys as well, they must all be
// overridden too (ErrPartialOverride), so that the real task's effects can never happen unnoticed.
// Tasks which have no other purpose than to produce the inputs of a task which is not run are still
// run, as are tasks which depend on the overridden key; they see the override as if the task which
// it replaces had produced it. The overrides are included in the Binder returned, as the replaced
// tasks' results would have been, and inputs which only a replaced task needed are not required.
//
// An override may be absent, which replaces the task with a failure:
//
// keyClient.BindError(errUnreachable)
//
// Overriding a key which no task provides is an error (ErrUnknownOverride), as is overriding the same
// key twice, or overriding a key which is also given as an input. An input for a key which a task
// provides is still an error (ErrDuplicateBinding), so a key can only ever be replaced on purpose.
//
// Overrides apply to the graph's own tasks. A graph run as a task with Graph.AsTask is replaced as a
// whole, by overriding every key it exposes, but its tasks can't be overridden individually.
//
// This is a function, not a method of Graph, so that the Graph interface, which other code may
// implement, is not changed. It only works on graphs made by New.
func RunWithOverrides(
ctx context.Context,
g Graph,
inputs, overrides []Binding,
) (Binder, error) {
graph, err := asGraph(g)
if err != nil {
return nil, err
}
replaced, err := graph.withOverrides(inputs, overrides)
if err != nil {
return nil, err
}
return replaced.Run(ctx, inputs...)
}

// CheckWithOverrides is Graph.Check, with the overrides RunWithOverrides would be given: it finds
// unknown, duplicated and partial overrides, and which inputs are still needed, without running
// anything.
func CheckWithOverrides(g Graph, inputs, overrides []Binding) error {
graph, err := asGraph(g)
if err != nil {
return err
}
replaced, err := graph.withOverrides(inputs, overrides)
if err != nil {
return err
}
return replaced.Check(inputs...)
}

// An Overrider runs graphs with one set of overrides. It is for something which runs several graphs,
// such as an operator with a graph to create and one to delete, and which can only be given one set:
// each graph is run with those of the overrides which are for keys it provides, and Check makes sure
// that every override is for some graph. Run does not report an override which is for none of the
// graphs, as it can't tell that from one for another graph, so Check is how to find a stale one.
type Overrider struct {
overrides []Binding
}

// NewOverrider returns an Overrider with the overrides.
func NewOverrider(overrides ...Binding) *Overrider {
return &Overrider{overrides: overrides}
}

// Run runs the graph with the inputs, and those of the Overrider's overrides which are for keys the
// graph provides. Its signature is that of a function for running a graph, so it can be handed to
// code which runs graphs.
func (o *Overrider) Run(ctx context.Context, g Graph, inputs ...Binding) (Binder, error) {
graph, err := asGraph(g)
if err != nil {
return nil, err
}
var own []Binding
for _, override := range o.overrides {
if graph.allProvided.Contains(override.ID()) {
own = append(own, override)
}
}
return RunWithOverrides(ctx, g, inputs, own)
}

// Check returns an error if any of the overrides is for a key which none of the graphs provide, which
// is nearly always one left behind by a task which has been renamed or removed.
func (o *Overrider) Check(graphs ...Graph) error {
var unknown []string
for _, override := range o.overrides {
provided := false
for _, g := range graphs {
if graph, err := asGraph(g); err == nil && graph.allProvided.Contains(override.ID()) {
provided = true
break
}
}
if !provided {
unknown = append(unknown, override.ID().String())
}
}
if len(unknown) > 0 {
sort.Strings(unknown)
return fmt.Errorf("%w: %s", ErrUnknownOverride, strings.Join(unknown, ", "))
}
return nil
}

func asGraph(g Graph) (*graph, error) {
graph, ok := g.(*graph)
if !ok {
return nil, fmt.Errorf(
"overrides can only be applied to a Graph made by taskgraph.New, not %T",
g,
)
}
return graph, nil
}
98 changes: 98 additions & 0 deletions override_plan.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package taskgraph

import (
"context"
"fmt"
"sort"
"strings"

set "github.com/deckarep/golang-set/v2"
)

// withOverrides returns a graph like g, but with each task whose keys are all overridden replaced by a
// task which provides the overriding bindings. The graph is g itself if there are no overrides.
//
// A replacement has no dependencies, so the tasks which fed the task it replaces have nothing to
// signal, and inputs which only the replaced task needed are not required. Those tasks still run, as
// the graph can't know that its caller doesn't want their results.
//
// An override is an error if it is for a key which is also an input or is overridden twice, as that
// is a mistake in which one is meant; if no task provides its key, which is nearly always a task
// which has been renamed or removed; or if it is for only some of the keys which a task provides,
// as the task would then have to run for the others and its effects could not be ruled out.
func (g *graph) withOverrides(inputs, overrides []Binding) (*graph, error) {
if len(overrides) == 0 {
return g, nil
}

given := set.NewSet[ID]()
for _, input := range inputs {
given.Add(input.ID())
}
overriddenBy := map[ID]Binding{}
for _, o := range overrides {
if given.Contains(o.ID()) {
return nil, wrapStackErrorf(
"override of a key which is also an input: %w: %q", ErrDuplicateBinding, o.ID())
}
if _, twice := overriddenBy[o.ID()]; twice {
return nil, wrapStackErrorf("key overridden twice: %w: %q", ErrDuplicateBinding, o.ID())
}
overriddenBy[o.ID()] = o
}

var unknown []string
for id := range overriddenBy {
if !g.allProvided.Contains(id) {
unknown = append(unknown, id.String())
}
}
if len(unknown) > 0 {
sort.Strings(unknown)
return nil, wrapStackErrorf("%w: %s", ErrUnknownOverride, strings.Join(unknown, ", "))
}

var tasks []TaskSet
var partial []string
for _, t := range g.tasks {
var covered []Binding
var uncovered []string
for _, id := range t.Provides() {
if o, ok := overriddenBy[id]; ok {
covered = append(covered, o)
} else {
uncovered = append(uncovered, id.String())
}
}
switch {
case len(covered) == 0:
tasks = append(tasks, t)
case len(uncovered) == 0:
tasks = append(tasks, replacement(t, covered))
default:
partial = append(partial, fmt.Sprintf(
"task %s also provides [%s], which are not overridden",
t.Name(), strings.Join(uncovered, ", ")))
}
}
if len(partial) > 0 {
sort.Strings(partial)
return nil, wrapStackErrorf("%w: %s", ErrPartialOverride, strings.Join(partial, "; "))
}

replaced, err := New(g.name, WithTasks(tasks...), WithTracer(g.tracer), WithLogger(g.logger))
if err != nil {
return nil, err
}
return replaced.(*graph), nil
}

// replacement is a task in place of t which provides the overrides for all of the keys of t.
func replacement(t Task, overrides []Binding) Task {
return NewTask(
t.Name()+" (overridden)",
func(context.Context, Binder) ([]Binding, error) { return overrides, nil },
nil,
t.Provides(),
)
}
Loading
Loading