Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion crates/accuracy/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ pub(crate) fn satisfies(guarantee: &AccuracyGuarantee, target: &AccuracyTarget)
/// error has no distribution-free value error), which the default model
/// does not make: it does not satisfy. The scope is compared separately
/// ([`scope_meets`]).
fn metric_meets(guarantee: AccuracyMetric, target: AccuracyMetric) -> bool {
pub(crate) fn metric_meets(guarantee: AccuracyMetric, target: AccuracyMetric) -> bool {
guarantee == target
|| (guarantee == AccuracyMetric::ABSOLUTE_STREAM_L2
&& target == AccuracyMetric::ABSOLUTE_STREAM_L1)
Expand Down
16 changes: 15 additions & 1 deletion crates/accuracy/src/model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
//! guarantee says.

use crate::OperandDomain;
use asap_types::ir::properties::{AccuracyGuarantee, AccuracyTarget};
use asap_types::ir::properties::{AccuracyGuarantee, AccuracyMetric, AccuracyTarget};
use asap_types::ir::{OperatorNode, QueryRoot};

/// The deployment's accuracy decisions in plan selection: whether a
Expand Down Expand Up @@ -67,6 +67,20 @@ pub trait AccuracyModel {
fn may_accept_regardless_of_guarantee(&self) -> bool {
true
}

/// Whether a guarantee stated in `metric` can meet `target` with some
/// error bound and failure probability, however small: Pass 1 prunes a
/// summary with which no reading query's output can. The default is
/// [`Self::satisfies`]'s built-in rule: an exact target accepts only an
/// exact output, and a target that sets a metric only a bound in that
/// metric (or an L2-scaled bound for an L1-scaled target). A model that
/// converts between metrics overrides it.
fn metric_can_meet(&self, metric: AccuracyMetric, target: &AccuracyTarget) -> bool {
!target.is_exact()
&& target
.accuracy_metric
.is_none_or(|wanted| crate::metric_meets(metric, wanted))
}
}

/// The planner's built-in analytical rules as an [`AccuracyModel`]: the
Expand Down
4 changes: 3 additions & 1 deletion crates/devtools/src/bin/stage_pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,9 @@
// accuracy guarantee, and candidates are compared on cost alone;
// executability checks still apply. `--enforce-accuracy-guarantees` clears
// it, and Stage 3 rejects an output whose guarantee does not meet its
// query's target, as `plan_installable` does with the default models. The document's top-level `accuracy_guarantees` is
// query's target, as `plan_installable` does with the default models; Pass 1
// then also prunes each summary with which no query reading it can meet its
// target (`NoErrorBoundForQueryTarget`). The document's top-level `accuracy_guarantees` is
// `"ignored"` or `"enforced"`.
// `--sample-interval-ms` is the sample interval declared for every metric the
// PromQL queries read (default 15000). `--series-count <n>`,
Expand Down
25 changes: 17 additions & 8 deletions crates/devtools/tests/stage_pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1268,10 +1268,11 @@ fn plan_selection_settings_flag_loads_the_settings_from_json() {
}

/// stage_pipeline ignores accuracy guarantees by default: every sketch
/// candidate of an average of p99s, which the built-in model rejects as
/// having no guarantee under `--enforce-accuracy-guarantees`, is valid with
/// no flag, accepted regardless of its guarantee, and records why the
/// guarantee check would have rejected it. The document says which mode it
/// candidate of an average of p99s, which has no guarantee and which Pass 1
/// prunes under `--enforce-accuracy-guarantees`
/// (`NoErrorBoundForQueryTarget`), is valid with no flag, accepted
/// regardless of its guarantee, and records why the guarantee check would
/// have rejected it. The document says which mode it
/// planned under; there is no `--ignore-accuracy-guarantees`.
#[test]
fn accuracy_guarantees_are_ignored_by_default_and_enforced_on_request() {
Expand All @@ -1290,14 +1291,22 @@ fn accuracy_guarantees_are_ignored_by_default_and_enforced_on_request() {
enforcing.push("--enforce-accuracy-guarantees");
let enforced = document_of(&enforcing);
assert_eq!(enforced["accuracy_guarantees"], "enforced");
let unguaranteed: Vec<String> = enforced["reference_pricing"]["rejected"]
let no_bound: Vec<&str> = enforced["stage1_logical_asap_optimization"]["pruned_candidates"]
.as_array()
.unwrap()
.iter()
.filter(|r| r["cause"] == "no_accuracy_guarantee")
.map(|r| r["id"].as_str().unwrap().to_string())
.filter(|pruned| pruned["reason"] == "NoErrorBoundForQueryTarget")
.map(|pruned| pruned["candidate"].as_str().unwrap())
.collect();
assert!(!unguaranteed.is_empty(), "{enforced}");
assert!(!no_bound.is_empty(), "{enforced}");
assert!(
!enforced["reference_pricing"]["rejected"]
.as_array()
.unwrap()
.iter()
.any(|r| r["cause"] == "no_accuracy_guarantee"),
"{enforced}"
);
let document = document_of(&args);
assert_eq!(document["accuracy_guarantees"], "ignored");
let accepted = &document["reference_pricing"]["accepted"];
Expand Down
Loading
Loading