Repository navigation
feat: dbt_cost_estimate tool for BigQuery dry-run scan estimates #1732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b7cfc04
1eb722e
d5e7bc8
d191355
21d0494
d90e334
33e2655
9f19a75
1c23d35
4fa4703
9f4ae46
99e953c
6c2ab6f
d8c3469
f2f4546
dace1c2
2d3c5fb
18a3fc4
6024ac9
f5159c5
f57de3d
8c8265d
6514705
d7938df
aaff414
3a98dff
34fa3dd
29ee2cd
ac43c53
67f4dae
5c61e7b
460cdc9
3bad7cc
23dac88
10a9f1f
b48df04
29b8f05
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,17 +1,156 @@ | ||
| import z from "zod" | ||
| import { Tool } from "../../tool/tool" | ||
| import { dbtAdapterFor } from "./dbt-adapter-access" | ||
|
|
||
| /** | ||
| * Every dbt integration but the Python bridge writes the SQL into a `{{ validate_sql('...') }}` Jinja | ||
| * argument, so a quote or backslash in it must be escaped or any filter on a string breaks the dry run. | ||
| * The bridge, which a config without an integration also means, binds the SQL as a parameter and takes it as is. | ||
| */ | ||
| export function forDryRun(sql: string, integration: unknown): string { | ||
| return integration == null || integration === "core" ? sql : sql.replace(/\\/g, "\\\\").replace(/'/g, "\\'") | ||
| } | ||
|
|
||
| /** The units dbt-bigquery's `format_bytes` writes, smallest first; each is 1024 times the one before. */ | ||
| const UNITS = ["Bytes", "KiB", "MiB", "GiB", "TiB", "PiB"] as const | ||
|
|
||
| /** | ||
| * The byte count in a dry run's `bytes_processed`, which dbt-bigquery formats as text ("4.2 TiB", | ||
| * "512.0 Bytes"), and the most it can be. The text carries one decimal, so a figure past a byte is | ||
| * rounded: the true count is within half a tenth of its unit. Undefined for anything else, so a changed | ||
| * format is reported, never read as a count. | ||
| */ | ||
| /** Whether the SQL is a script (more than one statement, or EXECUTE IMMEDIATE): a dry run only sees its first statements. */ | ||
|
hkaltai marked this conversation as resolved.
Resolved
hkaltai marked this conversation as resolved.
Resolved
|
||
| export function isScript(sql: string): boolean { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: |
||
| // One pass over comments and quoted text together: a quote inside a comment, or a comment marker inside a quote, is not either. | ||
| const bare = sql | ||
| .replace(/--[^\n]*|#[^\n]*|\/\*[\s\S]*?\*\/|'(?:[^'\\]|\\.)*'|"(?:[^"\\]|\\.)*"|`[^`]*`/g, " ") | ||
|
hkaltai marked this conversation as resolved.
Resolved
|
||
| .replace(/;\s*$/, "") | ||
| return bare.includes(";") || /\bexecute\s+immediate\b/i.test(bare) | ||
| } | ||
|
|
||
| export function parseBytesProcessed(display: unknown): { bytes: number; maxBytes: number } | undefined { | ||
| if (typeof display === "number") { | ||
| if (!Number.isFinite(display) || display < 0) return undefined | ||
| const bytes = Math.round(display) | ||
| return { bytes, maxBytes: bytes } | ||
| } | ||
| if (typeof display !== "string") return undefined | ||
| const match = /^\s*(\d+(?:\.\d+)?)\s*(Bytes|KiB|MiB|GiB|TiB|PiB)\s*$/i.exec(display) | ||
| if (!match) return undefined | ||
| const unit = UNITS.findIndex((u) => u.toLowerCase() === match[2].toLowerCase()) | ||
| const scale = 1024 ** unit | ||
| const bytes = Math.round(Number(match[1]) * scale) | ||
| if (unit === 0) return { bytes, maxBytes: bytes } | ||
| // Exact in integers: the shown figure is rounded to its last digit, so the true count is at most half | ||
| // of that digit above it. A float product is off by whole bytes past a few TiB. | ||
| const [whole, fraction = ""] = match[1].split(".") | ||
| const digits = BigInt(whole + fraction) | ||
| let upper = ((digits * 2n + 1n) * 1024n ** BigInt(unit) + 2n * 10n ** BigInt(fraction.length) - 1n) / (2n * 10n ** BigInt(fraction.length)) | ||
| // Python divides in floats, so a count within a float's last place of a tie is shown rounded down. | ||
| upper += upper >> 50n | ||
| let maxBytes = Number(upper) | ||
| if (BigInt(Math.trunc(maxBytes)) < upper) maxBytes *= 1 + Number.EPSILON | ||
| return { bytes, maxBytes: Math.max(bytes, maxBytes) } | ||
| } | ||
|
|
||
| /** Starts `start` and waits for it, but gives up as soon as the call is cancelled; a cancelled call starts nothing. */ | ||
| export function untilAbort<T>(start: () => Promise<T>, signal: AbortSignal): Promise<T> { | ||
| if (signal.aborted) return Promise.reject(signal.reason) | ||
| return new Promise<T>((resolve, reject) => { | ||
| const stop = () => reject(signal.reason) | ||
| signal.addEventListener("abort", stop, { once: true }) | ||
| Promise.resolve() | ||
| .then(() => (signal.aborted ? Promise.reject(signal.reason) : start())) | ||
| .then(resolve, reject) | ||
| .finally(() => signal.removeEventListener("abort", stop)) | ||
| }) | ||
| } | ||
|
|
||
| function failure(title: string, message: string, extra: Record<string, unknown> = {}) { | ||
| return { title: `dbt_cost_estimate: ${title}`, metadata: { success: false, error: message, ...extra }, output: message } | ||
| } | ||
|
|
||
| export const DbtCostEstimateTool = Tool.define("dbt_cost_estimate", { | ||
| description: "Estimate the bytes a dbt model or a SQL query would scan, from a warehouse dry run (BigQuery).", | ||
| description: | ||
| "Estimate the bytes a dbt model or a SQL query would scan, from a warehouse dry run. Only BigQuery has a dry run: any other warehouse answers `supported: false` and nothing is compiled. Starts the dbt adapter (which runs `dbt deps`), so it needs write permission (`--allow-write` on `tool run` or the MCP server). The dry run never executes the final query, but compiling either input can render Jinja, and macros such as `run_query` can run real queries that are billed. Pass `model` (a model name, compiled as dbt would) or `sql` (may use Jinja).", | ||
| parameters: z.object({ | ||
| model: z.string().optional().describe("Model name to compile and estimate"), | ||
| sql: z.string().optional().describe("SQL (may use Jinja) to compile and estimate"), | ||
| }), | ||
| async execute() { | ||
| async execute(args, ctx) { | ||
| if (!args.model && !args.sql) return failure("nothing to estimate", "Pass a model name or SQL.") | ||
|
hkaltai marked this conversation as resolved.
Resolved
|
||
| if (args.model && args.sql) return failure("ambiguous input", "Pass a model name or SQL, not both.") | ||
| const access = await dbtAdapterFor(ctx) | ||
| if (!access.adapter) return failure("not available", `No estimate: ${access.reason}.`, { refusal: access.refusal }) | ||
| const adapter = access.adapter | ||
| const type: string | undefined = adapter.getAdapterType?.() | ||
| // Every dbt-integration refuses the dry run off BigQuery, so nothing is compiled there. | ||
| if (type !== "bigquery") { | ||
| const adapterName = type ?? "unknown" | ||
| return { | ||
| title: "dbt_cost_estimate: no estimate", | ||
| metadata: { success: true, supported: false, adapter: adapterName }, | ||
| output: `No cost estimate: dry runs are only available on BigQuery (this project uses ${adapterName}).`, | ||
| } | ||
| } | ||
| const signal = ctx.abort | ||
| let compiled: unknown | ||
| try { | ||
| // A model compiles by name, so its node context (`this`, config) is kept; SQL compiles inline. | ||
| compiled = await untilAbort( | ||
| () => (args.model ? adapter.unsafeCompileNode(args.model) : adapter.unsafeCompileQuery(args.sql!, undefined)), | ||
| signal, | ||
| ) | ||
| } catch (e) { | ||
| if (signal.aborted) throw e | ||
| return failure("compile failed", e instanceof Error ? e.message : String(e), { adapter: type }) | ||
| } | ||
| if (typeof compiled !== "string" || !compiled.trim()) { | ||
| return failure("compile failed", "The query compiled to nothing.", { adapter: type }) | ||
| } | ||
| let display: unknown | ||
| try { | ||
| const integration = await untilAbort(async () => { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: this reads the dbt config again instead of taking the integration from the adapter
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changed here. Taking the integration from the adapter would mean exposing the config from |
||
| const { dbtAdapterInternals } = await import("../native/connections/register") | ||
| const read = dbtAdapterInternals.readConfig ?? (await import("../../../../dbt-tools/src/config")).read | ||
| return ((await read()) as { dbtIntegration?: unknown } | null)?.dbtIntegration | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: this reads the dbt config again instead of the config the running adapter was created from
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changed here. Taking the integration from the adapter would mean exposing the config from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Prompt for AI agents
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changed here, for the same reason as the thread above: it needs the foundation's adapter access to carry the integration, and the case needs |
||
| }, signal) | ||
| const result = await untilAbort(() => adapter.validateSQLDryRun(forDryRun(compiled as string, integration)), signal) | ||
| if (result == null) { | ||
| return failure("dry run failed", "The dry run returned no estimate: dbt-core before 1.6 can't dry run.", { adapter: type }) | ||
| } | ||
| display = (result as { bytes_processed?: unknown }).bytes_processed | ||
| } catch (e) { | ||
| if (signal.aborted) throw e | ||
| return failure("dry run failed", e instanceof Error ? e.message : String(e), { adapter: type }) | ||
| } | ||
| const parsed = parseBytesProcessed(display) | ||
| if (!parsed) { | ||
| return failure("unreadable estimate", `The dry run returned an amount this tool can't read: ${JSON.stringify(display)}.`, { | ||
| adapter: type, | ||
| }) | ||
| } | ||
| const name = args.model | ||
| const shown = typeof display === "number" ? `${parsed.bytes} Bytes` : String(display) | ||
| // A dry run can report 0 for a query that still reads data (BigQuery federated sources), and it skips | ||
| // what a script runs after its first statement or inside EXECUTE IMMEDIATE: those figures are lower | ||
| // bounds and carry no upper bound. | ||
| const zero = parsed.bytes === 0 | ||
| const lower = zero || isScript(compiled as string) | ||
| return { | ||
| title: "dbt_cost_estimate: not implemented", | ||
| metadata: { success: false, error: "not implemented" }, | ||
| output: "dbt_cost_estimate is not implemented in this build of altimate-code yet.", | ||
| title: `dbt_cost_estimate: ${shown}`, | ||
| metadata: { | ||
| success: true, | ||
| supported: true, | ||
| adapter: type, | ||
| ...(lower ? { bytes: parsed.bytes, lowerBound: true } : parsed), | ||
| display: shown, | ||
| ...(name && { model: name }), | ||
| }, | ||
| output: `${name ? `${name} ` : "The query "}would process ${shown} ${ | ||
| lower ? "(a dry run can undercount a script or a zero estimate, so this is a lower bound, not a cap)." : `(at most ${parsed.maxBytes} bytes).` | ||
| }`, | ||
| } | ||
| }, | ||
| }) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WARNING: the omitted-integration default assumed here is not the one the adapter applies
integration == nullis treated as "the Python bridge" here, but the adapter never treats a missing integration as the bridge:packages/dbt-tools/src/adapter.ts:47resolvescfg.dbtIntegration ?? DEFAULT_CONFIGURATION_VALUES.dbtIntegration, and this repo's own CLI default is"corecommand"(packages/dbt-tools/src/config.ts:105,packages/dbt-tools/src/commands/init.ts:19) - a value this function escapes. Nothing here tiesforDryRun's fallback to the library constant the adapter uses, so adbt.jsonthat omitsdbtIntegrationgets no escaping while the adapter runs whatever the library default is; if that default iscorecommand(as the two repo defaults suggest), every compiled query with a string literal breaks the Jinja argument. Resolve the default from the same source the adapter uses (or ask the adapter for its integration) instead of assuming the bridge.dbt-cost-estimate.test.ts:291pins this assumption, not the library's default.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not changed: the premise does not hold. The adapter reads
cfg.dbtIntegration ?? DEFAULT_CONFIGURATION_VALUES.dbtIntegration, and the library's default (@altimateai/dbt-integration0.3.17) is"core", the Python bridge.corecommandis only whatinitwrites into a new file. So a config naming no integration is the bridge, which is what d7938df pins and its test shows.