Add composite MySQL safety checks and stable recovery

[pradeepkintali]

Summary

• Compose MySQL checks with opt-in RequiredClusters dependencies.
• Evaluate ShadowRequiredClusters without changing the source cluster's admission decision, and report would-reject observations.
• Fail closed when configured safety probes have no eligible hosts or required runtime metrics are unavailable.
• Support separate recovery thresholds and continuous healthy recovery periods, including after startup and leadership changes.
• Include the blocking metric name in composite responses and export recovery-latch metrics.
• Add leader-only, read-only /explain/mysql/<cluster> diagnostics for loaded thresholds, recovery settings, exclusions, and cached required/shadow metric state.
• Support canonical Vitess PRIMARY discovery while accepting MASTER as a deprecated input alias.
• Validate dependency references, cycles, and tablet types when loading configuration.

Configuration and compatibility

RequiredClusters, ShadowRequiredClusters, FailOnNoHosts, and
RecoveryDurationMillis are opt-in. Existing clusters retain legacy behavior
when these settings are absent.

Shadow observations do not simulate application-specific throttling policies.
Background observations represent aggregation cycles, not client requests or
independent database samples. Missing observations are not evidence of health.

Diagnostics do not run admission checks or SQL, advance recovery, or record
synthetic check/shadow observations. Missing measurements are null; credentials,
queries, connection details, and raw probe errors are not exported. A successful
diagnostic request is not permission to write.

Deployment and rollback

Deploy a compatible binary before configuring the new fields. Validate probe
discovery, permissions, sampling cost, and recovery behavior before enabling
required dependencies. Review thresholds and client retry/wait budgets for each
workload, and enable enforcement incrementally.

To roll back dependency enforcement, remove the relevant RequiredClusters
entry while preserving existing replica-lag checks. Keep it in
ShadowRequiredClusters if continued observation is desired. Configuration
deployment and startup recovery still apply; this is not an instantaneous
runtime toggle.

See doc/mysql.md for configuration examples, behavior changes, and rollout
warnings, and doc/http.md for the diagnostic endpoint contract.

Validation

• Race-enabled tests: go test -mod=vendor -race ./cmd/... ./pkg/...
• Static analysis: go vet -mod=vendor ./cmd/... ./pkg/...
• Binary build: go build -mod=vendor -o /tmp/freno ./cmd/freno
• Diagnostic coverage includes dependency modes, recovery, unavailable/non-finite values, redaction, read-only behavior, and HTTP error handling.
• Go formatting and git diff --check.

Scope

These are cooperative admission checks, not universal database load shedding.
Deployments remain responsible for metric selection, thresholds, discovery,
monitoring, and safe client retry/defer behavior.

[Copilot]

Copilot review overview

🟡 Changes recommended

Recovery semantics and direct-memcache behavior can bypass or inconsistently apply the new safety gates.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Removing IsValidReplica breaks source compatibility · New
• Direct client path bypasses required cluster checks · New
• Recovery state is not reset after leadership regain · New
• Recovery ignores the effective read threshold · New

What changed in this PR

Adds composite MySQL safety checks, recovery hysteresis, and configurable Vitess tablet discovery.

Changes:

• Composes replica checks with required primary and ProxySQL metrics.
• Adds fail-closed behavior, recovery thresholds, cooling periods, and metrics.
• Supports validated REPLICA, MASTER, and PRIMARY Vitess tablet configuration.

File	Description
pkg/​base/​throttle_metric.go	Adds recovery-state metric errors.
pkg/​config/​config_test.go	Tests new configuration validation.
pkg/​config/​mysql_config.go	Adds composite and recovery settings.
pkg/​config/​vitess_config.go	Validates tablet types.
pkg/​throttle/​check.go	Evaluates required clusters.
pkg/​throttle/​check_result.go	Reports the blocking metric.
pkg/​throttle/​throttler.go	Implements recovery latching.
pkg/​throttle/​throttler_test.go	Tests composition and recovery.
pkg/​vitess/​api_client.go	Filters configured tablet types.
pkg/​vitess/​api_client_test.go	Tests primary discovery.
doc/​mysql.md	Documents configuration and rollout scope.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[andyedison]

LGTM 🚀