Repository navigation
feat(cost): tune budgets, Config recording and log retention for cost control - #159
Merged
Merged
Conversation
Terraform Plan🚧 Changes detected — Plan: 15 to import, 2 to add, 17 to change, 2 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan imports 15 existing CloudWatch log groups, creates 2 resources, updates 17 (budget limit/notifications, Config recorder scope, log retention), and destroys a billing alarm and its SNS topic, with no database or security-group changes.
|
Terraform Plan🚧 Changes detected — Plan: 15 to import, 2 to add, 17 to change, 2 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan imports 15 existing CloudWatch log groups and sets 365-day retention on them, creates 2 resources, updates 17 (budget limit/notifications, Config recorder frequency, log retention), and destroys the billing alarm and its SNS topic in us-east-1; no database, security group, or IAM changes, and nothing is internet-exposed.
|
… control Budgets, Cost Anomaly Detection, Config recording ENI/VPC/Subnet daily (not excluded, so Security Hub EC2.180 and Config.1 keep working), and bounded retention for every platform Lambda log group (existing groups adopted with import blocks). Rebased onto #163 (which already expires Config history after two years, so this PR's duplicate lifecycle rule is dropped) and #158 (adds a log group for the new javabin-auth-claims-external trigger). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Alexanderamiri
force-pushed
the
feat/cost-guardrails
branch
from
October 26, 2026 14:11
bb2a925 to
edbf02f
Compare
Terraform Plan🚧 Changes detected — Plan: 15 to import, 2 to add, 18 to change, 2 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan imports 15 existing CloudWatch log groups and makes 18 in-place updates, creates 2 new resources, and destroys the billing alarm and its SNS topic; nothing here opens security groups or changes IAM, but the billing alarm removal and budget changes warrant a look before apply.
|
Alexanderamiri
added a commit
that referenced
this pull request
Oct 26, 2026
## Why `testteam-platform-test-app` has been crash-looping for weeks with desired=1 and running=0, because of a `CannotPullContainerError` (its ECR repo is empty). That is about 80 failed task starts a day, and nobody noticed. Each attempt creates and deletes a Fargate ENI, which AWS Config records. This was the main driver of about $29/mo in Config charges found in the 2026-10 cost audit. PR #159 reduces the Config side. This PR makes the crash loop itself visible. ## What - `monitoring/main.tf`: new EventBridge rule `javabin-ecs-task-start-impaired`. It matches `source=aws.ecs`, `detail-type=ECS Service Action` and `eventName=SERVICE_TASK_START_IMPAIRED`, for the `javabin-platform` cluster only (legacy CDK clusters are excluded). The target is the `javabin-security` SNS topic, whose policy already allows `events.amazonaws.com`. slack-alert sends non-finding events from that topic to #javabin-infra-alerts. I chose that channel over the cost channel because this is an operational problem. - `slack_alert/handler.py`: - New `format_ecs_service_impaired` formatter, routed before the generic EventBridge branch. Without it, the event would fall through to the IAM-change formatter. - The message shows the service, the cluster and a link to the service's events page in the ECS console. - Dedup uses the existing persistent `javabin-alert-dedup` table, keyed per service ARN per UTC day. ECS keeps emitting the event while the service is impaired. - New test `terraform/lambda-src/tests/test_slack_alert_ecs.py` checks the formatting, once-per-day dedup per service, and that other ECS service actions are ignored. - Docs: added a line to the CLAUDE.md alert routing section and to `docs/lambda-functions.md`. ## Expected plan 2 to add: `aws_cloudwatch_event_rule.ecs_task_start_impaired` and `aws_cloudwatch_event_target.ecs_task_start_impaired_sns`. 1 in-place update: the slack-alert Lambda code. No IAM changes, because slack-alert already has GetItem/PutItem on the dedup table. ## How verified - `python3 -m unittest discover -s terraform/lambda-src/tests` (67 OK, 3 new) and `-s scripts/tests` pass. - `terraform fmt -recursive -check`, then `terraform init -backend=false` and `terraform validate` pass. ## Follow-ups - Once this is applied, the first alert should be for `testteam-platform-test-app`. Scale it to 0 or push its image. - It is independent of #159. Both touch `monitoring/main.tf`, but in different blocks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Alexanderamiri
added a commit
that referenced
this pull request
Oct 26, 2026
app.yaml calls/provides declare private app-to-app edges, optionally with OAuth scopes. Every app gets its own task security group and a Cloud Map name in the new javabin.internal namespace; app-peers-sync opens ingress only where both sides agree and manages the M2M resource servers and clients. The developer boundary protects the shared namespace and the private zone's records (6,067 of 6,144 bytes). Rebased onto routing.login (#158), RDS/DNS adoption (#155), Lambda log retention (#159, adds the app-peers-sync log group) and the app-auth-sync restart rules (#167). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A cost audit (2026-10, live Cost Explorer/CloudWatch/Config data) found that the cost guardrails we already run have gaps:
javabin-billing-alarm(us-east-1) does nothing. It has beenINSUFFICIENT_DATAsince 2026-03-26 because the AWS/Billing EstimatedCharges metric is not enabled for the account. Its SNS topic has 0 subscribers, and its $200 threshold is below monthly spend anyway.testteam-platform-test-appcrash-looping (~80 starts/day), so fixing that service is the main saving; this PR only trims what remains./aws/lambda/javabin-*log groups and auth-claims never expire. Lambda auto-created them, and Terraform doesn't manage them.What
cost-analytics:javabin-alerts, which the slack-alert Lambda posts to #javabin-cost-alerts.aws_cloudwatch_metric_alarm.billing,aws_sns_topic.billing_alarmand thebilling_alarm_threshold_usdvariable. The us-east-1 provider alias stays because the CUR export uses it.monitoring:AWS::EC2::NetworkInterface,AWS::EC2::VPCandAWS::EC2::Subnetuse a DAILY override. Security groups stay CONTINUOUS. ENIs are deliberately not excluded: Security Hub EC2.180 (ENI source/destination check, enabled) evaluatesAWS::EC2::NetworkInterface, and Config.1 fails when an enabled control's resource type is not recorded. Trade-off: EC2.180 evaluates daily instead of on every change, and an ENI created and deleted within one day may never be recorded. Expected saving ~$15–20/mo while the crash loop lasts, a few dollars a month after it is fixed.javabin-config-<acct>bucket now expires objects after 730 days. The bucket is not versioned.lambdas/log-groups.tfandidentity/log-groups.tf:aws_cloudwatch_log_groupwith 365-day retention for every platform Lambda.imports.tf:for_eachimport blocks (TF 1.7) adopt the 14 existing groups plus auth-claims. budget-enforcer has never logged, so its group is created fresh.scripts/tests/test_lambda_log_groups.py: scans everyterraform/platform/**/*.tfand fails if a${var.project}-*Lambda has no log group (sharedlocal.platform_lambda_namesor an explicit/aws/lambda/...group in its module).docs/cost-management.mdcovering alerts, tags, CUR, built-in controls and why Config records ENIs daily, and the post-apply checks.docs/platform-modules.mdanddocs/lambda-functions.md.Not changed, on purpose:
Expected plan
module.cost_analytics.aws_budgets_budget.account: updated in place.module.cost_analytics.aws_cloudwatch_metric_alarm.billingandaws_sns_topic.billing_alarm: destroyed. Both are inert, as explained above.module.monitoring.aws_config_configuration_recorder.main: updated in place.module.monitoring.aws_s3_bucket_lifecycle_configuration.config_logs: created.How verified
terraform fmt -recursive -check, thenterraform init -backend=falseandterraform validateinterraform/platform. All pass.python3 -m unittest discover -s scripts/tests(46 OK) and-s terraform/lambda-src/tests(OK). Negative check: renaming the auth-claims log group makes the guard test fail.git merge-treeagainstfeat/ecs-impaired-alert(feat(monitoring): alert on ECS services that cannot start tasks #160): no conflict.AWS::EC2::NetworkInterface.javabin-config-*bucket is unversioned with no lifecycle, the recorder isALL_SUPPORTED_RESOURCE_TYPES+ CONTINUOUS, and the list of existing/aws/lambda/javabin-*log groups matches the import list.Follow-ups
AWS/Config ConfigurationItemsRecordedper ResourceType. If SecurityGroup still dominates, add it to the DAILY override.AWS::EC2::NetworkInterface.testteam-platform-test-appto 0, or re-push its image. It is the root cause of the Config churn.account_budget_usdas the legacy cleanup lands. The cleanup is in the cost audit report: idle EB environments, Default-ALB, Coolify, NAT.🤖 Generated with Claude Code