[davidberenstein1957]

Description

CodeCarbon can now send a small, anonymous report about the environment it runs in, so we can see which platforms and versions people use. It is a reduced first version of #1106. Tier 2 (extensive) and endpoint hardening move to #1429.

What users see

• Telemetry is on by default at the minimal level. Until someone picks a level, every run prints a short notice to stderr explaining what is sent and how to turn it off. The notice stops once a level is chosen.
• Interactive codecarbon config and codecarbon monitor ask which level to use.
• To opt out: codecarbon telemetry set disabled (applies in every directory), CODECARBON_TELEMETRY_LEVEL=disabled, or telemetry_level= on the tracker.
• Offline mode (OfflineEmissionsTracker, codecarbon monitor --offline) never sends anything.

What is sent

• Environment and hardware only: OS, country and region, CPU/GPU/RAM, Python and CodeCarbon versions. No coordinates, no host names, paths or hashes, and the timestamp is rounded to the hour.
• If geolocation fails, no location is sent rather than the Canada fallback. The cuDNN version is only read when the program already imported torch.
• Every field is listed in docs/how-to/telemetry.md. Rows contain no IP address and are deleted automatically after 3 years.

When

• At most once per process, only for runs of at least 1 s. The send happens on a background thread with a time budget and never slows down or breaks tracking.

Settings

• Level: tracker argument, then CODECARBON_TELEMETRY_LEVEL, then telemetry_level in .codecarbon.config, then the default. An old extensive value falls back to minimal; an empty or unknown value (off, false, 0) means disabled.
• Telemetry has its own URL setting (telemetry_api_url) and does not follow the dashboard's api_endpoint.

Server

• POST /telemetry needs no token. It rejects unknown fields, caps strings at 256 characters, and returns 429 above 60 requests a minute per IP.
• New alembic revision 20260927_add_telemetry. It creates the table, or, where the table from feat: first version of telemetry #1171 already exists, drops the removed columns, keeps existing rows and converts timestamp to a time-zone-aware column. It also adds an index on timestamp and a trigger that deletes rows older than 3 years on insert, so retention needs no separate job.

Related Issue

Part of #1106. Follow-up: #1429.

Motivation and Context

The project has no view of how CodeCarbon is used. This sends the smallest useful set of environment fields, with a clear notice, a one-line opt-out and a documented field list. A token shipped inside an open-source package would be public and would also allow writes to /runs and /emissions, so the endpoint takes no token and relies on validation and rate limiting.

How Has This Been Tested?

• uv run task test-package and uv run task test-api-unit: all passed.
• Coverage of the changed lines is close to 100% for codecarbon/core/telemetry/, the CLI, the router and the migration.
• scripts/e2e_telemetry.sh against a real Postgres and a local API: two processes produce exactly two rows with only the documented columns.
• The migration was run on a real Postgres, both empty and seeded with master's telemetry table and a row: upgrade keeps the row, downgrade restores the old columns, upgrade again works.
• Retention on a real Postgres: a row dated 4 years ago is deleted on the next insert, recent rows stay; downgrade removes the trigger, function and index. carbonserver/tests/database/test_telemetry_retention_db.py repeats this when DATABASE_URL points at a Postgres.

Screenshots (if appropriate):

N/A

Before deploying

1. Deploy order: the server (with the migration) must go out before a package release that sends telemetry; older servers answer 404, which the client ignores. @inimaz, could you take care of the server deploy?
2. Proxy addresses (accepted for now): uvicorn runs with --forwarded-allow-ips=*, so the per-IP limit can be bypassed with a forged X-Forwarded-For, and the limit is per process. Tracked in Telemetry follow-up: tier 2, endpoint hardening, retention #1429.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

Please refer to docs/how-to/ai-policy.md for detailed guidelines on how to disclose AI usage in your PR. Accurately completing this section is mandatory.

• 🟥 AI-vibecoded: You cannot explain the logic. Car analogy : the car drive by itself, you are outside it and just tell it where to go.
• 🟠 AI-generated: Car analogy : the car drive by itself, you are inside and give instructions.
• ⭐ AI-assisted. Car analogy : you drive the car, AI help you find your way.
• ♻️ No AI used. Car analogy : you drive the car.

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the docs/how-to/contributing.md document.
• I have added tests to cover my changes.
• All new and existing tests passed.

Commits since the original PR

• 442e781 feat(carbonserver): delete telemetry rows older than 3 years with a database trigger
• c42ea37 docs(telemetry): keep telemetry rows for at most 3 years
• 9005900 fix(telemetry): show the notice once, and skip the level prompt with --offline
• 4401307 feat(telemetry): show the notice every run until a level is chosen
• f0b6b4d Merge remote-tracking branch 'origin/feat/add-telemetry' into pr-1200-merge
• fe01571 fix(migration): make telemetry.timestamp timezone-aware
• 15bf9de test(config): drop leftover CODECARBON_TELEMETRY_PROJECT_TOKEN references
• 3d94a71 fix(telemetry): stop reusing dashboard api_endpoint for telemetry URL
• 5934207 fix(telemetry): treat empty telemetry_level as invalid, not unset
• bc9c808 fix(telemetry): never send telemetry from OfflineEmissionsTracker
• 1ad9810 docs(telemetry): fix precedence order (env overrides config file)
• 54b328d fix(telemetry): read cudnn version from sys.modules, never import torch
• 2d388f8 fix(telemetry): print the one-time notice to stderr so it isn't hidden
• 1db720e fix(telemetry): write opt-out level to the global config by default
• 992992d fix(scripts): run e2e Postgres on a configurable host port
• 2025c20 fix(telemetry): do not report the Canada geolocation fallback as a location
• 35e6d09 fix(scripts): hard-fail e2e_telemetry.sh on Postgres or API startup problems
• a9d9c13 fix(telemetry): make the API rate limiter thread-safe
• d9b0627 fix(telemetry): truncate the payload timestamp to the hour
• f7acc2d fix(telemetry): make send_at_stop resilient to thread start failures
• 9423cb7 fix(telemetry): fail closed to disabled on an unparseable telemetry_level
• f608203 fix(telemetry): migrate pre-existing telemetry table instead of assuming it's absent
• faeb497 test(telemetry): add end-to-end script checking rows in Postgres
• dd772f5 test(telemetry): cover every changed telemetry line
• c2e01c5 feat(telemetry): send once per process and rate limit the endpoint
• e5a4f83 feat(carbonserver): add alembic migration for the telemetry table
• 50d377d refactor(telemetry): keep only populated schema fields and cap strings
• 79ba393 refactor(telemetry): drop extensive tier, API token and coordinates
• 11ec802 Merge remote-tracking branch 'origin/master' into pr-1200-merge

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.08%. Comparing base (e159a68) to head (81d0a9a).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1200      +/-   ##
==========================================
+ Coverage   93.66%   94.08%   +0.42%     
==========================================
  Files          50       55       +5     
  Lines        5591     5853     +262     
==========================================
+ Hits         5237     5507     +270     
+ Misses        354      346       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[davidberenstein1957]

@inimaz, before merging, we should still add the hardcoded experiment and project for the codecarbon api.

[inimaz]

Thanks for this PR @davidberenstein1957 . I have left some comments

[inimaz]

Maybe this is not needed. get_hierarchical_config handles it?

[davidberenstein1957]

Kept on purpose: set has to pick one file to write to, while get_hierarchical_config only merges what it reads.

[benoit-cty]

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: 🔧 Request changes

Maintainer direction on the telemetry policy (@benoit-cty): telemetry should be opt-out by default, and the user should be asked the question explicitly. A suggested implementation:

• On first interactive use (e.g. codecarbon config / codecarbon monitor in a TTY), ask once which telemetry level the user wants. Explain exactly what each level sends, and store the answer in the config (telemetry_level).
• In non-interactive contexts (library use, CI, SLURM, no TTY), never block on a prompt. Apply the default, and show a clear notice once per machine (remember that it was shown) describing what is sent and how to opt out (codecarbon telemetry set disabled / CODECARBON_TELEMETRY_LEVEL).
• docs/how-to/telemetry.md should list every field sent, the retention period, and how to opt out.

Must fix:

1. The default experiment_id changes for every existing user (emissions_tracker.py:609).
   • The default is swapped from 5b0fa12a-… to DEFAULT_TELEMETRY_EXPERIMENT_ID (d2d69403-…).
   • That value goes into EmissionsData.experiment_id (L1073), so into every CSV row. It is also used for save_to_api runs where the user set no experiment, so those users' emissions would be posted into the telemetry experiment.
   • Please revert this line.
2. stop() blocks about 3 s, contradicting "never block".
   • dispatcher.py:57 calls build_payload() synchronously, which calls collect.py:87 get_env_cloud_details(). That is uncached and probes 3 cloud metadata endpoints with a 1 s timeout each, plus NVML and find_spec lookups.
   • Your own tests/test_telemetry.py::test_stop_does_not_block_on_hanging_endpoint fails locally (3.01 s against 0.5 s). It passes in CI only because GitHub runners are Azure VMs, where the metadata endpoint answers.
   • Fix: build the payload inside the background thread, and/or reuse the cloud and hardware info the tracker has already detected.
3. The warning is misleading and shown on every run.
   • send_at_stop returns early without a telemetry_api_key (dispatcher.py:51), and there's no built-in key, so nothing is sent by default.
   • Yet every user without telemetry_level gets a WARNING saying "Minimal telemetry sends on each stop" (dispatcher.py:20-45), and codecarbon telemetry status prints the same.
   • The message must match what is actually sent, and should be shown once per machine, as part of the prompt/notice flow above.
4. Authentication of /telemetry. Shipping a hardcoded project/experiment write token inside the public package (as planned in your comment) makes the new token check (routers/telemetry.py:36-43) ineffective. Anyone can extract it. Consider a dedicated ingestion endpoint that needs no secret, with strict schema validation and rate limiting, rather than a shared write token that looks like a secret. The same applies to extensive creating runs in a shared experiment through ApiClient.
5. Unknown coordinates are sent as 0,0 (collect.py:244-247, _round_coordinate_or_zero).
   • This sends 0.0, 0.0 when the location is unknown, which pollutes the data. Send null instead, reusing _round_or_none as @inimaz suggested.
   • Also, conf.get("longitude", emissions.longitude) returns None, and so 0.0, when the key exists with a None value.
6. Most telemetry is lost. It's sent on a daemon thread (dispatcher.py:61), which is killed at interpreter exit. That covers the common cases: a script ending, @track_emissions, or the CLI exiting right after stop(). Consider an atexit join with a short cap (≤1 s). In extensive mode, ApiClient also logs at ERROR/INFO from that background thread on offline machines. Lower those logs to DEBUG for telemetry.

Housekeeping:
7. Merge conflicts in codecarbon/cli/main.py and codecarbon/core/config.py. The config.py part has to be redone on top of #1389's get_hierarchical_config refactor (read_config_file, source logging), not just rebased.
8. The description is stale. It mentions send_telemetry and "once-per-session deduplication", but the code sends on every stop().

Nits:

• _set_from_conf(level, "telemetry_level") puts telemetry_level into _conf, so it ends up in logged and exported config.
• The Telemetry import inside __init__ is unnecessary.
• _integration() sniffs sys.argv, which is fragile.
• About 60 schema fields are duplicated on the client side. The drift test helps, but consider generating one from the other.

[davidberenstein1957]

Made the changes in cd8ccf6: opt-out with a one-time prompt/notice, stop() non-blocking, experiment_id reverted, null coords, atexit join, docs list every field. Merged master, redid config.py on top of get_hierarchical_config, refreshed the description.
Not done: /telemetry auth (a shipped token vs an open rate-limited endpoint is your call), the retention period for the docs, and the nits on argv sniffing and client/server schema generation.

[davidberenstein1957]

@benoit-cty @inimaz I'd like to move ahead with a simpler version of this PR and get it merged, then build on it. What changes:

• Only the minimal tier and disabled remain. extensive is gone for now, and an existing telemetry_level = extensive config falls back to minimal. Any other value we don't recognise (off, false, 0) means disabled.
• No API token, project or experiment. /telemetry is an open endpoint with request validation (unknown fields rejected, strings capped at 256 characters) and a per-IP rate limit (429). A token shipped in the package would be public anyway, and it would also grant write access to /runs and /emissions for that experiment.
• No latitude or longitude. Country and region stay. The timestamp is rounded to the hour.
• Sent at most once per process, not at every stop(). Runs shorter than 1 s are skipped. Sending never blocks or breaks the tracker.
• Opt-out by default with the one-time notice, as you asked. The docs list every field sent.
• An alembic migration for the telemetry table. It also handles databases where the table already exists from feat: first version of telemetry #1171: it drops the removed columns and keeps the rows.

Three things I need from you before this can be deployed:

1. Production table. The table from feat: first version of telemetry #1171 probably exists in prod. Are there rows worth keeping? The migration keeps them either way.
2. Proxy addresses. uvicorn runs with --forwarded-allow-ips=*, so the per-IP limit can be bypassed by setting X-Forwarded-For. Can we restrict it to the load balancer's addresses, or rate-limit at the proxy? Is there more than one API instance?
3. Retention. How long should telemetry rows be kept? The docs say "TBD" until we pick a number.

Tier 2, the token question, retention and the endpoint hardening move to #1429.

[benoit-cty]

Thanks @davidberenstein1957 !

Good idea to start small and iterate.

• Production table ? If it's for telemetry, then it's just your test, isn't it ?
• Proxy addresses ? @inimaz do you understand the question ?
• Retention : I think GDPR said 3 years, after that we need to anonymize and maybe aggregate ?

I did some fixes, with Opus 5.5.

Review fixes: telemetry collection & CLI

Four fixes on the package side of the telemetry feature. The server side (migration, rate limiter, schema, repository) needed no change.

1. Don't import torch to read the cuDNN version

_cudnn_version() imported torch whenever it was installed, even if the user's process never used it. That import can take several seconds and load CUDA/cuDNN libraries. The 2s telemetry budget didn't cover it, because the budget is only checked before the POST. At process exit, the background thread could be killed mid-import.

Fix: read sys.modules.get("torch"), so we only report cuDNN when the user's code already imported torch. _package_installed() is no longer used and was removed.

2. Keep cloud location when IP geolocation falls back to Canada

When both geo APIs fail, the tracker assumes Canada, and telemetry dropped country_name, country_iso_code and region. On cloud runs, those values come from the cloud-region mapping, not from IP geolocation, so correct data was thrown away (e.g. an AWS eu-west-3 run lost "France").

Fix: only drop them when on_cloud != "Y".

3. codecarbon telemetry config picker

• Ctrl-C at the file picker made questionary return None, then Path(None) raised a TypeError. It now exits cleanly with typer.Exit(0), like the level picker.
• The duplicate check compared a Path to a list of str, so it never matched. Run from $HOME, the same file was listed twice. It now compares strings.

4. Detect editable installs

importlib.metadata.Distribution has no editable attribute, so editable installs were reported as uv/pip.

Fix: read direct_url.json (PEP 610) and check dir_info.editable.

Tests

• test_default_geo_fallback_keeps_cloud_location
• test_cudnn_version_does_not_import_torch; test_cudnn_version now relies on sys.modules
• test_codecarbon_install_method now builds fake direct_url.json metadata
• test_pick_config_path_lists_same_file_once, test_pick_config_path_interactive_cancel_exits

All new tests fail against the previous code, except test_cudnn_version_does_not_import_torch, which can only catch the bug where torch is installed. Telemetry suites: 78 passed.

[davidberenstein1957]

@inimaz could you deploy the server side of this PR to api.codecarbon.io before the next package release? It needs the alembic migration 20260927_add_telemetry, which creates the telemetry table, or updates the one #1171 already created and keeps its rows. The migration also adds a trigger that deletes rows older than 3 years, so no separate cleanup job is needed. Until the server is deployed, clients get a 404 on /telemetry, which they ignore, so there is no user impact, only lost data.

Decisions made: telemetry stays on by default at minimal, rows are deleted automatically after 3 years, and the per-IP rate limit limitation is accepted for now (#1429).

[benoit-cty]

Rebased onto master (squashed fd7a63b+cd8ccf6 and the two master merges, tree unchanged; conflicts in emissions_tracker.py, test_monitor.py, testutils.py kept both sides). Alembic chain OK: master added no migrations, 20260927_add_telemetry still revises 20251119_add_utilization.

• inimaz thread (telemetry_cli.py): resolve_config_path's local/global lookup could never run since set writes the global file; removed it and get_config_file_settings, fixed stale --config help (00e375e)
• Retention: the purge keyed on a client timestamp, so a future-dated row would never be deleted; the server now caps it at now (UTC) and rounds it to the hour (cb1f970)
• Test that docs/how-to/telemetry.md lists exactly the schema fields (b15867a); drop duplicate monitor test (5feb40f)
• Docs: say telemetry is on by default; OfflineEmissionsTracker never sends; correct --config default (245532c); black (81d0a9a)
• Coverage: 397/397 changed lines in codecarbon/; project 93.6%. test-api-unit, migration tests and docs build pass. Pre-existing local failures: RAPL permission tests (root) and flaky test_logging_output, both also on master.
• Open for maintainers: privacy_policy_en.md doesn't mention SDK telemetry or the 3-year retention; the purge only runs when a row is inserted; proxy-header/per-process rate limit (Telemetry follow-up: tier 2, endpoint hardening, retention #1429); server deploy before release.

---

Generated by Claude Code