Skip to content

Fix mtrl eval hyperparameter names - #6595

Open
aviruthen wants to merge 3 commits into
aws:masterfrom
aviruthen:fix-mtrl-eval-hyperparameter-names
Open

aviruthen wants to merge 3 commits into
aws:masterfrom
aviruthen:fix-mtrl-eval-hyperparameter-names

Conversation

@aviruthen

Copy link
Copy Markdown
Collaborator

Issue

MultiTurnRLEvaluator silently ignored most user-set evaluation hyperparameters.
For example, sampling_max_tokens could not be raised above the 4096 default.

Root cause

_build_job_config_doc copied six hard-coded names into
EvaluationConfig.HyperParameters: eval_group_size, sampling_temperature,
top_p, max_tokens, pass_k_values and success_threshold. The hub
MTRLEvaluation recipes use temperature, sampling_top_p, sampling_max_tokens
and rollout_* instead. hparams.get() returned None for each mismatched name,
and the value was dropped without a warning. I confirmed the names for all four
MTRL models in SageMakerPublicHub: gemma-4-31b-it, qwen3-6-27b, nova-lite-v2 and
gpt-oss-20b.

Fix

This follows MultiTurnRLTrainer._build_training_config:

  • Hub defaults are recorded when the hyperparameter spec loads.
  • Every value the user changes is sent under the recipe's own name. This covers
    evaluator.hyperparameters.<name>, recipe and overrides.
  • List values such as pass_k_values are forwarded.
  • A failed hub lookup now logs a WARNING. Before, it logged at INFO level.
  • Six unused template-context keys that carried the old names are removed.
  • Docstrings now list the recipe's parameter names.

⚠️ Behavior change

Before this change, eval_group_size, pass_k_values and success_threshold were
always sent, even when they were at their defaults. Now HyperParameters contains
only values that differ from the recipe defaults, and the block is omitted when
nothing changed. The service applies the recipe defaults in that case, so the
values used are the same, but the generated pipeline JSON is smaller. If the
service's defaults ever differ from the hub spec's defaults, values the user didn't
set will follow the service. This matches how MultiTurnRLTrainer already behaves.

Testing

  • 7 new unit tests in test_mtrl_evaluator.py, using a mocked copy of the hub
    spec. 5 of them fail on the old code.
  • tests/unit/train/evaluate/ and test_trainer_recipe_integration.py:
    639 passed, 7 skipped.
  • Checked locally against the live hub spec for gpt-oss-20b:
    sampling_max_tokens=8192, rollout_timeout and pass_k_values appear in the
    JobConfigDocument, and 16384 is rejected because it exceeds the max of 8192.
  • black and flake8 report no problems.

MultiTurnRLEvaluator copied only six hard-coded hyperparameter names
(sampling_temperature, top_p, max_tokens, ...) into the JobConfigDocument.
The hub MTRLEvaluation recipes use different names (temperature,
sampling_top_p, sampling_max_tokens, rollout_*), so user-set values were
silently dropped and jobs always ran with service defaults (e.g. a 4096
output-token cap).

Follow MultiTurnRLTrainer: snapshot the hub defaults when the spec loads
and send every value the user changed, keyed by the recipe's own names.
Values from recipe/overrides are honored, list values such as
pass_k_values are forwarded, and a failed hub lookup now logs a warning.

Behavior change: values left at their recipe defaults are no longer sent;
the service applies the recipe defaults instead.
---
X-AI-Prompt: Fix MultiTurnRLEvaluator so hyperparameters such as the output token cap are forwarded under the hub recipe's names, following how MultiTurnRLTrainer sends user-changed values.
X-AI-Tool: Kiro CLI
Shorten docstrings and inline comments added in the previous commit,
and drop the hard-coded parameter list from the hyperparameters
docstring since the names come from the hub recipe.
---
X-AI-Prompt: Reduce the over-explanatory comments added in the MTRL evaluator hyperparameter fix and keep only the useful ones.
X-AI-Tool: Kiro CLI
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 19:41 — with GitHub Actions Active
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 19:41 — with GitHub Actions Active
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 19:41 — with GitHub Actions Active

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR fixes MultiTurnRLEvaluator silently dropping user-set evaluation hyperparameters by keying them under the recipe's own parameter names instead of a hard-coded list, sending only values that differ from the hub defaults, and upgrading the hub-fetch-failure log to WARNING. The approach mirrors MultiTurnRLTrainer and is backed by new unit tests. The core logic is sound; one real gap remains around list-valued hyperparameters supplied via recipe/overrides.

Reviewed commit f17827d. Automated review; verify before acting.

Returns:
Dict[str, str]: Changed values, keyed by recipe parameter name.
"""
final = self.hyperparameters.to_dict()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Medium: List-valued hyperparameters (e.g. pass_k_values) set via recipe or overrides are silently dropped. _get_effective_hyperparameters() uses flatten_resolved_recipe, which explicitly skips list values (elif not isinstance(v, list) in recipe_resolver.flatten_resolved_recipe), so the loop here never sees them. The only list values that survive are those captured from self.hyperparameters.to_dict(), i.e. those set via direct attribute assignment. The PR body states "This covers evaluator.hyperparameters.<name>, recipe and overrides" and "List values such as pass_k_values are forwarded", but the recipe/overrides path for list values does not work. The new test_list_values_are_forwarded only exercises direct assignment and test_overrides_dict_is_forwarded only uses scalars, so this gap is untested. Consider merging list values from the resolved recipe before dropping them, or documenting the limitation.

…overrides

flatten_resolved_recipe skips list values, so pass_k_values set through
recipe or overrides was dropped. Take list values from the resolved
recipe's inference section as well.
---
X-AI-Prompt: Address the review comment that list-valued hyperparameters like pass_k_values set via recipe or overrides are silently dropped, without changing which keys are forwarded.
X-AI-Tool: Kiro CLI
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 22:48 — with GitHub Actions Active
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 22:48 — with GitHub Actions Active
@aviruthen
aviruthen deployed to manual-approval November 19, 2026 22:49 — with GitHub Actions Active

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI code review

This PR fixes MultiTurnRLEvaluator silently dropping user-set evaluation hyperparameters. The old _build_job_config_doc copied six hard-coded names (eval_group_size, sampling_temperature, top_p, max_tokens, pass_k_values, success_threshold) that don't match the hub MTRLEvaluation recipe names (temperature, sampling_top_p, sampling_max_tokens, rollout_*), so hparams.get() returned None and the values were discarded. The new _get_user_changed_hyperparameters captures hub defaults and forwards only user-changed values under the recipe's own names, closely mirroring MultiTurnRLTrainer._build_training_config/_apply_recipe_to_hyperparameters. The change is consistent with the established trainer pattern, well covered by 7 new unit tests, and I found no blocking issues.

Reviewed commit 69093b0. Automated review; verify before acting.

)
for k, v in values.items():
if v is not None:
final[k] = v if isinstance(v, str) else str(v)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Low: In _get_user_changed_hyperparameters, recipe/override-sourced values are stringified (final[k] = str(v)) before the final comparison v != defaults.get(k), but defaults holds native-typed values from to_dict() (e.g. int 4096). So a value supplied via recipe/overrides that is equal to the recipe default (e.g. sampling_max_tokens: 4096) still compares unequal ("4096" != 4096) and gets sent in HyperParameters, whereas the same value set via direct attribute assignment is correctly suppressed. The effect is harmless since the service applies the same default, and MultiTurnRLTrainer has the identical behavior, so this is only a minor inconsistency. The new test test_value_equal_to_default_is_not_sent only exercises the direct-assignment path and wouldn't catch this; consider a test for the recipe/overrides-equals-default case if you want the suppression to be exact.

This branch was successfully deployed

1 active deployment
manual-approval — 69093b02 Deployed Nov 19, 2026 by aviruthen via wait-for-approval #906
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants