Repository navigation
FIX: Export converted data type and use displayed type in HTML report - #3798
Closed
Sandiyo Christan (sandiyochristan) wants to merge 1 commit into
Conversation
JSON conversation export previously omitted piece.converted_value_data_type, recording only the original data type under the generic 'data_type' field. Consequently, the HTML scenario result renderer displayed the converted value while testing against original data_type, causing converted media (e.g. text-to-image) to render as plain text without [image_path] placeholders, or converted text (e.g. image-to-text) to be mislabeled as media. This commit: - Adds explicit 'original_value_data_type' and 'converted_value_data_type' to the serialized piece dict in JsonConversationPrinter while preserving 'data_type' for backward compatibility. - Updates the HTML scenario report template to select the data type corresponding to the displayed representation (converted_value_data_type if converted_value is present, falling back to original_value_data_type or data_type). - Adds unit tests in test_json.py and test_html.py covering text-to-image and image-to-text conversions through both builders. Closes microsoft#3742
Author
|
@microsoft-github-policy-service agree |
Contributor
|
Thanks for the contribution. Shivam Sharma (@ShivamSharma43) opened #3782 first for #3742. I compared the current heads, and both export the converted data type and render the displayed representation using that type. I did not find separate behavior here that needs to be carried over. #3782 also includes real-builder and backward-compatibility coverage and is already coordinating its overlapping HTML changes with #3771. I am closing this later duplicate so we can review one implementation. Thank you for understanding. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3742
Description
JSON conversation export previously omitted
piece.converted_value_data_type, serializing only the original data type under the generic"data_type"key. WhenHtmlScenarioReportPrinterrendered the conversation transcript, it evaluatedp.data_typewhile renderingp.converted_value or p.original_value.This caused a mismatch for converted representations:
original_value_data_type="text"andconverted_value_data_type="image_path") were rendered as plain text without the expected[image_path]placeholder.original_value_data_type="image_path"andconverted_value_data_type="text") were rendered with an incorrect[image_path]prefix.Changes
pyrit/output/conversation/json.py):"original_value_data_type"and"converted_value_data_type"to the piece dictionary in_build_piece."data_type"(piece.original_value_data_type) for backward compatibility with existing consumers.pyrit/output/scenario_result/html.py):display_typebased on the displayed representation (p.converted_value_data_typeifp.converted_valueis present, falling back top.original_value_data_typeorp.data_type).test_build_async_includes_distinct_converted_data_typeintests/unit/output/conversation/test_json.py.test_html_report_renders_text_to_image_converted_media_with_labelandtest_html_report_renders_image_to_text_converted_piece_as_plain_textintests/unit/output/scenario_result/test_html.py.Tests and Documentation
uv run pytest tests/unit/output/conversation/test_json.py tests/unit/output/scenario_result/test_html.py tests/unit/output/scenario_result/test_json.py # 38 passeduv run pytest tests/unit/output/ # 394 passed in 3.97suv run ruff check pyrit/output/ tests/unit/output/ uv run ruff format --check pyrit/output/ tests/unit/output/ # All checks passed!