Skip to content

FIX: Export converted data type and use displayed type in HTML report - #3798

Closed
Sandiyo Christan (sandiyochristan) wants to merge 1 commit into
microsoft:mainfrom
sandiyochristan:fix/json-conversation-export-converted-data-type
Closed

Sandiyo Christan (sandiyochristan) wants to merge 1 commit into
microsoft:mainfrom
sandiyochristan:fix/json-conversation-export-converted-data-type

Conversation

@sandiyochristan

Copy link
Copy Markdown

Closes #3742

Description

JSON conversation export previously omitted piece.converted_value_data_type, serializing only the original data type under the generic "data_type" key. When HtmlScenarioReportPrinter rendered the conversation transcript, it evaluated p.data_type while rendering p.converted_value or p.original_value.

This caused a mismatch for converted representations:

  • Converted media pieces (such as text-to-image conversions where original_value_data_type="text" and converted_value_data_type="image_path") were rendered as plain text without the expected [image_path] placeholder.
  • Converted text pieces (such as image-to-text OCR where original_value_data_type="image_path" and converted_value_data_type="text") were rendered with an incorrect [image_path] prefix.

Changes

  1. JSON Conversation Printer (pyrit/output/conversation/json.py):
    • Added explicit "original_value_data_type" and "converted_value_data_type" to the piece dictionary in _build_piece.
    • Retained "data_type" (piece.original_value_data_type) for backward compatibility with existing consumers.
  2. HTML Scenario Report (pyrit/output/scenario_result/html.py):
    • Updated the template to determine display_type based on the displayed representation (p.converted_value_data_type if p.converted_value is present, falling back to p.original_value_data_type or p.data_type).
  3. Unit Tests:
    • Added test_build_async_includes_distinct_converted_data_type in tests/unit/output/conversation/test_json.py.
    • Added test_html_report_renders_text_to_image_converted_media_with_label and test_html_report_renders_image_to_text_converted_piece_as_plain_text in tests/unit/output/scenario_result/test_html.py.

Tests and Documentation

  • Run unit tests:
    uv run pytest tests/unit/output/conversation/test_json.py tests/unit/output/scenario_result/test_html.py tests/unit/output/scenario_result/test_json.py
    # 38 passed
  • Run full output suite:
    uv run pytest tests/unit/output/
    # 394 passed in 3.97s
  • Formatting & linting:
    uv run ruff check pyrit/output/ tests/unit/output/
    uv run ruff format --check pyrit/output/ tests/unit/output/
    # All checks passed!

JSON conversation export previously omitted piece.converted_value_data_type,
recording only the original data type under the generic 'data_type' field.
Consequently, the HTML scenario result renderer displayed the converted value
while testing against original data_type, causing converted media (e.g. text-to-image)
to render as plain text without [image_path] placeholders, or converted text
(e.g. image-to-text) to be mislabeled as media.

This commit:
- Adds explicit 'original_value_data_type' and 'converted_value_data_type' to
  the serialized piece dict in JsonConversationPrinter while preserving 'data_type'
  for backward compatibility.
- Updates the HTML scenario report template to select the data type corresponding
  to the displayed representation (converted_value_data_type if converted_value
  is present, falling back to original_value_data_type or data_type).
- Adds unit tests in test_json.py and test_html.py covering text-to-image and
  image-to-text conversions through both builders.

Closes microsoft#3742
@sandiyochristan

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@romanlutz

Copy link
Copy Markdown
Contributor

Thanks for the contribution. Shivam Sharma (@ShivamSharma43) opened #3782 first for #3742. I compared the current heads, and both export the converted data type and render the displayed representation using that type. I did not find separate behavior here that needs to be carried over. #3782 also includes real-builder and backward-compatibility coverage and is already coordinating its overlapping HTML changes with #3771.

I am closing this later duplicate so we can review one implementation. Thank you for understanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON conversation export omits the converted data type, making converted media appear as text in HTML

2 participants