Skip to content

Fix logs --since and /proxy/logs?since= to filter by date and time - #1996

Merged
waldekmastykarz merged 9 commits into
dotnet:mainfrom
waldekmastykarz:symmetrical-tribble
Oct 26, 2026
Merged

waldekmastykarz merged 9 commits into
dotnet:mainfrom
waldekmastykarz:symmetrical-tribble

Conversation

@waldekmastykarz

@waldekmastykarz waldekmastykarz commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1961

Problem

devproxy logs --since and GET /proxy/logs?since= assumed log lines start with [hh:mm:ss.fff] and combined that with DateTime.Today. Detached log lines don't have that prefix (e.g. info …, req ╭ GET …). Any line that didn't match was included, so since filtered nothing. For example, --since 2030-01-01T00:00:00 returned the whole log. Even when it did match, the date was ignored.

Fix

New LogSinceFilter (shared by the CLI command and the API controller) filters full timestamp instants:

  • Before the first timestamp in legacy logs, lines use the selected process's actual start time, which remains stable across hot reloads.
  • Request blocks (req ╭ … ╰, including the preceding blank line) use the full ISO timestamp from their time │ line.
  • Detached regular text messages include full UTC timestamps at every surviving log level, including Warning/Error-only logging. Buffered regular messages retain their emission timestamps.
  • Legacy debug/trace [HH:mm:ss.fffffff] prefixes (and [HH:mm:ss.fff]) are reconstructed against the previous timestamp, accounting for midnight, small out-of-order writes, and the repeated daylight-saving hour.
  • JSON log lines retain their timestamp instants.
  • Other lines use the most recent preceding timestamp.
  • Detached logs always retain timestamp metadata, even with showTimestamps: false. Interactive output continues to honor that setting and keeps its existing formatting.
  • Shared CLI/API cutoff parsing and comparisons use DateTimeOffset, preserving explicit offsets and correct ordering across both occurrences of the repeated daylight-saving hour.

CLI --follow and API SSE streaming share an incremental reader that filters by event time, retains timestamp reconstruction state, and buffers partial lines and incomplete request blocks across reads. Requests that start before the cutoff remain excluded even if they finish afterward.

Validation

  • All 491 solution tests pass, including regression coverage for process start after hot reload, buffered requests across reads, partial lines, JSON/text/request timestamps, midnight rollover, daylight-saving rollback and instant ordering, elapsed-time cutoff parsing, Warning-level timestamp metadata, and unchanged interactive output.
  • All 38 log-filtering, CLI log, and logging-configuration tests also pass with TZ=America/New_York.
  • Solution build succeeds without warnings or errors after integrating upstream main 94a9579 (Count intercepted requests for mock nth #1987).
  • Detached end-to-end checks with showTimestamps: false confirm CLI/API snapshot and follow filtering at Information and Warning levels. Newly emitted Error records remain visible after a post-start cutoff at Warning level; older records and future-cutoff output are excluded.

…loses dotnet#1961

Detached log lines don't start with a [hh:mm:ss.fff] timestamp, so the
previous filter matched no lines, included all of them and ignored the
date. Reconstruct each line's full date and time from the instance start
time, request blocks' 'time' lines, debug/trace time-of-day prefixes (with
midnight rollover) and JSON 'timestamp' properties, and filter on that.
The log line format is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3b2ff54c-c6ab-4c98-a477-c522a6c02ac2
Copilot AI balanced review requested due to automatic review settings October 21, 2026 14:04
@waldekmastykarz
waldekmastykarz requested a review from a team as a code owner October 21, 2026 14:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Filtering remains incorrect after hot reload, with timestamps disabled, and for buffered requests in follow mode.

3 open issues
What changed in this PR

Addresses #1961 by sharing full-date log filtering between the CLI and management API without changing the log format.

Changes:

  • Reconstructs timestamps from request blocks, debug/trace prefixes, and JSON entries.
  • Integrates shared filtering into both log readers.
  • Adds tests for multi-day filtering, midnight rollover, and timestamp fallback.
File Description
DevProxy/​Logging/​LogSinceFilter.cs Implements shared timestamp reconstruction and filtering.
DevProxy/​Commands/​LogsCommand.cs Integrates filtering and changes follow-mode handling.
DevProxy/​ApiControllers/​ProxyController.cs Uses shared filtering with process start time.
DevProxy.Tests/​LogSinceFilterTests.cs Covers timestamp formats and date boundaries.

Review effort: Balanced

Comment thread DevProxy/Commands/LogsCommand.cs Outdated
Comment thread DevProxy/Logging/LogSinceFilter.cs
Comment thread DevProxy/Commands/LogsCommand.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Daylight-saving clock rollback can incorrectly date logs one day ahead and include them beyond the requested cutoff.

4 open issues

Review effort: Balanced

Comment thread DevProxy/Logging/LogSinceFilter.cs Outdated
Preserve detached timestamps, use process start across hot reloads, share event-time filtering for CLI/API follow mode, and handle DST rollback. Add regression coverage and document detached timestamp retention.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate upstream main c9a4a0b while preserving the reviewed log filtering fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Repeated-hour comparisons and missing timestamps in warning/error-only text logs can still produce incorrect filtering results.

2 open issues
4 resolved since last review

Review effort: Balanced

Comment thread DevProxy/Extensions/ILoggingBuilderExtensions.cs
Comment thread DevProxy/Logging/LogSinceFilter.cs
waldekmastykarz and others added 2 commits October 26, 2026 11:48
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Include full UTC timestamps on detached regular records at every log level, preserve DateTimeOffset instants through snapshot/follow filtering, and share offset-aware CLI/API cutoff parsing. Add repeated-hour and Warning-level regressions while retaining interactive formatting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Integrate upstream main b9865a6; complete solution build and all 494 tests pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Blank-label request terminators are not recognized, delaying filtered streaming output and misdating subsequent snapshot messages.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread DevProxy/Logging/LogSinceFilter.cs Outdated
waldekmastykarz and others added 2 commits October 26, 2026 12:05
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Accept Normal request entries with an empty label before the closing corner. Add real-formatter regressions for snapshot timestamps and immediate plain/ANSI follow completion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues remain, and the supplied regression coverage addresses the timestamp and streaming failures reported previously.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Integrate c8363ef including command exit codes and inactivity shutdown. Full solution build and all 526 tests pass; 41 log tests also pass in America/New_York.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Stateful streaming and timezone-sensitive legacy reconstruction warrant final human validation across supported platforms.

No open issues 🎉

Review effort: Balanced

@waldekmastykarz

Copy link
Copy Markdown
Collaborator Author

The latest summary reports no open issues and does not identify a failing scenario. The stateful paths are covered by regressions for incomplete request blocks across reads, partial lines/CRLF, event-time exclusion of delayed requests, midnight rollover, blank-label/ANSI terminators, and JSON/text/request timestamps. Repeated-hour tests exercise both UTC-offset orderings and New York, Berlin, and Lord Howe rollback rules. On exact head f67be85, all 526 solution tests pass on macOS; all 41 log-related tests pass with TZ=America/New_York, UTC, Europe/Berlin, and Australia/Lord_Howe. Earlier detached macOS runtime checks also exercised CLI/API snapshots, CLI follow, and API SSE at Information and Warning levels with showTimestamps:false.

Platform limits: the green GitHub Actions build runs on ubuntu-latest and does not execute tests; I have not executed Windows tests or Linux runtime tests. Therefore I am not claiming all-platform validation. With no concrete defect or missing scenario specified, I do not see a justified implementation change in the generic recommendation. Please distinguish any specific remaining correctness issue from a discretionary additional-platform sign-off. Requesting one further pass with this evidence; no code changes or repeated review requests solely to obtain a different verdict.

@waldekmastykarz
waldekmastykarz requested a balanced review from Copilot October 26, 2026 12:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The shared implementation and targeted regression coverage address the reported filtering failures with no unresolved blocking issues identified.

No open issues 🎉

Review effort: Balanced

@waldekmastykarz
waldekmastykarz merged commit 25fcff3 into dotnet:main Oct 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

logs --since and /proxy/logs?since= compare only the time of day

2 participants