Skip to content

[Feature] The frame is posed for its predicted presentation time, and the game receives it - #1415

Merged
untoldengine merged 5 commits into
untoldengine:developfrom
miolabs:feature/presentation_time_frame
Nov 18, 2026
Merged

untoldengine merged 5 commits into
untoldengine:developfrom
miolabs:feature/presentation_time_frame

Conversation

@miogds

@miogds miogds commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The engine side of the frame-time work the viewer bench uses (untoldengine/UntoldArcade#42): that pull request does not build against develop without it (cannot find type 'FrameTime'), and builds with this branch.

What was wrong

On the headset the head pose is predicted for the instant the frame will be shown, but every object was posed for "now at update time": the XR loop ran the game update before its wait for the compositor's optimal input time, on a wall-clock delta between two updates, and queried the device anchor at the drawable's presentation time afterwards. The gap between the two instants is the wait plus the encode, and it varies with the previous frame's CPU work, so a moving object stepped unevenly from frame to frame while the head did not: judder at full frame rate, which the viewer's dartboard tour showed and RealityKit did not. A skipped refresh then doubled the step, and motion stepped by the time between updates drifted off the display's timeline after it.

What a frame does now

  • The XR loop converts the compositor's predicted presentation time (predictTiming().presentationTime, through the compositor clock's epoch, which is the boot like CACurrentMediaTime()) and hands it to updateXR(presentationTime:). The game update stays before the wait, where its cost sits inside the wait anyway; the engine's delta (timeSinceLastUpdate) is the time between two presentations, not between two updates, so every delta-driven path (skeletal clips, the camera path, scripts, the physics accumulator, extensions) moves in step with the display without changing.
  • FrameTime carries the presentation time, the delta, the period, the refreshes skipped and the frame index; FrameClock makes it, learning the period and counting the refreshes a frame skipped. The game receives it through setupCallbacks(frameUpdate:handleInput:), renderer.frameTime or the currentFrameTime global.
  • The MTKView and AR loops pose the frame one of the view's periods ahead of the draw, or for a display link's target set in predictedPresentationTime, and name the view's period to the clock (updateXR(presentationTime:nominalPeriod:) for the AR draw), so the API is the same on every platform. The benefit is not: see the limitation below.
  • TransformTrack: keys with easing sampled at an absolute time, so a frame the display skipped lands where the timeline says and the frames after it step by one period again; applyTransformTrack poses an entity from it.
  • The engine stats report the presentation delta and the skipped refreshes; the XR loop counts the frames whose drawable's presentation time differs from the prediction (presentationTimeRevisions) and prints once how far the compositor clock's epoch is from the media time.
  • docs/Architecture/xrRenderingSystem.md and docs/Tutorials/StarterDemo.md say it.

On top of today's develop

Two feature commits written on #1359's branch in October, cherry-picked onto develop. They meet two later changes in calculateDeltaTime and the XR loop, and both are kept: the first frame after a GPU pause still has no delta (#1391's resume count resets the frame clock), and every frame the loop takes is still ended. The simulation stepping (#1393) takes the presentation delta like every other delta-driven path.

Checklist

  • Relevant tests have been added or updated
  • Existing functionality and API compatibility have been considered
  • Relevant builds and tests pass locally
  • Recoverable failures are handled appropriately (no silent failures, no unnecessary crashes)
  • Significant architectural changes and trade-offs are explained below
  • Documentation has been updated where necessary
  • Performance-sensitive changes have been measured (profiling/benchmarks), if applicable

Verified

  • Twenty new tests pass: the frame clock (a steady cadence, a skipped refresh counted once and the next frame back to one period, jitter that is not a skip, a stall, a nominal period, time going backwards), the track, and, with a real renderer, that a pose the game samples in frameUpdate is the pose for the frame's presentation time rather than the wall clock, that the update event and the extensions receive the same time, and that a two-period gap reports one skipped frame.
  • The core suite (UntoldEngineTests): 1766 tests, one failure that is this checkout's folder name, not the change (ExternalRenderExtensionPackageTests builds a fixture that depends on a package named UntoldEngine, and the worktree is UntoldEngine-frametime).
  • make strict-concurrency-check passes. SwiftFormat 0.62.1 reports nothing on the touched files beyond wrapIfStatementBodies in lines this does not touch; CI's 0.60.1 was not run locally.
  • UntoldEngineAR builds for iOS (generic/platform=iOS).
  • The viewer of UntoldViewer: a comparison bench for the engine against RealityKit on Vision Pro UntoldArcade#42 at its head, with its package pointed at this branch, builds for visionOS (generic/platform=visionOS), which compiles UntoldEngineXR for the device.
  • On the headset: not this branch. The same two commits, on Headset rendering: foveation, temporal anti-aliasing, a forward path, and RealityKit's look in a mixed space #1359's branch, with the viewer's dartboard tour at 90 Hz on 2026-10-15: the compositor epoch differs from the media time by 0.000 ms, no presentation revisions, no skipped refreshes, and the small per-frame steps are gone. They have been in the fork's develop since.

Architectural notes / trade-offs

  • Nothing existing breaks. gameUpdate(deltaTime:) keeps its parameter, which is now the presentation delta. UpdateEvent, EngineExtensionUpdateContext, RenderGraphBuildContext and RenderPassContext gain a frameTime with a default, so every caller compiles.
  • The update stays before the wait. Running it after the wait would give game logic input about 6 ms fresher, at the cost of a third more CPU work after the optimal input time on every frame, with the update's variance landing on the rendering deadline. Before the wait, head and objects share whatever error the prediction has, which is what removes the judder.
  • Only the headset gets the exact time. An MTKView predicts no presentation time, so on macOS, iOS and in the AR renderer the frame's time is an estimate one period ahead of the draw and the delta is still the wall-clock spacing of the draws, jitter included, as before this change. A host that drives the view from its own display link gets the exact time by setting predictedPresentationTime to the link's target before each draw. A clock left to learn its period keeps the shortest spacing it has seen, which is right for the compositor's times and wrong for such estimates (one short gap between two draws would make every later frame report a skipped refresh), so those loops name the view's period; FrameClockTests.testAShortGapBecomesTheLearnedPeriod pins both sides. The docs say it (StarterDemo.md, the comparison table in xrRenderingSystem.md).
  • What it does not fix. A hop remained on the tour's fast lateral sweep with every counter at zero: the GPU finishing after the compositor's rendering deadline. That is the frame's GPU cost, a separate matter from when the frame is posed.

Javier Segura added 4 commits November 13, 2026 20:07
… the game receives it

On the headset the head pose is predicted for the instant the frame will be
shown, but every object was posed for "now at update time", a varying lag
behind: a moving object stepped unevenly from frame to frame and read as
judder at full frame rate. The XR loop now converts the compositor's
predicted presentation time (through the compositor clock's epoch, the boot,
like CACurrentMediaTime) and hands it to updateXR(presentationTime:); the
engine's delta is the time between two presentations, not between two
updates.

FrameTime (presentation time, delta, period, skipped refreshes, frame index)
and FrameClock, which learns the period and counts the refreshes a frame
skipped, make the frame's time. It reaches the game through
setupCallbacks(frameUpdate:handleInput:), renderer.frameTime and the
currentFrameTime global; gameUpdate(deltaTime:) keeps its parameter.
UpdateEvent, EngineExtensionUpdateContext, RenderGraphBuildContext and
RenderPassContext gain a frameTime field with a default. The MTKView and AR
loops pose the frame one of the view's periods ahead of the draw, or for the
host's display-link target set in predictedPresentationTime. The engine stats
report the presentation delta and the skipped refreshes; the XR loop counts
the frames whose drawable's presentation time differs from the prediction.
Transform keys with easing, sampled at an absolute time: a frame's pose is a
function of the frame's presentation time alone, so a frame the display
skipped lands where the timeline says and the frames after it step by one
period again. applyTransformTrack poses an entity from a track for a frame.

The tests show a repeated frame does not double-step an absolute-time
animation while delta-stepped motion drifts, and, with a real renderer, that
a pose the game samples in frameUpdate matches the frame's presentation time
rather than the wall clock, that the update event and the extensions receive
the same frame time, and that a two-period gap reports one skipped frame.
@miogds
miogds requested a review from untoldengine as a code owner November 13, 2026 19:13

@untoldengine untoldengine left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this.

One thing worth a quick discussion before merging: updateXR(presentationTime:) doesn't have a way to pass a nominalPeriod, so UntoldEngineAR.swift ends up hand-computing CACurrentMediaTime() + period and handing that to updateXR without ever telling FrameClock the known period. The clock then has to learn the period from deltas between those self-computed values, which in practice collapses back to measuring wall-clock draw-to-draw spacing — basically what the old code did. So it looks like the AR path doesn't quite get the same "immune to CPU/encode jitter" benefit that the XR path does, since only XR hands in a real compositor-predicted time rather than now + period.

Totally possible this is intentional (the inline comment hints at it), in which case it'd be great to call it out explicitly in the PR description/docs so it's a known, documented limitation rather than something a future reader has to rediscover. If it's not intentional, exposing nominalPeriod on updateXR (or a lead:-based variant) would let AR reuse the engine's own fallback logic instead of duplicating it, and give FrameClock the real period instead of learning a noisy one.

Not a blocker either way — just flagging since it's the one spot where the fix's reach seemed narrower than the rest of the PR.

…lay link target, through updateXR(nominalPeriod:)
@miogds

miogds commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

You're right, and it was half intentional. Fixed in 1f86c8ad.

The intentional half: an MTKView predicts no presentation time, so on macOS, iOS and AR the frame's time is an estimate one period ahead of the draw, and the delta is still the wall-clock spacing of the draws. Only the headset gets the jitter fix. The description and the docs now say so (StarterDemo.md, the table in xrRenderingSystem.md). A host that drives the view from its own display link can set predictedPresentationTime to the link's target and get the exact time there too.

The other half was an oversight. updateXR now takes nominalPeriod:, and the AR draw passes the view's period and the display-link target instead of computing now + period, so it reuses the engine's fallback like the Mac loop. It mattered more than noise: the clock keeps the shortest spacing it has seen, so one short gap between two draws would have made every later frame report a skipped refresh. FrameClockTests.testAShortGapBecomesTheLearnedPeriod pins that.

The first build failure was GaussianPagingTest.testWaitingReadsHoldNoThreadBeyondTheConcurrentReads, the timing flake; I re-ran it.

@untoldengine
untoldengine merged commit 31d3686 into untoldengine:develop Nov 18, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants