Repository navigation
[Feature] The frame is posed for its predicted presentation time, and the game receives it - #1415
Conversation
… the game receives it On the headset the head pose is predicted for the instant the frame will be shown, but every object was posed for "now at update time", a varying lag behind: a moving object stepped unevenly from frame to frame and read as judder at full frame rate. The XR loop now converts the compositor's predicted presentation time (through the compositor clock's epoch, the boot, like CACurrentMediaTime) and hands it to updateXR(presentationTime:); the engine's delta is the time between two presentations, not between two updates. FrameTime (presentation time, delta, period, skipped refreshes, frame index) and FrameClock, which learns the period and counts the refreshes a frame skipped, make the frame's time. It reaches the game through setupCallbacks(frameUpdate:handleInput:), renderer.frameTime and the currentFrameTime global; gameUpdate(deltaTime:) keeps its parameter. UpdateEvent, EngineExtensionUpdateContext, RenderGraphBuildContext and RenderPassContext gain a frameTime field with a default. The MTKView and AR loops pose the frame one of the view's periods ahead of the draw, or for the host's display-link target set in predictedPresentationTime. The engine stats report the presentation delta and the skipped refreshes; the XR loop counts the frames whose drawable's presentation time differs from the prediction.
Transform keys with easing, sampled at an absolute time: a frame's pose is a function of the frame's presentation time alone, so a frame the display skipped lands where the timeline says and the frames after it step by one period again. applyTransformTrack poses an entity from a track for a frame. The tests show a repeated frame does not double-step an absolute-time animation while delta-stepped motion drifts, and, with a real renderer, that a pose the game samples in frameUpdate matches the frame's presentation time rather than the wall clock, that the update event and the extensions receive the same frame time, and that a two-period gap reports one skipped frame.
… starter tutorial
untoldengine
left a comment
There was a problem hiding this comment.
Nice work on this.
One thing worth a quick discussion before merging: updateXR(presentationTime:) doesn't have a way to pass a nominalPeriod, so UntoldEngineAR.swift ends up hand-computing CACurrentMediaTime() + period and handing that to updateXR without ever telling FrameClock the known period. The clock then has to learn the period from deltas between those self-computed values, which in practice collapses back to measuring wall-clock draw-to-draw spacing — basically what the old code did. So it looks like the AR path doesn't quite get the same "immune to CPU/encode jitter" benefit that the XR path does, since only XR hands in a real compositor-predicted time rather than now + period.
Totally possible this is intentional (the inline comment hints at it), in which case it'd be great to call it out explicitly in the PR description/docs so it's a known, documented limitation rather than something a future reader has to rediscover. If it's not intentional, exposing nominalPeriod on updateXR (or a lead:-based variant) would let AR reuse the engine's own fallback logic instead of duplicating it, and give FrameClock the real period instead of learning a noisy one.
Not a blocker either way — just flagging since it's the one spot where the fix's reach seemed narrower than the rest of the PR.
…lay link target, through updateXR(nominalPeriod:)
|
You're right, and it was half intentional. Fixed in The intentional half: an The other half was an oversight. The first |
Summary
The engine side of the frame-time work the viewer bench uses (untoldengine/UntoldArcade#42): that pull request does not build against
developwithout it (cannot find type 'FrameTime'), and builds with this branch.What was wrong
On the headset the head pose is predicted for the instant the frame will be shown, but every object was posed for "now at update time": the XR loop ran the game update before its wait for the compositor's optimal input time, on a wall-clock delta between two updates, and queried the device anchor at the drawable's presentation time afterwards. The gap between the two instants is the wait plus the encode, and it varies with the previous frame's CPU work, so a moving object stepped unevenly from frame to frame while the head did not: judder at full frame rate, which the viewer's dartboard tour showed and RealityKit did not. A skipped refresh then doubled the step, and motion stepped by the time between updates drifted off the display's timeline after it.
What a frame does now
predictTiming().presentationTime, through the compositor clock's epoch, which is the boot likeCACurrentMediaTime()) and hands it toupdateXR(presentationTime:). The game update stays before the wait, where its cost sits inside the wait anyway; the engine's delta (timeSinceLastUpdate) is the time between two presentations, not between two updates, so every delta-driven path (skeletal clips, the camera path, scripts, the physics accumulator, extensions) moves in step with the display without changing.FrameTimecarries the presentation time, the delta, the period, the refreshes skipped and the frame index;FrameClockmakes it, learning the period and counting the refreshes a frame skipped. The game receives it throughsetupCallbacks(frameUpdate:handleInput:),renderer.frameTimeor thecurrentFrameTimeglobal.predictedPresentationTime, and name the view's period to the clock (updateXR(presentationTime:nominalPeriod:)for the AR draw), so the API is the same on every platform. The benefit is not: see the limitation below.TransformTrack: keys with easing sampled at an absolute time, so a frame the display skipped lands where the timeline says and the frames after it step by one period again;applyTransformTrackposes an entity from it.presentationTimeRevisions) and prints once how far the compositor clock's epoch is from the media time.docs/Architecture/xrRenderingSystem.mdanddocs/Tutorials/StarterDemo.mdsay it.On top of today's develop
Two feature commits written on #1359's branch in October, cherry-picked onto
develop. They meet two later changes incalculateDeltaTimeand the XR loop, and both are kept: the first frame after a GPU pause still has no delta (#1391's resume count resets the frame clock), and every frame the loop takes is still ended. The simulation stepping (#1393) takes the presentation delta like every other delta-driven path.Checklist
Verified
frameUpdateis the pose for the frame's presentation time rather than the wall clock, that the update event and the extensions receive the same time, and that a two-period gap reports one skipped frame.UntoldEngineTests): 1766 tests, one failure that is this checkout's folder name, not the change (ExternalRenderExtensionPackageTestsbuilds a fixture that depends on a package namedUntoldEngine, and the worktree isUntoldEngine-frametime).make strict-concurrency-checkpasses. SwiftFormat 0.62.1 reports nothing on the touched files beyondwrapIfStatementBodiesin lines this does not touch; CI's 0.60.1 was not run locally.UntoldEngineARbuilds for iOS (generic/platform=iOS).generic/platform=visionOS), which compilesUntoldEngineXRfor the device.developsince.Architectural notes / trade-offs
gameUpdate(deltaTime:)keeps its parameter, which is now the presentation delta.UpdateEvent,EngineExtensionUpdateContext,RenderGraphBuildContextandRenderPassContextgain aframeTimewith a default, so every caller compiles.MTKViewpredicts no presentation time, so on macOS, iOS and in the AR renderer the frame's time is an estimate one period ahead of the draw and the delta is still the wall-clock spacing of the draws, jitter included, as before this change. A host that drives the view from its own display link gets the exact time by settingpredictedPresentationTimeto the link's target before each draw. A clock left to learn its period keeps the shortest spacing it has seen, which is right for the compositor's times and wrong for such estimates (one short gap between two draws would make every later frame report a skipped refresh), so those loops name the view's period;FrameClockTests.testAShortGapBecomesTheLearnedPeriodpins both sides. The docs say it (StarterDemo.md, the comparison table inxrRenderingSystem.md).