Repository navigation
Conversation
A worker that fails to start aborts the whole run, even though starting can fail for reasons unrelated to the tests: on a heavily loaded system (e.g. 62 workers on an Aurora node, with code coverage enabled) a worker can miss Malt's 30 s deadline for reporting its port. addworker now tries starting the process three times. Errors from init_worker_code still propagate right away, as they are not transient.
Collaborator
Author
|
Not sure we want this, alternatively we just wait for JuliaPluto/Malt.jl#113. More than 60s worker startup time is probably indicative of some other failure. |
Collaborator
Agreed on both points. |
Collaborator
Author
|
Malt 1.5 is released and has this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A worker that fails to start aborts the whole run:
addworkeris called outside the per-testtry, so its error ends up in "Caught an error, stopping..." and every test still queued is reported as interrupted. Starting a worker can fail for reasons that have nothing to do with the tests. On oneAPI.jl's Aurora CI (62 workers on one node, with code coverage) a respawned worker missed Malt's hard-coded 30 s deadline for reporting its port:addworkernow tries starting the process up to three times before rethrowing. Errors frominit_worker_codestill propagate right away, as they are deterministic.The test uses a launcher as
exenamewhose first start prints something other than a port, which fails the same way without waiting for the deadline. It fails without the change and passes with it.JuliaPluto/Malt.jl#113 raises Malt's deadline to 60 s and makes it configurable, but it is unreleased, and this keeps runs from dying on a single slow start regardless of the deadline. That PR also fixes something PTR can't: Malt 1.4.1 leaves a process that timed out running, with no handle to stop it, so a failed attempt can leave a stray worker behind (as an aborted run already does today).