fix: a launch blocked on devpod's workspace lock says so while it waits - #601
Merged
Merged
Conversation
devpod's lock acquire is a blocking flock with no deadline behind it. It logs `Trying to lock workspace …` every five seconds for as long as the holder lives, and the usual holder is a `devpod up` that outlived the dl that started it. `dl <ws> rm` has watched for that line since #484 and says what clears it; a launch ran its `devpod up` as a plain passthrough and read nothing, so the same wedge left every launch verb sitting silent behind devpod's own log. The launch's `up` now goes through `devpod::run_watching_stderr`, as the delete's does, and says once, on the first matching line, that devpod is waiting for another process to let go of the workspace and will wait for as long as that takes, that `dl <ws> kill` in another terminal clears it, and that `kill` deletes the workspace so the launch is typed again after. Every devpod line is still forwarded in order. One thing underneath had to move with it. `ProcessRunner::session` spawned every child with `OwnGroup::No` whatever the spec said, which was right while its only caller was an interactive `ssh -t`. `devpod up` is spawned leading its own group so the interrupt drain can killpg it (#304), and a plain swap would have dropped that silently. `session` now honours `SpawnSpec::own_group`, and the group bookkeeping it shares with `passthrough` is one helper rather than two copies. Pinned at three seams: the launch flow with a fake devpod whose `up` writes the line (said once, however many times; the launch still ends as the `up` ended), the runner (a `session` child leads its own group when asked and stays in ours by default), and the dl binary (a shim `up` that prints the line and blocks; the notice is judged mid-block, then a Ctrl-C drains at 130). Fixes #600
Reviewer's GuideLaunches now monitor Sequence diagram for a launch blocked on the workspace locksequenceDiagram
participant User
participant DL as dl
participant Runner
participant Devpod
participant OtherProcess
User->>DL: launch workspace
DL->>Runner: session(own_group=true)
Runner->>Devpod: devpod up
Devpod->>OtherProcess: waits for workspace lock
Devpod-->>Runner: Trying to lock workspace...
Runner-->>DL: stderr line callback
DL-->>User: UpBlockedOnTheLock notice
Note over User,DL: In another terminal: dl <workspace> kill
Devpod-->>Runner: forwards all stderr lines in order
User->>DL: Ctrl-C
DL->>Runner: terminate process group
Runner->>Devpod: killpg
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report❌ Patch coverage is Additional details and impacted files
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Found by running the first cut against the very orphan #600 reports, still on this host: two of devpod's "Trying to lock workspace" lines went by on stdout and dl said nothing. devpod's logger splits by level, `info` to stdout and only `error` and `fatal` to stderr, and the lock line is an `info`. The stderr-only watch the launch borrowed from `rm` was a watch on a guess about severity, which also means `rm`'s own notice from #484 has never fired against a real devpod. `Runner` gains `watched`: stdin inherited, both output streams read a line at a time and written straight back to the stream each came from, every line handed to the caller to read and not own. `session` and `watched` share one body, which is where the own-group handling now lives, so the group the interrupt drain `killpg`s is kept on both. `devpod::run_watching` replaces `run_watching_stderr` and both the delete and the launch go through it, so `rm` is fixed by the same change. Red before green at three seams: a launch unit test with the line on stdout (zero notices before), the binary test's shim now printing the lock line on stdout the way devpod does and a build line on stderr (no notice before; both streams forwarded, each to its own, after), and a delete unit test with the line on stdout. Runner tests pin `watched`: both streams reach the callback, order within a stream is kept, own group honoured, a timeout kills. The delete's deadline test matched on the recorded call being a `Session`; it is a `Watched` now. Also corrects the comments this route change had left stale: the runner's module docs and `passthrough` still named `devpod up` as their child, and `Call::own_group`'s docs spoke of a passthrough. Verified against the live orphan (kinisi-ros-nb2-jf78: notice printed once, devpod's lines forwarded to stdout) and with a complete real `devpod up` of a scratch workspace (returned on exit, provisioning ran after).
The launch-blocked-on-devpod's-lock entry, which is the whole of what is new since 0.40.0, and it fixes `rm`'s notice from #484 along with it: the watch both verbs use now reads the stream devpod actually logs that line on. Cut on this branch rather than one of its own, as 0.39.0 and 0.40.0 were: the merge that lands the fix is the push that publishes it. 0.41.0 is untaken, which `version_untaken.py` says against the pull request's base rather than against the moving tip.
This was referenced Sep 10, 2026
blooop
added a commit
that referenced
this pull request
Sep 10, 2026
devlaunch#602 item 2: "If every holder is `Attended`, there is nothing to heal: keep today's notice and today's wait, which are right for that case." The wait was kept; the notice was not. #601's line named `dl <ws> kill` as the way out, this branch took that out of the blocked notice -- rightly, because it is said before dl knows whether it can clear the lock and dl now does it unasked -- and put nothing back in the arms where dl looks and deliberately does not act. `docs/cli.md` already claimed the attended case "keeps the notice and the wait"; only the wait was true. Behind a spared build the command is named with what it costs, since `kill` deletes the workspace under somebody else's build. Behind an orphan that sat through SIGKILL no command is named, because `kill` would fail there for the reason the sweep just did, that the process is almost certainly another user's; who can end it is said instead. The comment on `up_blocked.rs`'s stderr assertion said the instruction was gone from the whole of a launch's output, which this makes untrue. It now says what the assertion actually pins: this run's sweep finds no holder, so no arm that names the verb is reachable in it.
Merged
JSmithRobotics
pushed a commit
to JSmithRobotics/devlaunch
that referenced
this pull request
Sep 16, 2026
devlaunch#601 gave a blocked launch a voice: it printed, once, that devpod was waiting for another process to let go of the workspace and that 'dl <ws> kill' in another terminal would clear it. The advice was right and the launch still waited for as long as the holder lived, which for a devpod up reparented to init is until the machine reboots (devlaunch#602). Everything that line told the reader to do, dl now does. On devpod's lock line the launch reaches the same sweep 'dl <ws> kill' runs, kills whatever holds the workspace that nothing is waiting on, and reports it by pid and command line. The up is neither restarted nor abandoned, and that is the measurement the fix rests on: devpod's acquire polls behind its five-second line, so the blocked up takes the freed flock itself. Measured on a host, one second after the holder was killed. So the sweep releases the lock and returns, and the up it is watching goes on to build. What it will not do: - Signal a holder somebody is waiting on. The distinction is kill's own Parentage, reused rather than rewritten, so it cannot drift between the two verbs. Behind a live build the launch keeps blooop#601's notice and wait. - Delete the workspace. kill deletes because you are finished with it; a launch is you asking for it, so a built workspace wedged behind an orphan gets its lock back and keeps its container and volumes. - Unlink the flock, touch devpod's busy marker, or kill a container. The marker and the containers belong to the build this launch is running. The escalation is factored into sweep_holders so workspace_kill and the new release_the_lock share one implementation; what dl <ws> kill does is unchanged, and its tests are untouched and still pass. Verified on the host: both reproductions in the ticket, cold and warm. Each second command ends connected, nothing is left holding either workspace, and fuser on both locks names nobody.
JSmithRobotics
pushed a commit
to JSmithRobotics/devlaunch
that referenced
this pull request
Sep 16, 2026
Two changes since 0.41.0, both of them merged today. A launch blocked on devpod's workspace lock now clears the orphan holding it and connects, rather than printing advice and waiting until the machine reboots (blooop#602, blooop#603). Everything blooop#601's notice told the reader to do, dl does: it reaches the same sweep `dl <ws> kill` runs, kills what nothing is waiting on, and says what it killed. The blocked `up` takes the freed flock itself, so nothing is restarted and no workspace is deleted to fix a lock. And a page for the other kind of agent, the one driving `dl` as a tool, with a `--help` pointer to it (blooop#604), corrected where it wrote down shapes the code does not produce (blooop#607): `unsaved` has five, not two, and two of the three it omitted are the ones that mean "do not delete this". Cut on a branch of its own rather than on either of theirs: both had already merged by the time the version moved, so there was no landing push left to carry it. 0.42.0 is untaken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #600.
What was wrong
devpod's lock acquire is a blocking
flockwith no deadline. It logsTrying to lock workspace …every five seconds for as long as the holder lives, and the usual holder is adevpod upthat outlived thedlthat started it.dl <ws> rmhas watched for that line since #484 and says what clears it. A launch ran itsdevpod upthrough a plain passthrough and read nothing, so the same wedge leftdl <ws>,up,restart,recreate,reset,codeanddotfilessitting silent behind devpod's own log, with nothing on the terminal about the way out.What changed
up_under_stageruns itsupthroughdevpod::run_watching, keepingleading_its_own_group(). On the first linesays_it_is_blockedmatches it says a newLaunchNotice::UpBlockedOnTheLock, once per launch. Every devpod line is still forwarded, each to the stream it came from, so the build's output is unchanged.delete_blockedand keeps its two judgements: it names another terminal, because this one is holding the launch, and it says the wait has no deadline. It adds the one thingrm's line does not owe:killdeletes the workspace, so launch again after it.Runnergainswatched(second commit). Running the first cut against the real orphan from the ticket, still on this host, showed devpod's lock line going by on stdout: devpod's logger splits by level,infoto stdout and onlyerror/fatalto stderr, and the line is aninfo. A stderr-only watch never fires against a real devpod, which also meansrm's notice from an orphaned devpod up holds the workspace flock forever, and dl has no way out #484 never has.watchedinherits stdin, reads both output streams a line at a time, writes each line straight back to its own stream, and hands every line to the caller to read and not own.sessionandwatchedshare one body.run_watching_stderrbecomesrun_watchingover it, and both the delete and the launch use it, sormis fixed by the same change. It is a widening of thedevlaunch-runnerseam: a new required trait method, recorded inpublic-api.txt, and every test wrapper in the workspace gained the method.SpawnSpec::own_group.sessionused to spawn every child withOwnGroup::Nowhatever the spec said, right while its only caller was an interactivessh -t. A plain swap would have silently dropped the process group the Ctrl-C drainkillpgs (SIGTERM runs the SIGINT drain #304) and recreated the very orphan the ticket is about. The group bookkeepingpassthroughshares with it is one helper.docs/cli.md'skillsection, CHANGELOG, and thepublic-apisnapshots (four core rows for the new arm, three runner rows forwatched).Tests, red before green
flows::launchunit tests with a fake devpod whoseupwrites the line: on stderr, on stdout, once for three repeats, still said when theupis then refused, not said for an ordinary build line. The stderr ones failed on the passthrough; the stdout one failed on the stderr-only watch.flows::lifecycle: a delete with the line on stdout stalls once (failed beforerun_watchingread both streams).devlaunch-runner:watchedhands over lines from both streams, keeps order within a stream, honours own group, kills on timeout;sessionhonours own group when asked and stays in ours by default.dl/tests/up_blocked.rs: a shimdevpodwhoseupprints a build line on stderr and the lock line twice on stdout, then blocks. dl's streams go to files so the notice is judged while dl is still blocked, then a SIGINT drains at 130. Asserts the notice nameskilland another terminal, appears exactly once, the stdout lines came back on stdout and the stderr line on stderr. Red with the shim on stdout before the second commit.Verified on the host
devpod up … kinisi-ros-nb2-jf78, PPID 1 since yesterday): the newdl kinisi-ros-nb2-jf78 upforwards devpod's lines to stdout and prints the notice once on stderr. The first cut printed nothing.devpod upof a scratch workspace of this repo throughwatched: returned on devpod's exit in about 30s, provisioning ran after it, workspace removed afterwards.cargo test --workspace, clippy-D warnings, fmt, and the Python suite (774 passed) are green locally. The interrupt suite passing is the check that theupstill comes down withdl.Not done here
The optional second half of the ticket, acting on
ContainerState::Busybefore theupis attempted, is left for its own PR as the ticket suggests. Gap 3 (why the orphan exists) is out of scope per the ticket.🤖 Generated with Claude Code
Summary by Sourcery
Make launches identify workspace-lock waits while they remain blocked and provide the command needed to recover.
Bug Fixes:
Enhancements:
Build:
Documentation:
Tests:
Chores: