fix: sha256:de4600779cefea10176aeb9f4590ed488b3a0b40300d5389b311d226fde67ca8 · gabriel/musehub

fix: terminally fail stale jobs that already exhausted their attempts

reclaim_stale_jobs() only ever reset stale 'running' rows with attempt < _MAX_ATTEMPTS back to 'pending' for retry. Rows already at _MAX_ATTEMPTS were silently skipped -- correct in isolation (no more retries), but nothing else ever moved them to a terminal 'failed' state either, since fail_job() is only called from the normal execution path's exception handler, which a hard worker crash/restart never reaches (attempt is incremented at claim time, before the job body runs).

This went unnoticed for months because dedup (enqueue_job) only ever checked status='pending'. It became a live-blocking bug the moment MWP9's dedup fix started treating 'running' as coverage for self-coalescing job types: two fetch.mpack.prebuild rows found stuck at status=running, attempt=3 since July 2 and September 22 permanently blocked every new prebuild trigger for gabriel/musehub on staging, because dedup correctly (but unknowingly) saw them as already-covering-the-work rows that would in fact never complete. Confirmed live: muse clone wedged indefinitely after the MWP9 dedup fix deployed, for exactly this reason.

Fix: reclaim_stale_jobs now does two passes -- exhausted stale rows are marked 'failed' (terminal, matching fail_job's own terminal branch); under-budget stale rows keep resetting to 'pending' as before.

symbols added

symbols modified

files changed

1084

files in snapshot

dead code introduced

Semantic Changes 9 symbols

~ musehub/services/musehub_jobs.py .py 1 symbol modified

~ reclaim_stale_jobs

~ tests/test_mwp9_running_job_dedup.py .py 5 symbols added, 3 symbols modified

+ _MAX_ATTEMPTS import import _MAX_ATTEMPTS L40–40

+ reclaim_stale_jobs import import reclaim_stale_jobs L40–40

+ timedelta import import timedelta L32–32

+ test_mwp9_05_exhausted_stale_running_job_marked_failed_not_left_running function async_function test_mwp9_05_exhausted_stale_running_job_marked_failed_not_left_running L169–216

+ test_mwp9_06_stale_running_job_under_budget_still_resets_to_pending function async_function test_mwp9_06_stale_running_job_under_budget_still_resets_to_pending L220–248

~ datetime

~ enqueue_push_intel

~ timezone

Files Changed

1084 in snapshot

~ musehub/services/musehub_jobs.py .py

~ tests/test_mwp9_running_job_dedup.py .py

← Older Oldest on fix/reclaim-exhausted-stale-jobs

All commits

Newer → Latest on fix/reclaim-exhausted-stale-jobs

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:de4600779cefea10176aeb9f4590ed488b3a0b40300d5389b311d226fde67ca8 --body "your comment"