Koha Solutions knowledge base: run log of a live check Article https://kohasolutions.com/knowledge-base/koha-background-jobs-stuck-new Check scripts/kb/live/koha-background-jobs-stuck-new.mjs (SHA-256 feed6c6f155e9c118a0bc026b2903e7f5230e236d8db2e8eda01635c201d353c) Ran 2026-11-05T15:00:32Z (UTC), 59 s Koha 26.05.02 (Koha::version 26.05.02.002, database version 26.0600002) Source https://git.koha-community.org/Koha-community/Koha/src/commit/d3fc229bc5eaea25dfe88f86149386408fe5d32d System Debian GNU/Linux 13 (trixie) Result PASS: all 13 claims held Each claim is something the article says. "seen" is what the check read back when it is not just a yes/no. Changes the check made to the test server were undone afterwards, as its header describes. == Claims == ok both queues running at the start seen: worker running for kohadev (default):. worker running for kohadev (long_tasks):. ok --status --all-queues → "worker running for (default):." / "worker not running for (long_tasks): failed!" seen: worker running for kohadev (default):. worker not running for kohadev (long_tasks): failed! ok --all-queues --start → "Error: worker already running for (default)" / "Starting Koha worker daemon for (long_tasks)" / " failed!" seen: "Error: worker already running for kohadev (default)\nStarting Koha worker daemon for kohadev (long_tasks)\n failed!" ok …yet a few seconds later both queues are running seen: worker running for kohadev (default):. worker running for kohadev (long_tasks):. ok worker-error.log: " -koha-worker-long_tasks: client (pid ) killed by signal 15, stopping" for that stop seen: pawning 20261105 14:52:06 kohadev-koha-worker: client (pid 226492) killed by signal 15, respawning 20261105 14:52:06 kohadev-koha-worker-long_tasks: client (pid 226493) killed by signal 15, respawning 20261105 15:00:32 kohadev-koha-worker-long_tasks: client (pid 226689) killed by signal 15, stopping ok enqueue with the broker down warns "Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused …" seen: Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at ok …and the two jobs are saved as new seen: new,new ok broker back and workers restarted (STOMP): the jobs are STILL new seen: new,new ok --all-queues --restart → "Restarting Koha worker daemon for (default)" / "Restarting Koha worker daemon for (long_tasks)" / "." seen: "Restarting Koha worker daemon for kohadev (default)\nRestarting Koha worker daemon for kohadev (long_tasks)\n." ok JobsNotificationMethod polling + restart: both jobs finish within about ten to thirty seconds seen: finished,finished ok Ubuntu 24.04: systemctl status rabbitmq-server → "● rabbitmq-server.service - RabbitMQ Messaging Server", "Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)", "Active: active (running) since …" seen: ● rabbitmq-server.service - RabbitMQ Messaging Server Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled) Active: active (running) since Thu 2026-11-05 15:01:21 UTC; 92ms ago ok rabbitmq-plugins enable rabbitmq_stomp (already on) → "Enabling plugins on node rabbit@library:" … "Plugin configuration unchanged." seen: Enabling plugins on node rabbit@library: rabbitmq_stomp The following plugins have been configured: rabbitmq_stomp Applying plugin configuration to rabbit@library... Plugin configuration unchanged. ok systemctl restart rabbitmq-server prints nothing, and STOMP listens on 61613 == The check that produced this log (scripts/kb/live/koha-background-jobs-stuck-new.mjs) == /** * Live check for prisma/ks-kb/37-koha-background-jobs-stuck-new.ts. On live Koha: stops the long_tasks worker and * starts both queues again (the status output the article shows); then the stuck-at-New case: RabbitMQ (which runs * inside the koha container) is stopped, two empty BatchCancelHold jobs are queued, RabbitMQ is started and the * workers restarted (jobs stay new), JobsNotificationMethod is switched to polling through set_preference and the * workers restarted (jobs finish), then STOMP is put back. The test jobs are deleted afterwards. The systemctl / * rabbitmq-plugins block runs in a throwaway kbv-koha-ubuntu:24.04 box (systemd). About 2.5 minutes. */ import { spawnSync } from 'node:child_process' export const meta = { koha: '26.05.02', passed: '2026-11-05' } const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) // milliseconds const IMG = 'kbv-koha-ubuntu:24.04' const BOX = 'kbv-37-check' export default async function run(ctx) { const I = ctx.INSTANCE const status = () => ctx.sh(`koha-worker --status --all-queues ${I} 2>&1`).out const pref = (v) => { ctx.sh(`printf '%s\\n' 'use C4::Context; C4::Context->set_preference($ARGV[0], $ARGV[1]);' > /tmp/kbv-p.pl; chmod 644 /tmp/kbv-p.pl`) ctx.sh(`koha-shell ${I} -c "perl /tmp/kbv-p.pl JobsNotificationMethod ${v}"; rm -f /tmp/kbv-p.pl`) } const oldMethod = ctx.sqlValue("SELECT value FROM systempreferences WHERE variable = 'JobsNotificationMethod'") const firstJob = Number(ctx.sqlValue('SELECT COALESCE(MAX(id), 0) FROM background_jobs')) ctx.restore('broker running, JobsNotificationMethod as before, both workers running, test jobs deleted', () => { ctx.sh('service rabbitmq-server start >/dev/null 2>&1; true') pref(oldMethod ?? 'STOMP') ctx.sh(`koha-worker --all-queues --restart ${I} >/dev/null 2>&1; true`) ctx.sql(`DELETE FROM background_jobs WHERE id > ${firstJob} AND type = 'batch_hold_cancel'`) }) ctx.check('both queues running at the start', status() === `worker running for ${I} (default):.\nworker running for ${I} (long_tasks):.`, status()) // 1. The half-working state and the misleading " failed!". ctx.sh(`koha-worker --stop --queue long_tasks ${I} >/dev/null 2>&1`) await sleep(2000) const half = status() ctx.check('--status --all-queues → "worker running for (default):." / "worker not running for (long_tasks): failed!"', half === `worker running for ${I} (default):.\nworker not running for ${I} (long_tasks): failed!`, half) const start = ctx.sh(`koha-worker --all-queues --start ${I} 2>&1`).out ctx.check('--all-queues --start → "Error: worker already running for (default)" / "Starting Koha worker daemon for (long_tasks)" / " failed!"', start === `Error: worker already running for ${I} (default)\nStarting Koha worker daemon for ${I} (long_tasks)\n failed!`, JSON.stringify(start)) await sleep(3000) ctx.check('…yet a few seconds later both queues are running', status() === `worker running for ${I} (default):.\nworker running for ${I} (long_tasks):.`, status()) const log = ctx.sh(`tail -n 50 /var/log/koha/${I}/worker-error.log`).out ctx.check('worker-error.log: " -koha-worker-long_tasks: client (pid ) killed by signal 15, stopping" for that stop', new RegExp(`^\\d{8} \\d\\d:\\d\\d:\\d\\d ${I}-koha-worker-long_tasks: client \\(pid \\d+\\) killed by signal 15, stopping$`, 'm').test(log), log.slice(-300)) // 2. Jobs queued while the broker was down stay at New after it comes back. ctx.sh('service rabbitmq-server stop >/dev/null 2>&1') await sleep(2000) ctx.sh(`printf '%s\\n' 'use Modern::Perl; use Koha::BackgroundJob::BatchCancelHold; for (1..2) { print Koha::BackgroundJob::BatchCancelHold->new->enqueue({ hold_ids => [], reason => q{kbverify} }), "\\n" }' > /tmp/kbv-enq.pl; chmod 644 /tmp/kbv-enq.pl`) const enq = ctx.sh(`koha-shell ${I} -c "perl /tmp/kbv-enq.pl" 2>&1; rm -f /tmp/kbv-enq.pl`).out ctx.check('enqueue with the broker down warns "Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused …"', enq.includes('Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused'), enq.slice(0, 200)) const jobs = () => ctx.sql(`SELECT status FROM background_jobs WHERE id > ${firstJob} AND type = 'batch_hold_cancel' ORDER BY id`).map((r) => r[0]).join(',') ctx.check('…and the two jobs are saved as new', jobs() === 'new,new', jobs()) ctx.sh('service rabbitmq-server start >/dev/null 2>&1') await sleep(8000) ctx.sh(`koha-worker --all-queues --restart ${I} >/dev/null 2>&1`) await sleep(15000) ctx.check('broker back and workers restarted (STOMP): the jobs are STILL new', jobs() === 'new,new', jobs()) // 3. The fix: polling, restart; then back to STOMP. pref('polling') const rs = ctx.sh(`koha-worker --all-queues --restart ${I} 2>&1`).out ctx.check('--all-queues --restart → "Restarting Koha worker daemon for (default)" / "Restarting Koha worker daemon for (long_tasks)" / "."', rs === `Restarting Koha worker daemon for ${I} (default)\nRestarting Koha worker daemon for ${I} (long_tasks)\n.`, JSON.stringify(rs)) let done = '' for (let i = 0; i < 15 && done !== 'finished,finished'; i++) { await sleep(2000); done = jobs() } ctx.check('JobsNotificationMethod polling + restart: both jobs finish within about ten to thirty seconds', done === 'finished,finished', done) // 4. The RabbitMQ commands, on a server with systemd. if (spawnSync('docker', ['image', 'inspect', IMG]).status !== 0) { ctx.check(`image ${IMG} present`, false); return } spawnSync('docker', ['rm', '-f', BOX]) ctx.restore('remove the Ubuntu box', () => spawnSync('docker', ['rm', '-f', BOX])) spawnSync('docker', ['run', '-d', '--name', BOX, '--hostname', 'library.example.org', '--privileged', '--cgroupns=host', '-v', '/sys/fs/cgroup:/sys/fs/cgroup:rw', IMG, '/lib/systemd/systemd']) const u = (cmd) => (spawnSync('docker', ['exec', BOX, 'bash', '-c', `{ ${cmd}\n} 2>&1`], { encoding: 'utf8', timeout: 300000 }).stdout ?? '').replace(/\s+$/, '') u('for i in $(seq 1 60); do systemctl is-system-running --wait >/dev/null 2>&1 && break; sleep 1; done; for i in $(seq 1 30); do systemctl is-active rabbitmq-server >/dev/null && break; sleep 1; done') const st = u('systemctl status rabbitmq-server --no-pager | head -3') ctx.check('Ubuntu 24.04: systemctl status rabbitmq-server → "● rabbitmq-server.service - RabbitMQ Messaging Server", "Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)", "Active: active (running) since …"', /^● rabbitmq-server\.service - RabbitMQ Messaging Server\n\s+Loaded: loaded \(\/usr\/lib\/systemd\/system\/rabbitmq-server\.service; enabled; preset: enabled\)\n\s+Active: active \(running\) since /.test(st), st) const pl = u('rabbitmq-plugins enable rabbitmq_stomp') ctx.check('rabbitmq-plugins enable rabbitmq_stomp (already on) → "Enabling plugins on node rabbit@library:" … "Plugin configuration unchanged."', pl === 'Enabling plugins on node rabbit@library:\nrabbitmq_stomp\nThe following plugins have been configured:\n rabbitmq_stomp\nApplying plugin configuration to rabbit@library...\nPlugin configuration unchanged.', pl) ctx.check('systemctl restart rabbitmq-server prints nothing, and STOMP listens on 61613', u('systemctl restart rabbitmq-server') === '' && /:61613\s/.test(u('ss -ltn'))) }