Koha Solutions knowledge base: run log of a live check Article https://kohasolutions.com/knowledge-base/koha-background-jobs-stuck-new Check scripts/kb/live/koha-background-jobs-stuck-new.mjs (SHA-256 7c4b611b43f17474b969786a14e32c011cab60fb5d13463eabfbc7cebc528613) Ran 2026-11-08T05:56:41Z (UTC), 207 s Koha 26.05.05 (Koha::version 26.05.05.000, database version 26.0505000) Source https://git.koha-community.org/Koha-community/Koha/src/commit/d3fc229bc5eaea25dfe88f86149386408fe5d32d System Ubuntu 24.04.5 LTS Setup A fresh Ubuntu 24.04 server with Koha installed from the Koha repository as our installation guide leaves it (MariaDB, RabbitMQ, one instance), reached over SSH: every command run as the sudo user librarian on a terminal, then a reboot. Result PASS: all 20 claims held Each claim is something the article says. "seen" is what the check read back when it is not just a yes/no. Changes the check made to the test server were undone afterwards, as its header describes. == Claims == ok the server box starts with Koha installed ok after boot, --status --all-queues shows both queues running: " * worker running for library (default) … [ OK ]" and (long_tasks) seen: * worker running for library (default) [ OK ] * worker running for library (long_tasks) [ OK ] ok --stop --queue long_tasks → "Stopping Koha worker daemon for library (long_tasks)" / " ...done." seen: Stopping Koha worker daemon for library (long_tasks) ...done. ok --status --all-queues → default "[ OK ]", " * worker not running for library (long_tasks) … [fail]" seen: * worker running for library (default) [ OK ] * worker not running for library (long_tasks) [fail] ok --all-queues --start → "Error: worker already running for library (default)" / "Starting Koha worker daemon for library (long_tasks)" / " ...fail!" seen: Error: worker already running for library (default) Starting Koha worker daemon for library (long_tasks) ...fail! ok …yet a few seconds later both queues show "[ OK ]" seen: * worker running for library (default) [ OK ] * worker running for library (long_tasks) [ OK ] ok worker-error.log: " library-koha-worker-long_tasks: client (pid ) killed by signal 15, stopping" for that stop seen: 20261108 05:41:31 library-koha-worker-long_tasks: client (pid 12306) killed by signal 15, respawning / 20261108 05:41:35 library-koha-worker: client (pid 12516) killed by signal 15, stopping / 20261108 05:41:35 library-koha-worker-long_tasks: client (pid 12517) killed by signal 15, stopping / 20261108 05:57:18 library-koha-worker-long_tasks: client (pid 400) killed by signal 15, stopping ok worker-error.log also has the "killed by signal 15, respawning" lines of a restart seen: 20261108 05:41:31 library-koha-worker: client (pid 12299) killed by signal 15, respawning / 20261108 05:41:31 library-koha-worker-long_tasks: client (pid 12306) killed by signal 15, respawning / 20261108 05:41:35 library-koha-worker: client (pid 12516) killed by signal 15, stopping ok systemctl stop rabbitmq-server prints nothing ok enqueue with the broker down warns "Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused" seen: Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at ok …and both jobs are saved as new seen: new,new ok --all-queues --restart → "Restarting Koha worker daemon for library (default)" / "(long_tasks)" / " ...done." seen: Restarting Koha worker daemon for library (default) Restarting Koha worker daemon for library (long_tasks) ...done. ok broker back and workers restarted (STOMP): the jobs are STILL new seen: new,new ok polling + the same restart (same three lines) seen: Restarting Koha worker daemon for library (default) Restarting Koha worker daemon for library (long_tasks) ...done. ok JobsNotificationMethod polling + restart: both jobs finish within about ten to thirty seconds seen: finished,finished ok systemctl status rabbitmq-server → "● rabbitmq-server.service - RabbitMQ Messaging Server", "Loaded: … enabled; preset: enabled)", "Active: active (running) since …" seen: ● rabbitmq-server.service - RabbitMQ Messaging Server Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled) Active: active (running) since Sun 2026-11-08 05:58:06 UTC; 1min 1s ago ok rabbitmq-plugins enable rabbitmq_stomp (already on) → "Enabling plugins on node rabbit@koha-server:" … "Plugin configuration unchanged." seen: Enabling plugins on node rabbit@koha-server: rabbitmq_stomp The following plugins have been configured: rabbitmq_stomp Applying plugin configuration to rabbit@koha-server... Plugin configuration unchanged. ok systemctl restart rabbitmq-server prints nothing, and STOMP listens on 61613 ok systemctl is-enabled koha-common → "enabled" seen: enabled ok after a reboot both queues run again with nobody starting them seen: * worker running for library (default) [ OK ] * worker running for library (long_tasks) [ OK ] == The check that produced this log (scripts/kb/live/koha-background-jobs-stuck-new.mjs) == /** * Live check for prisma/ks-kb/37-koha-background-jobs-stuck-new.ts, on a real server since 2026-11-08 (plan212). * * A fresh Ubuntu 24.04 server with Koha installed as the installation guide leaves it (box.mjs ensureKohaImage, the * image is rebuilt from the Koha repository at most a day old), reached over SSH as the sudo user `librarian`; every * command the article shows is run there, on a terminal, as a reader runs it: * 1. both queues running after boot; long_tasks stopped; --status, --all-queues --start, --status again; * 2. worker-error.log's lines for that stop; * 3. RabbitMQ stopped, two empty BatchCancelHold jobs queued ("Cannot connect to broker"), saved as new; RabbitMQ * started and the workers restarted: still new; JobsNotificationMethod set to polling (set_preference, as the * preferences screen does) and the workers restarted: finished; * 4. systemctl status rabbitmq-server, rabbitmq-plugins enable rabbitmq_stomp, systemctl restart; * 5. the callout: koha-common is enabled, and after a reboot both queues run again with nobody starting them. * Finding of the first server run: on a terminal koha-worker prints Debian's init style (" * worker running for … * [ OK ]", "[fail]", "...fail!", "...done."), not the ":." / " failed!" / "." of a run with no terminal, which is * what the article used to show. The box is removed at the end. ~3 minutes once the image exists. */ import { createBox, ensureKohaImage, KOHA_IMAGE, rebootAndWait, removeBox, ssh } from './server-box/box.mjs' /** liveKoha: false — runs only on its own server box, so kb:verify neither starts nor reads live Koha. */ export const meta = { koha: '26.05.05', passed: '2026-11-08', liveKoha: false } const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) // milliseconds /** The terminal line koha-worker prints: init style, "[ OK ]" or "[fail]" at column 74. */ const line = (text, ok) => ` * ${text}`.padEnd(73) + (ok ? '[ OK ]' : '[fail]') export default async function run(ctx) { const err = ensureKohaImage() ?? createBox({ image: KOHA_IMAGE }) ctx.check('the server box starts with Koha installed', err === null, err ?? '') if (err) return try { await body(ctx) } finally { removeBox() } } async function body(ctx) { const L = (cmd) => ssh('librarian', cmd).replace(/\s+$/, '') const b64 = (s) => Buffer.from(s).toString('base64') const perl = (code, args = '') => L(`echo ${b64(code)} | base64 -d > /tmp/kbv.pl && chmod 644 /tmp/kbv.pl && sudo koha-shell library -c "perl /tmp/kbv.pl ${args}"; rm -f /tmp/kbv.pl`) const jobs = () => L(`sudo koha-mysql library -N -B -e "SELECT status FROM background_jobs WHERE type='batch_hold_cancel' ORDER BY id"`).replace(/\n/g, ',') const status = () => L('sudo koha-worker --status --all-queues library') const both = `${line('worker running for library (default)', true)}\n${line('worker running for library (long_tasks)', true)}` let s = '' for (let i = 0; i < 20 && s !== both; i++) { await sleep(3000); s = status() } ctx.check('after boot, --status --all-queues shows both queues running: " * worker running for library (default) … [ OK ]" and (long_tasks)', s === both, s) // 1. The half-working state. const stop = L('sudo koha-worker --stop --queue long_tasks library') ctx.check('--stop --queue long_tasks → "Stopping Koha worker daemon for library (long_tasks)" / " ...done."', stop === 'Stopping Koha worker daemon for library (long_tasks)\n ...done.', stop) await sleep(2000) const half = status() ctx.check('--status --all-queues → default "[ OK ]", " * worker not running for library (long_tasks) … [fail]"', half === `${line('worker running for library (default)', true)}\n${line('worker not running for library (long_tasks)', false)}`, half) const start = L('sudo koha-worker --all-queues --start library') ctx.check('--all-queues --start → "Error: worker already running for library (default)" / "Starting Koha worker daemon for library (long_tasks)" / " ...fail!"', start === 'Error: worker already running for library (default)\nStarting Koha worker daemon for library (long_tasks)\n ...fail!', start) await sleep(4000) const after = status() ctx.check('…yet a few seconds later both queues show "[ OK ]"', after === both, after) // 2. The worker log. const log = L('sudo tail -n 50 /var/log/koha/library/worker-error.log') ctx.check('worker-error.log: " library-koha-worker-long_tasks: client (pid ) killed by signal 15, stopping" for that stop', /^\d{8} \d\d:\d\d:\d\d library-koha-worker-long_tasks: client \(pid \d+\) killed by signal 15, stopping$/m.test(log), log.split('\n').slice(-4).join(' / ')) ctx.check('worker-error.log also has the "killed by signal 15, respawning" lines of a restart', /^\d{8} \d\d:\d\d:\d\d library-koha-worker: client \(pid \d+\) killed by signal 15, respawning$/m.test(log), log.split('\n').slice(0, 3).join(' / ')) // 3. Jobs queued while the broker was down. ctx.check('systemctl stop rabbitmq-server prints nothing', L('sudo systemctl stop rabbitmq-server') === '') await sleep(2000) const enq = perl("use Modern::Perl; use Koha::BackgroundJob::BatchCancelHold; for (1..2) { print Koha::BackgroundJob::BatchCancelHold->new->enqueue({ hold_ids => [], reason => q{kbverify} }), qq{\\n} }") ctx.check('enqueue with the broker down warns "Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused"', enq.includes('Cannot connect to broker (Failed to connect: Error connecting to localhost:61613: Connection refused'), enq.slice(0, 200)) ctx.check('…and both jobs are saved as new', jobs() === 'new,new', jobs()) L('sudo systemctl start rabbitmq-server') await sleep(8000) const rs1 = L('sudo koha-worker --all-queues --restart library') ctx.check('--all-queues --restart → "Restarting Koha worker daemon for library (default)" / "(long_tasks)" / " ...done."', rs1 === 'Restarting Koha worker daemon for library (default)\nRestarting Koha worker daemon for library (long_tasks)\n ...done.', rs1) await sleep(15000) ctx.check('broker back and workers restarted (STOMP): the jobs are STILL new', jobs() === 'new,new', jobs()) perl('use C4::Context; C4::Context->set_preference($ARGV[0], $ARGV[1]);', 'JobsNotificationMethod polling') const rs2 = L('sudo koha-worker --all-queues --restart library') ctx.check('polling + the same restart (same three lines)', rs2 === rs1, rs2) let done = '' for (let i = 0; i < 15 && done !== 'finished,finished'; i++) { await sleep(2000); done = jobs() } ctx.check('JobsNotificationMethod polling + restart: both jobs finish within about ten to thirty seconds', done === 'finished,finished', done) perl('use C4::Context; C4::Context->set_preference($ARGV[0], $ARGV[1]);', 'JobsNotificationMethod STOMP') L('sudo koha-worker --all-queues --restart library') // 4. RabbitMQ. const st = L('sudo systemctl status rabbitmq-server --no-pager | head -3') ctx.check('systemctl status rabbitmq-server → "● rabbitmq-server.service - RabbitMQ Messaging Server", "Loaded: … enabled; preset: enabled)", "Active: active (running) since …"', /^● rabbitmq-server\.service - RabbitMQ Messaging Server\n\s+Loaded: loaded \(\/usr\/lib\/systemd\/system\/rabbitmq-server\.service; enabled; preset: enabled\)\n\s+Active: active \(running\) since /.test(st), st) const pl = L('sudo rabbitmq-plugins enable rabbitmq_stomp') ctx.check('rabbitmq-plugins enable rabbitmq_stomp (already on) → "Enabling plugins on node rabbit@koha-server:" … "Plugin configuration unchanged."', pl === 'Enabling plugins on node rabbit@koha-server:\nrabbitmq_stomp\nThe following plugins have been configured:\n rabbitmq_stomp\nApplying plugin configuration to rabbit@koha-server...\nPlugin configuration unchanged.', pl) ctx.check('systemctl restart rabbitmq-server prints nothing, and STOMP listens on 61613', L('sudo systemctl restart rabbitmq-server') === '' && /:61613\s/.test(L('ss -ltn'))) // 5. The callout: the service starts both queues at boot. const en = L('sudo systemctl is-enabled koha-common') ctx.check('systemctl is-enabled koha-common → "enabled"', en === 'enabled', en) const back = rebootAndWait('librarian') let r = '' for (let i = 0; back && i < 20 && r !== both; i++) { await sleep(3000); r = status() } ctx.check('after a reboot both queues run again with nobody starting them', back && r === both, r) const env = L(`grep PRETTY_NAME /etc/os-release; perl -I/usr/share/koha/lib -MKoha -e 'print qq{KOHA=}, Koha::version, qq{\\n}'; sudo koha-mysql library -N -B -e "SELECT CONCAT('DB=', value) FROM systempreferences WHERE variable='Version'"`) ctx.ranIn({ osRelease: (env.match(/^PRETTY_NAME=.*$/m) ?? [''])[0], kohaVersion: (env.match(/^KOHA=(\S+)/m) ?? [])[1] ?? '', db: (env.match(/^DB=(\S+)/m) ?? [])[1] ?? '' }, 'A fresh Ubuntu 24.04 server with Koha installed from the Koha repository as our installation guide leaves it (MariaDB, RabbitMQ, one instance), reached over SSH: every command run as the sudo user librarian on a terminal, then a reboot.') }