What's the right approach to handle restarting a service in case one of its dependencies fails on startup (but succeeds after retry). Here's a contrived repro to make the problem clearer.a.service (simulates failure on first try and success on second try)[Unit]Description=A[Service]ExecStartPre=/bin/sh -x -c [ -f /tmp/success ] || (touch /tmp/success && sleep 10)ExecStart=/bin/trueTimeoutStartSec=5Restart=on-failureRestartSec=5RemainAfterExit=yesb.service (trivially succeeds after A starts)[Unit]Description=BAfter=a.serviceRequires=a.service[Service]ExecStart=/bin/trueRemainAfterExit=yesRestart=on-failureRestartSec=5Let's start b:# systemctl start bA dependency job for b.service failed. See 'journalctl -xe' for details.Logs:Jun 30 21:34:54 debug systemd[1]: Starting A...Jun 30 21:34:54 debug sh[1308]: + '[' -f /tmp/success ']'Jun 30 21:34:54 debug sh[1308]: + touch /tmp/successJun 30 21:34:54 debug sh[1308]: + sleep 10Jun 30 21:34:59 debug systemd[1]: a.service start-pre operation timed out. Terminating.Jun 30 21:34:59 debug systemd[1]: Failed to start A.Jun 30 21:34:59 debug systemd[1]: Dependency failed for B.Jun 30 21:34:59 debug systemd[1]: Job b.service/start failed with result 'dependency'.Jun 30 21:34:59 debug systemd[1]: Unit a.service entered failed state.Jun 30 21:34:59 debug systemd[1]: a.service failed.Jun 30 21:35:04 debug systemd[1]: a.service holdoff time over, scheduling restart.Jun 30 21:35:04 debug systemd[1]: Starting A...Jun 30 21:35:04 debug systemd[1]: Started A.Jun 30 21:35:04 debug sh[1314]: + '[' -f /tmp/success ']'A has been successfully started but B is left in a failed state and won't retry.EDITI added the following to both services and now B successfully starts when A starts, but I can't explain why.[Install]WantedBy=multi-user.targetWhy would this affect the relationship between A and B?EDIT2Above fix doesn't work in systemd 220.systemd 219 debug logssystemd219 systemd[1]: Trying to enqueue job b.service/start/replacesystemd219 systemd[1]: Installed new job b.service/start as 3454systemd219 systemd[1]: Installed new job a.service/start as 3455systemd219 systemd[1]: Enqueued job b.service/start as 3454systemd219 systemd[1]: About to execute: /bin/sh -x -c '[ -f /tmp/success ] || (touch oldcoreossystemd219 systemd[1]: Forked /bin/sh as 1502systemd219 systemd[1]: a.service changed dead -> start-presystemd219 systemd[1]: Starting A...systemd219 systemd[1502]: Executing: /bin/sh -x -c '[ -f /tmp/success ] || (touch /tmpoldcoreossystemd219 sh[1502]: + '[' -f /tmp/success ']'systemd219 sh[1502]: + touch /tmp/successsystemd219 sh[1502]: + sleep 10systemd219 systemd[1]: a.service start-pre operation timed out. Terminating.systemd219 systemd[1]: a.service changed start-pre -> final-sigtermsystemd219 systemd[1]: Child 1502 belongs to a.servicesystemd219 systemd[1]: a.service: control process exited, code=killed status=15systemd219 systemd[1]: a.service got final SIGCHLD for state final-sigtermsystemd219 systemd[1]: a.service changed final-sigterm -> failedsystemd219 systemd[1]: Job a.service/start finished, result=failedsystemd219 systemd[1]: Failed to start A.systemd219 systemd[1]: Job b.service/start finished, result=dependencysystemd219 systemd[1]: Dependency failed for B.systemd219 systemd[1]: Job b.service/start failed with result 'dependency'.systemd219 systemd[1]: Unit a.service entered failed state.systemd219 systemd[1]: a.service failed.systemd219 systemd[1]: a.service changed failed -> auto-restartsystemd219 systemd[1]: a.service: cgroup is emptysystemd219 systemd[1]: a.service: cgroup is emptysystemd219 systemd[1]: a.service holdoff time over, scheduling restart.systemd219 systemd[1]: Trying to enqueue job a.service/restart/failsystemd219 systemd[1]: Installed new job a.service/restart as 3718systemd219 systemd[1]: Installed new job b.service/restart as 3803systemd219 systemd[1]: Enqueued job a.service/restart as 3718systemd219 systemd[1]: a.service scheduled restart job.systemd219 systemd[1]: Job b.service/restart finished, result=donesystemd219 systemd[1]: Converting job b.service/restart -> b.service/startsystemd219 systemd[1]: a.service changed auto-restart -> deadsystemd219 systemd[1]: Job a.service/restart finished, result=donesystemd219 systemd[1]: Converting job a.service/restart -> a.service/startsystemd219 systemd[1]: About to execute: /bin/sh -x -c '[ -f /tmp/success ] || (touch oldcoreossystemd219 systemd[1]: Forked /bin/sh as 1558systemd219 systemd[1]: a.service changed dead -> start-presystemd219 systemd[1]: Starting A...systemd219 systemd[1]: Child 1558 belongs to a.servicesystemd219 systemd[1]: a.service: control process exited, code=exited status=0systemd219 systemd[1]: a.service got final SIGCHLD for state start-presystemd219 systemd[1]: About to execute: /bin/truesystemd219 systemd[1]: Forked /bin/true as 1561systemd219 systemd[1]: a.service changed start-pre -> runningsystemd219 systemd[1]: Job a.service/start finished, result=donesystemd219 systemd[1]: Started A.systemd219 systemd[1]: Child 1561 belongs to a.servicesystemd219 systemd[1]: a.service: main process exited, code=exited, status=0/SUCCESSsystemd219 systemd[1]: a.service changed running -> exitedsystemd219 systemd[1]: a.service: cgroup is emptysystemd219 systemd[1]: About to execute: /bin/truesystemd219 systemd[1]: Forked /bin/true as 1563systemd219 systemd[1]: b.service changed dead -> runningsystemd219 systemd[1]: Job b.service/start finished, result=donesystemd219 systemd[1]: Started B.systemd219 systemd[1]: Starting B...systemd219 systemd[1]: Child 1563 belongs to b.servicesystemd219 systemd[1]: b.service: main process exited, code=exited, status=0/SUCCESSsystemd219 systemd[1]: b.service changed running -> exitedsystemd219 systemd[1]: b.service: cgroup is emptysystemd219 sh[1558]: + '[' -f /tmp/success ']'systemd 220 debug logssystemd220 systemd[1]: b.service: Trying to enqueue job b.service/start/replacesystemd220 systemd[1]: a.service: Installed new job a.service/start as 4846systemd220 systemd[1]: b.service: Installed new job b.service/start as 4761systemd220 systemd[1]: b.service: Enqueued job b.service/start as 4761systemd220 systemd[1]: a.service: About to execute: /bin/sh -x -c '[ -f /tmp/success ] || (touch /tmp/success && sleep 10)'systemd220 systemd[1]: a.service: Forked /bin/sh as 2032systemd220 systemd[1]: a.service: Changed dead -> start-presystemd220 systemd[1]: Starting A...systemd220 systemd[2032]: a.service: Executing: /bin/sh -x -c '[ -f /tmp/success ] || (touch /tmp/success && sleep 10)'systemd220 sh[2032]: + '[' -f /tmp/success ']'systemd220 sh[2032]: + touch /tmp/successsystemd220 sh[2032]: + sleep 10systemd220 systemd[1]: a.service: Start-pre operation timed out. Terminating.systemd220 systemd[1]: a.service: Changed start-pre -> final-sigtermsystemd220 systemd[1]: a.service: Child 2032 belongs to a.servicesystemd220 systemd[1]: a.service: Control process exited, code=killed status=15systemd220 systemd[1]: a.service: Got final SIGCHLD for state final-sigterm.systemd220 systemd[1]: a.service: Changed final-sigterm -> failedsystemd220 systemd[1]: a.service: Job a.service/start finished, result=failedsystemd220 systemd[1]: Failed to start A.systemd220 systemd[1]: b.service: Job b.service/start finished, result=dependencysystemd220 systemd[1]: Dependency failed for B.systemd220 systemd[1]: b.service: Job b.service/start failed with result 'dependency'.systemd220 systemd[1]: a.service: Unit entered failed state.systemd220 systemd[1]: a.service: Failed with result 'timeout'.systemd220 systemd[1]: a.service: Changed failed -> auto-restartsystemd220 systemd[1]: a.service: cgroup is emptysystemd220 systemd[1]: a.service: Failed to send unit change signal for a.service: Transport endpoint is not connectedsystemd220 systemd[1]: a.service: Service hold-off time over, scheduling restart.systemd220 systemd[1]: a.service: Trying to enqueue job a.service/restart/failsystemd220 systemd[1]: a.service: Installed new job a.service/restart as 5190systemd220 systemd[1]: a.service: Enqueued job a.service/restart as 5190systemd220 systemd[1]: a.service: Scheduled restart job.systemd220 systemd[1]: a.service: Changed auto-restart -> deadsystemd220 systemd[1]: a.service: Job a.service/restart finished, result=donesystemd220 systemd[1]: a.service: Converting job a.service/restart -> a.service/startsystemd220 systemd[1]: a.service: About to execute: /bin/sh -x -c '[ -f /tmp/success ] || (touch /tmp/success && sleep 10)'systemd220 systemd[1]: a.service: Forked /bin/sh as 2132systemd220 systemd[1]: a.service: Changed dead -> start-presystemd220 systemd[1]: Starting A...systemd220 systemd[1]: a.service: Child 2132 belongs to a.servicesystemd220 systemd[1]: a.service: Control process exited, code=exited status=0systemd220 systemd[1]: a.service: Got final SIGCHLD for state start-pre.systemd220 systemd[1]: a.service: About to execute: /bin/truesystemd220 systemd[1]: a.service: Forked /bin/true as 2136systemd220 systemd[1]: a.service: Changed start-pre -> runningsystemd220 systemd[1]: a.service: Job a.service/start finished, result=donesystemd220 systemd[1]: Started A.systemd220 systemd[1]: a.service: Child 2136 belongs to a.servicesystemd220 systemd[1]: a.service: Main process exited, code=exited, status=0/SUCCESSsystemd220 systemd[1]: a.service: Changed running -> exitedsystemd220 systemd[1]: a.service: cgroup is emptysystemd220 systemd[1]: a.service: cgroup is emptysystemd220 systemd[1]: a.service: cgroup is emptysystemd220 systemd[1]: a.service: cgroup is emptysystemd220 sh[2132]: + '[' -f /tmp/success ']'
Restarting systemd service on dependency failure
systemd
I'll try to summarize my findings for this issue in case someone comes across this as information on this topic is scant.Restart=on-failure only applies to process failures (does not apply to failure due to dependency failures)The fact that dependent failed units get restarted under certain conditions when a dependency successfully restart was a bug in systemd < 220:http://lists.freedesktop.org/archives/systemd-devel/2015-July/033513.htmlIf there's even a small chance that a dependency might fail on start and you care about resiliency, don't use Before/After and instead perform a check on some artifact that the dependency producese.g.ExecStartPre=/usr/bin/test -f /some/thingRestart=on-failureRestartSec=5sYou could even use systemctl is-active <dependecy>.Very hacky, but I haven't found any better options.In my opinion, not having a way to handle dependency failures is a flaw in systemd.