We currently blindly send the configured reload signal (e.g. SIGHUP or
SIGUSR1) to the main service PID for Type=notify-reload units, even if
the service hasn't installed a userspace handler. This can lead to
invoking unintended default behavior (typically process termination) in
daemons which later deprecate and remove their reload handler.
This is a real problem we have seen in production on multiple occasions.
In one particularly egregious instance, a production distributed storage
service had a large percentage of its nodes all terminate at once when
sent a reload signal. In this case a signal handler had been removed, but
another place still sending the signal was missed.
To mitigate this, introduce signal handler validation for
Type=notify-reload services. We check both SigCgt (traditional handlers
via sigaction) and SigBlk (blocked signals for signalfd) to detect valid
handler configurations:
1. On first READY=1: When the service first sends READY=1 during initial
startup, we check for the handler. If it's missing, the service
startup is aborted with Result=protocol.
2. On reload: Immediately before sending the reload signal, we check
again. If the handler is missing, we log a warning but still send
the signal.
The rationale is that a missing handler at first startup represents a
definite misconfiguration that should be caught early. A missing handler
at reload time is warned about, but we defer to the operator's judgement
since the service already started successfully.
These checks are a best-effort, pragmatic safety net. They are naturally
racy, but in practice they prevent the common class of bugs from static
misconfiguration or software updates.
The ExitType= documentation stated that ExitType=main "cannot be used
with Type=oneshot", which is incorrect: ExitType=main is the default
and works fine with Type=oneshot. The restriction actually enforced by
the code is that ExitType=cgroup is refused for Type=oneshot services
(see service_verify() in src/core/service.c).
Move the "cannot be used with Type=oneshot" note from the ExitType=main
bullet to the ExitType=cgroup bullet where it belongs.
Fixes#42327
Signed-off-by: Liu Zheng <liuzheng@uniontech.com>
The example states that the first /bin/echo invocation (using ${ONE})
receives the argument 'one' (with literal single quotes). However,
Environment=ONE='one' strips the syntactic single quotes during
unquoting — see systemd.syntax(7), "Quotes themselves are removed" —
so ONE holds the value one, and ${ONE} (exact-value substitution,
always a single argument) yields the argument one without quotes.
Fixes#42442
Signed-off-by: Wang Yu <wangyu@uniontech.com>
We already support exponential backoff for automatic restarts via
RestartSec=/RestartSteps=/RestartMaxDelaySec=, but there is no way to
randomize the restart delay. When many instances of a service fail at
the same time (e.g. because a shared resource briefly went away) they
are all restarted in lockstep, creating a thundering herd problem.
So this commit adds a simple `RestartRandomizedDelaySec=` service
option which is similar to the timer `RandomizedDelaySec=` and
adds a randomized restart delay.
Acquiring a LUO session from /dev/liveupdate requires privileges,
and also the device is a single-owner driver so only a single
process can open it at any given time.
Add a LUOSession= service settings that allows units running
without privileges to get a session assigned to them.
The kernel imposes a 64 chars limit on session names, which is
too short to avoid clashes, so derive a hash from joining the
unit name with the parameter name, that way two units using
the same setting don't clash.
Unit-level Condition…= / Assert…= directives that fail prevent activation
entirely, so SuccessAction= / FailureAction= never fire. An ExecCondition=
skip in [Service] runs as part of the activation transition itself, so
the unit goes active → inactive and SuccessAction= is honored.
This asymmetry has tripped users up; document it explicitly in both
systemd.service(5) (ExecCondition= section) and systemd.unit(5)
(SuccessAction= / FailureAction= section) with cross-references.
Fixes: https://github.com/systemd/systemd/issues/42035
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Currently with FileDescriptorStorePreserve=yes the FD store is kept around
regardless of what happens to a unit, which is useful in many cases. But in
some cases, for example when complex services crash horribly, it's hard to
reason about what was in the intermediate state, and it's better to start
fresh.
Add a new 'on-success' option for the FileDescriptorStorePreserve= setting
that keeps it around only for as long as the unit doesn't go to a persistently
failed state.
This is especially useful in combination with LUO, where we don't want to
keep around LUO sessions created by units that then proceeded to crash and
burn, and might be in a bad state afterwards.
In order to allow FD Stores of user units to survive a user
session restart, propagate FDs received via the protocol up one
level from user to system manager via sd_notify.
And the other way around, propagate them down via LISTEN_FDS
tagging them with the unit name so that the child manager can
inject them in the appropriate unit.
Ensure units that are dead or not loaded can get FDs added to
their stores, and that they are correctly propagated once the
unit is started or loaded. When the unit is not loaded we don't
know what the FD max limit is, so simply increase it for each FD
injected, and then when the unit is realised prune it down to
match the unit's now available config in case the limit is lower
than the number of FDs in the store.
Each FD sent up or down is assigned a monotonic index, and the manager
also sends a JSON map that associates the index with the original
unit and FDNAME:
{
"unit-name.service": [
{ "name": "fdname1", "index": 1 },
{ "name": "fdname2", "index": 2 }
],
...
}
This allows the manager to assign back the FDs to the appropriate
unit using the appropriate name, given the FDNAMEs are not unique.
If a unit has FileDescriptorStorePreserve=yes we'll keep its FDs
around in case it starts again. But if there are no reverse
dependencies referencing it, we'll also GC it and lose all the FDs,
which defeats the point of the setting (which is opt-in).
Do not GC units that have FDs stored to avoid this.
Follow-up for b9c1883a9c
I found the existing explanation of RestartSteps to be simply
impenetrable. Even providing the full docs context to several of our new
AI overlords resulted in wildly different (and completely incorrect)
explanations of the final restart intervals.
Digging through the code, I found the restart delay computation in
`service_restart_usec_next` in `src/core/service.c`.
I've updated the documentation for RestartSteps with an example,
suggested value range and a detailed enough explanation that accurately
describes the current behavior.
Setting Restart=0 seems reasonable to have no delay on the first
restart, if you do not realize this is impossible with an exponential
restart. So explicitly mention that RestartSec must be set.
As announced by a few releases now, finally drop support for
sysvinit scripts.
Keep rc-local generator for now, as it's really a distinct
feature even though from the same era.
When Type=notify-reload got introduced, it wasn't intended to be
mutually exclusive with ExecReload=. However, currently ExecReload=
is immediately forked off after the service main process is signaled,
leaving states in between essentially undefined. Given so broken
it is I doubt any sane user is using this setup, hence I took a stab
to rework everything:
1. Extensions are refreshed (unchanged)
2. ExecReload= is forked off without signaling the process
3a. If RELOADING=1 is sent during the ExecReload= invocation,
we'd refrain from signaling the process again, instead
just transition to SERVICE_RELOAD_NOTIFY directly and
wait for READY=1
3b. If not, signal the process after ExecReload= finishes
(from now on the same as Type=notify-reload w/o ExecReload=)
4. To accomodate the use case of performing post-reload tasks,
ExecReloadPost= is introduced which executes after READY=1
The new model greatly simplifies things, as no control processes
will be around in SERVICE_RELOAD_SIGNAL and SERVICE_RELOAD_NOTIFY
states.
See also: https://github.com/systemd/systemd/issues/37515#issuecomment-2891229652
Type `simple` explicitly mentions that invocation failures like a missing binary
or `User=` name won’t get detected – whereas type `exec` mentions that it does.
Type `oneshot` refers to being similar to `simple`, which could lead one to
assume it doesn’t detect such invocation failures either – it seems however it
does.
Indicate this my changing its wording to be similar to `exec`.
Signed-off-by: Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>
When switching to another user it's oftentimes desirable to also spawn
the target user's shell. sudo supports this via -i flag, run0 currently
doesn't. We don't want to proactively query NSS ourselves, since
that would fall short when operating remotely. Let's instead teach
the service manager to spawn the command using the user's default shell.
I opted for "|" instead of "." in the end because the latter seems
a bit obscure. But happy to change it to something else if a better option
comes up.
Let's bump the kernel baseline a bit to 4.3 and thus require ambient
caps.
This allows us to remove support for a variety of special casing, most
importantly the ExecStart=!! hack.
Processes can easily survive the first kill operation we execute, hence
we shouldn't make strong claims about them having exited already. Let's
just say "likely" hence.
Fixes: #15032
The goal of RestartMode=direct is to make restarts invisible
to dependents, so auto restart jobs shouldn't bring them down
at all. So far we only skipped going through failed/dead states
in service_enter_dead(), i.e. the unit would never be considered
dead. But when constructing restart transaction, the stop job
would be propagated to dependents. Consider the following 2 units:
dependent.target:
[Unit]
BindsTo=a.service
After=a.service
a.service:
[Service]
ExecStart=bash -c 'sleep 100 && exit 1'
Restart=on-failure
RestartMode=direct
Before this commit, even though BindsTo= isn't triggered since
a.service never failed, when a.service auto-restarts, dependent.target
is also restarted. Let's suppress it by using JOB_REPLACE instead of
JOB_RESTART_DEPENDENCIES in service_enter_restart().
Fixes#34758
The example above is subtly different from the original report,
to illustrate that the new behavior makes sense for less exotic
use cases too.
The documentation claimed that ExecStartPre=/ExecStartPost= accepts
multiple command lines, in contrast to ExecStart=. This is half an
untruth, because ExecStart= allows that too – as long as Type=oneshot is
set.
Hence, reword this a bit, and do not emphasize the contrast.
Prompted by: #34570
So far we supported this syntax:
ExecStart=foo ; bar
as equivalent to:
ExecStart=foo
ExecStart=bar
With this change we'll "soft" deprecate the first syntax. i.e. it's
still supported in code, but not documented anymore.
The concept was originally added to make things easier for 3rd party
.ini readers, as it allowed writing unit files with a .ini framework
that doesn't allow multiple assignments for the same key. But frankly,
this is kinda pointless, as so many other of our knobs require the
double assignment.
Hence, let's just stop advertising the concept, let's simplify the docs,
by removing one entirely redundant feature from it.
Replaces: #34570
One of the major pait points of managing fleets of headless nodes is
that when something fails at startup, unless debug level was already
enabled (which usually isn't, as it's a firehose), one needs to manually
enable it and pray the issue can be reproduced, which often is really
hard and time consuming, just to get extra info. Usually the extra log
messages are enough to triage an issue.
This new option makes it so that when a service fails and is restarted
due to Restart=, log level for that unit is set to debug, so that all
setup code in pid1 and sd-executor logs at debug level, and also a new
DEBUG_INVOCATION=1 env var is passed to the service itself, so that it
knows it should start with a higher log level. Once the unit succeeds
or reaches the rate limit the original level is restored.
When credentials are used with Type=simple + ExecStartPost=,
i.e. when multiple sd-executor instances are running in parallel
for a single service, the state of final credential dir
might be unexpected wrt path_is_mount_point() and other
steps. So, let's imply Type=exec if not explicitly specified,
and emit a warning otherwise.
Make the warning for oneshot services (where RuntimeMaxSec= has no
effect) more actionable by pointing to the directive people can use
instead to effectively limit their runtime.
We are saying in public that the protocl is stable and can be easily
reimplemented, so provide an example doing so in the documentation,
license as MIT-0 so that it can be copied and pasted at will.
I think this was just overlooked in #13754, which removed
the restriction of Restart= on Type=oneshot services.
There's no reason to prevent RestartForceExitStatus=
now that Restart= has been allowed.
Closes#31148
The documentation for `RestartPreventExitStatus=` differs from that for `SuccessExitStatus=` in ways that are sometimes confusing (e.g. using `numeric exit codes` instead of `numeric termination statuses`), and other times plain incorrect (e.g. not mentioning `termination status names`, which I've just confirmed to work in systemd 255).
This patch modifies the documentation to be as similar as possible, so as to reduce the reader's cognitive load.
The only reason to recommend this would be if people had multiple commands
with the same name in the search path. This probably was never the best idea,
and it happens rarely anyway. Since the patch that dropped requirement for full
paths was introduced, we have dropped support for unmerged-usr and we're planning
to drop support for split-bin at some point too. Many people effectively have just
one directory in the search path, so there is even less reason to use an absolute
path. So let's recommend just using the command name, which makes the unit file
shorter and nicer to read.
This tries to add information about when each option was added. It goes
back to version 183.
The version info is included from a separate file to allow generating it,
which would allow more control on the formatting of the final output.
Chris Down
Liu Zheng
Wang Yu
Michael Vogt
Luca Boccassi
Rocker Zhang
Val Markovic
Mike Yuan
Matthijs Kooijman
Christoph Anton Mitterer
Taylan Kammer
Yu Watanabe
Lennart Poettering
Jan Engelhardt
Luca Boccassi
Arian van Putten
Lucas Werkmeister
hanjinpeng
Max Gautier
Andrew Sayers
David Tardon
Zbigniew Jędrzejewski-Szmek
Abderrahim Kitouni