Extend fake-report-server.py with optional --cert, --key, --port
arguments for TLS support. Add a test case that generates a
self-signed certificate and tests HTTPS upload of metrics and facts.
Also exercise the --header param.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Add a fake HTTP server (fake-report-server.py) that accepts JSON POST
requests and validates the report structure, and test cases in
TEST-74-AUX-UTILS.report.sh that exercise plain HTTP upload of both
metrics and facts.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Ensure only privileged users can call the system scope machined's
APIs that get data out of a machine
Follow-up for 1bd979dddb
Follow-up for 9153b02bb5
Test the async QMP drive pipeline with real QEMU:
Test 1 (multi-drive): launches vmspawn with --image plus two
--extra-drive flags. This exercises multiple fdset allocations,
pipelined blockdev-add commands relying on FIFO ordering, io_uring
retry callbacks, and multiple device_add commands — all fired
without waiting for responses.
Test 2 (ephemeral): launches vmspawn with --image --ephemeral. This
exercises the most complex async path: blockdev-create fires a
background job, JOB_STATUS_CHANGE events are watched via the event
callback, and when the job concludes the deferred continuation fires
the overlay format node + device_add. If the continuation fails, the
root drive is never attached, the kernel panics, and vmspawn exits
without registering — so successful registration proves the pipeline
works.
Both tests use a raw ext4 image with a minimal init (sleep infinity)
and direct kernel boot. No virtiofsd needed.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Add TEST-87-AUX-UTILS-VM.vmspawn.sh that validates the QMP-varlink
bridge end-to-end using a real QEMU instance:
- Launches vmspawn with --directory and --linux for direct kernel boot
(no UEFI firmware or bootable image needed)
- Waits for machine registration with machined
- Verifies varlinkAddress is exposed in Machine.List
- Tests machinectl pause, resume, poweroff
- Exercises MachineInstance varlink interface directly via varlinkctl:
QueryStatus state verification across pause/resume, Pause, Resume
Skipped automatically if vmspawn, QEMU, or a bootable kernel is not
available. Runs as part of TEST-87-AUX-UTILS-VM in the mkosi
integration test suite.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Using the format specifier +%s%6N with GNU date is honored, and only
prints 6 digits of the nanoseconds portion of the seconds since epoch.
The uutils implementation of date does not honor this, and always prints
all 9 digits. This is a known bug[1], but can be worked around by
adapting this test to use nanoseconds instead of microseconds.
[1] https://github.com/uutils/coreutils/issues/11658
Otherwise you run into errors such as:
"""
../meson.build:2899:28: ERROR: File src/test/test-loop-util.c does not exist.
"""
when deleting a file in git without staging the deletion.
Add CpuUsage, MemoryUsage, IOReadBytes, IOReadOperations, and
TasksCurrent in a standalone socket-activated varlink service. These
metrics are gathered from the kernel via cgroup files and PID1's only
role is mapping unit names to cgroup paths — a separate process can
query PID1 once for that mapping and then read the cgroup files
directly, minimizing PID1 involvement.
The new systemd-report-cgroup-metrics service listens at
/run/systemd/report/io.systemd.CGroup and exposes:
- io.systemd.CGroup.CpuUsage
- io.systemd.CGroup.IOReadBytes
- io.systemd.CGroup.IOReadOperations
- io.systemd.CGroup.MemoryUsage (with type=current/available/peak)
- io.systemd.CGroup.TasksCurrent
This is spun out of #41078 and based on top of it. Will rebase once
that's merged.
Add CpuUsage, MemoryUsage, IOReadBytes, IOReadOperations, and
TasksCurrent in a standalone socket-activated varlink service.
The new systemd-report-cgroup service listens at
/run/systemd/report/io.systemd.CGroup and exposes:
- io.systemd.CGroup.CpuUsage
- io.systemd.CGroup.IOReadBytes
- io.systemd.CGroup.IOReadOperations
- io.systemd.CGroup.MemoryUsage (with type=current/available/peak)
- io.systemd.CGroup.TasksCurrent
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
With the new protocol upgrade support in varlinkctl client we can
now do the equivalent for the server side. This commit adds a new
`serve` verb that will serve any command that speaks stdin/stdout
via varlink and its protocol upgrade feature. This is the
"inetd for varlink".
This is useful for various reasons:
1. Allows to e.g. provide a heavily sandboxed io.myorg.xz.Decompress
varlink endpoint, c.f. xz CVE-2024-3094)
2. Allow sftp over varlink which is quite useful with the
varlink-http-bridge (that has more flexible auth mechanism than
plain sftp).
3. Makes testing the varlinkctl client protocol upgrade simpler.
4. Because we can.
Teach swap units to support the _netdev option as well, which should
make swaps on iSCSI possible. This mirrors the logic we already have for
regular mounts in both the fstab-generator and the core
(mount.c/swap.c).
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Rename nspawn's --user=NAME option to --uid=NAME for selecting the
container user. The -u short option is preserved. --user=NAME and
--user NAME are still accepted but emit a deprecation warning. A
pre-parsing step stitches the space-separated --user NAME form into
--user=NAME before getopt sees it, preserving backwards compatibility
despite --user now being an optional_argument.
Repurpose --user (without argument) and --system as standalone
switches for selecting the runtime scope (user vs system service
manager).
Replace all uses of the arg_privileged boolean with
arg_runtime_scope comparisons throughout nspawn. The default scope
is auto-detected from the effective UID.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
TEST-74-AUX-UTILS has a number of subtests.
test/units/TEST-74-AUX-UTILS.capsule.sh runs first and starts and stops
capsule@foobar.service. Looking at the test, the unit is cleanly stopped.
But later test/units/TEST-74-AUX-UTILS.machine-id-setup.sh tests for
failed units. capsule@foobar.service is listed as failed, causing the
second subtest to fail.
Add the same test in test/units/TEST-74-AUX-UTILS.capsule.sh to see
if the failed test really originates from there.
When systemd-cryptenroll --tpm2-device=auto is called on a system where
a tpm2-pcr-public-key.pem exists it automatically creates tokens with a
signed PCR policy. Unlocking such a token via --unlock-tpm2-device=auto
requires a tpm2-pcr-signature.json file, which is not present.
This creates a race with systemd-tpm2-setup.service at boot: if the
service completes before the test, the key exists and the subsequent
--unlock-tpm2-device=auto calls fail, which I believe is the cause of
the test flakiness.
This also seems to mesh with the fact that this only flakes on Debian
CI, since that's built with ukify which installs a public key.
Let's hopefully fix this by passing --tpm2-public-key= to all
--tpm2-device= enrollment calls that aren't explicitly intended to test
signed PCR policy behaviour.
When systemd-cryptenroll --tpm2-device=auto is called on a system where
a tpm2-pcr-public-key.pem exists it automatically creates tokens with a
signed PCR policy. Unlocking such a token via --unlock-tpm2-device=auto
requires a tpm2-pcr-signature.json file, which is not present.
This creates a race with systemd-tpm2-setup.service at boot: if the
service completes before the test, the key exists and the subsequent
--unlock-tpm2-device=auto calls fail, which I believe is the cause of
the test flakiness.
This also seems to mesh with the fact that this only flakes on Debian
CI, since that's built with ukify which installs a public key.
Let's hopefully fix this by passing --tpm2-public-key= to all
--tpm2-device= enrollment calls that aren't explicitly intended to test
signed PCR policy behaviour.
Verify that GPT images with an ISO9660 El Torito boot catalog are
dissected via the GPT partition table rather than being treated as a
single iso9660 filesystem.
Follow-up for e33eb053fb ("dissect-image: Drop
blkid_probe_filter_superblocks_usage() call from probe_blkid_filter()")
During daemon-reload (or daemon-reexec), when a unit becomes an alias to
another unit, deserialising the alias's stale serialised state can
corrupt the canonical unit's live runtime state.
Consider this scenario:
1. Before reload:
- a.service is running
- b.service was stopped earlier and is dead
- Both exist as independent units
2. User creates an alias to migrate from b -> a:
- `ln -s /run/systemd/system/a.service /etc/systemd/system/b.service`
3. daemon-reload triggers serialisation. State file contains both units:
- a.service -> state=running, cgroup=/system.slice/a.service, PID=1234,
...
- b.service -> state=dead, cgroup=(empty), no PIDs, ...
4. During deserialisation:
- Processes a.service: loads Unit A, deserialises -> state=RUNNING
- Processes b.service: manager_load_unit() detects symlink, returns Unit
A
- unit_deserialize_state(Unit A, ...) overwrites with b's dead state
5. The result is that:
- Unit A incorrectly shows state=dead despite PID 1234 still running
- If a.service has Upholds= dependents, catch-up logic sees a.service
should be running but is dead
- systemd starts a.service again -> PID 5678
- Two instances run: PID 1234 (left-over) and PID 5678 (new)
This bug is deterministic when serialisation orders a.service before
b.service.
The root cause is that manager_deserialize_one_unit() calls
manager_load_unit(name, &u) which resolves aliases via
unit_follow_merge(), returning the canonical Unit object. However, the
code doesn't distinguish between two cases when u->id differs from the
requested name from the state file. In the corruption case, we're
deserialising an alias entry and unit_deserialize_state() blindly
overwrites the canonical unit's fields with stale data from the old,
independent unit. The serialised b.service then overwrites Unit A's
correct live state.
This commit first scans the serialised unit names, then adds a check
after manager_load_unit():
if (!streq(u->id, name) && set_contains(serialized_units, u->id))
...
This detects when the loaded unit's canonical ID (u->id) differs from
the serialised name, indicating the name is now an alias for a different
unit and the canonical unit also has its own serialised state entry.
If the canonical unit does not have its own serialised state entry, we
keep the state entry. That handles cases where the old name is really
just a rename, and thus the old name is the only serialised state for
the unit. In that case there is no bug, because there is no separate
canonical state entry for the stale alias entry to overwrite.
Skipping is safe because:
1. The canonical unit's own state entry will be correctly deserialised
regardless of order. This fix only prevents other stale alias entries
from corrupting it.
2. unit_merge() has already transferred the necessary data. When
b.service became an alias during unit loading, unit_merge() already
migrated dependencies and references to the canonical unit.
3. After merging, the alias doesn't have its own runtime state. The
serialised data represents b.service when it was independent, which is
now obsolete once the canonical unit also has its own serialised entry.
4. All fields are stale. unit_deserialize_state() would overwrite state,
timestamps, cgroup paths, pids, etc. There's no scenario where we want
this data applied on top of the canonical unit's own serialised state.
This fix also correctly handles unit precedence. For example, imagine
this scenario:
1. `b.service` is a valid, running unit defined in `/run`.
2. The sysadmin creates `ln -s .../a.service /etc/.../b.service`.
3. On reload, the new symlink in `/etc` overrides the unit in `/run`.
The new perspective from the manager side is that `b.service` is now an
alias for `a.service`.
In this case, systemd correctly abandons the old b.service unit, because
that's the intended general semantics of unit file precedence. We also
do that in other cases, like when a unit file in /etc/systemd/system/
masks a vendor-supplied unit file in /lib/systemd/system/, or when an
admin uses systemctl mask to explicitly disable a unit.
In all these scenarios, the configuration with the highest precedence
(in /etc/) is treated as the new source of truth. The old unit's
definition is discarded, and its running processes are (correctly)
abandoned. In that respect we are not doing anything new here.
Some may ask why we shouldn't just ignore the symlink if we think this
case will come up. I think there are multiple very strong reasons not to
do so:
1. It violates unit precedence. The unit design is built on a strict
precedence list. When an admin puts any file in /etc, they are
intentionally overriding everything else. If manager_load_unit were to
"ignore" this file based on runtime state, it would break this
fundamental precedent.
2. It makes daemon-reload stateful. daemon-reload is supposed to be a
simple, stateless operation, basically to read the files on disk and
apply the new configuration. But doing this would make daemon-reload
stateful, because we'd have to read the files on disk, but
cross-reference the current runtime state, and... maybe ignore some
files. This is complex and unpredictable.
3. It also completely ignores the user intent. The admin clearly has
tried to replace the old service with an alias. Ignoring their
instruction is the opposite of what they want.
Fixes: https://github.com/systemd/systemd/issues/38817
Fixes: https://github.com/systemd/systemd/issues/37482
While we're at it, rename to test-loop-util.c so it matches its
source file.
We also drop the root check and solely check for CAP_SYS_ADMIN since
that's sufficient to run the tests.
This is a rebased version of #40936:
- the first few commits are #41003, which is greenlighted for merging
after the release
- then there's #40923
- and some changes on top
During daemon-reload (or daemon-reexec), when a unit becomes an alias to
another unit, deserialising the alias's stale serialised state can
corrupt the canonical unit's live runtime state.
Consider this scenario:
1. Before reload:
- a.service is running
- b.service was stopped earlier and is dead
- Both exist as independent units
2. User creates an alias to migrate from b -> a:
- `ln -s /run/systemd/system/a.service /etc/systemd/system/b.service`
3. daemon-reload triggers serialisation. State file contains both units:
- a.service -> state=running, cgroup=/system.slice/a.service,
PID=1234, ...
- b.service -> state=dead, cgroup=(empty), no PIDs, ...
4. During deserialisation:
- Processes a.service: loads Unit A, deserialises -> state=RUNNING
- Processes b.service: manager_load_unit() detects symlink, returns
Unit A
- unit_deserialize_state(Unit A, ...) overwrites with b's dead state
5. The result is that:
- Unit A incorrectly shows state=dead despite PID 1234 still running
- If a.service has Upholds= dependents, catch-up logic sees
a.service should be running but is dead
- systemd starts a.service again -> PID 5678
- Two instances run: PID 1234 (left-over) and PID 5678 (new)
This bug is deterministic when serialisation orders a.service before
b.service.
The root cause is that manager_deserialize_one_unit() calls
manager_load_unit(name, &u) which resolves aliases via
unit_follow_merge(), returning the canonical Unit object. However, the
code doesn't distinguish between two cases when u->id differs from the
requested name from the state file. In the corruption case, we're
deserialising an alias entry and unit_deserialize_state() blindly
overwrites the canonical unit's fields with stale data from the old,
independent unit. The serialised b.service then overwrites Unit A's
correct live state.
This commit first scans the serialised unit names, then adds a check
after manager_load_unit():
if (!streq(u->id, name) && set_contains(serialized_units, u->id))
...
This detects when the loaded unit's canonical ID (u->id) differs from
the serialised name, indicating the name is now an alias for a different
unit and the canonical unit also has its own serialised state entry.
If the canonical unit does not have its own serialised state entry, we
keep the state entry. That handles cases where the old name is really
just a rename, and thus the old name is the only serialised state for
the unit. In that case there is no bug, because there is no separate
canonical state entry for the stale alias entry to overwrite.
Skipping is safe because:
1. The canonical unit's own state entry will be correctly deserialised
regardless of order. This fix only prevents other stale alias entries
from corrupting it.
2. unit_merge() has already transferred the necessary data. When
b.service became an alias during unit loading, unit_merge() already
migrated dependencies and references to the canonical unit.
3. After merging, the alias doesn't have its own runtime state. The
serialised data represents b.service when it was independent, which
is now obsolete once the canonical unit also has its own serialised
entry.
4. All fields are stale. unit_deserialize_state() would overwrite state,
timestamps, cgroup paths, pids, etc. There's no scenario where we
want this data applied on top of the canonical unit's own serialised
state.
This fix also correctly handles unit precedence. For example, imagine
this scenario:
1. `b.service` is a valid, running unit defined in `/run`.
2. The sysadmin creates `ln -s .../a.service /etc/.../b.service`.
3. On reload, the new symlink in `/etc` overrides the unit in `/run`.
The new perspective from the manager side is that `b.service` is now an
alias for `a.service`.
In this case, systemd correctly abandons the old b.service unit, because
that's the intended general semantics of unit file precedence. We also
do that in other cases, like when a unit file in /etc/systemd/system/
masks a vendor-supplied unit file in /lib/systemd/system/, or when an
admin uses systemctl mask to explicitly disable a unit.
In all these scenarios, the configuration with the highest precedence
(in /etc/) is treated as the new source of truth. The old unit's
definition is discarded, and its running processes are (correctly)
abandoned. In that respect we are not doing anything new here.
Some may ask why we shouldn't just ignore the symlink if we think this
case will come up. I think there are multiple very strong reasons not to
do so:
1. It violates unit precedence. The unit design is built on a strict
precedence list. When an admin puts any file in /etc, they are
intentionally overriding everything else. If manager_load_unit were
to "ignore" this file based on runtime state, it would break this
fundamental precedent.
2. It makes daemon-reload stateful. daemon-reload is supposed to be a
simple, stateless operation, basically to read the files on disk and
apply the new configuration. But doing this would make daemon-reload
stateful, because we'd have to read the files on disk, but
cross-reference the current runtime state, and... maybe ignore some
files. This is complex and unpredictable.
3. It also completely ignores the user intent. The admin clearly has
tried to replace the old service with an alias. Ignoring their
instruction is the opposite of what they want.
Fixes: https://github.com/systemd/systemd/issues/38817
Fixes: https://github.com/systemd/systemd/issues/37482
This commit tweaks the TEST-74-AUX-UTILS.varlinkctl.sh code
to use `systemd-notify --fork $UPGRADE_SERVER` instead of
the (ugly) timeout.
This also fixes a stale comment in around
`Test --upgrade with stdin redirected from a regular file`.
Thanks to Daan for suggesting this!
The collection of facts is entirely unprivileged and has very little to
do with PID1. PID1 is privileged and single-threaded and a point of
contention, so we shouldn't put things in PID1 that don't need to be
there. A separate service can be enabled/disabled/started/stopped at
will, is easy to sandbox, etc. If it turns out to be necessary to
collect some facts through PID1 in the future, we can always add a
smaller facts endpoint to PID1 again.
Having support for `--exec` when using `--upgrade` is nice so this
commit adds it. It does it by extracting a shared helper called
`exec_with_listen_fds()` and then use that in the `verb_call()`
and `varlink_call_and_upgrade()` calls.
The varlink spec supports protocol upgrades and they are very
useful to e.g. transfer binary data directly via varlink. So
far varlinkctl/sd-varlink was not supporting this. This commit
adds support for it in varlinkctl by using the new code in
sd-varlink and the generalized socket-forward code.
`updatectl enable --now` systematically fails because the update phase find
resources that are not pending. So instead we should ignore them.
Fixes#41254
AllowDiscards= will set allow-discards in the persistent flags which will make
activating the device automatically activate with that option. This is
useful for devices discovered through gpt-auto-generator without
needing to use some kernel command line to set the option.
Ubuntu is doing shenanigans with their coreutils so they are now
symlinks instead of binaries, so the grep fails. Check bash instead
to fix test failure on 26.04.
This has been requested previously for PCR 7 (#40567), but let's do that
for all firmware owned PCRs, since some firmwares forget to measure
their own separator. Let's hence measure our own guranteed one.
Fixes: #40567
Before this commit, the > 0 state of arg_newline tristate is
simply ignored.
Yes, this is a minor compat break, but I'd argue the previous
behavior was not useful as "yes" is treated the same as "auto".
An issue also reported that it was quite surprising.
Fixes#41348
Commit 0e1ede4b4b introduced a bug where
we'd always fetch the "original" (empty) list of groups when processing
a comma-separated list of groups from the --member-of= option, so only
the last group from the list would get applied. This bug was then later
(in 316e9887f2) refactored into a separate
function.
Follow-up for 0e1ede4b4b.
Resolves: #41286
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
Christian Brauner
Daan De Meyer
Nick Rosbrook
Lennart Poettering
Yaping Li
Michael Vogt
Frantisek Sumsal
Christian Brauner
Chris Down
Valentin David
Mike Yuan