This commit adds a simple version of io.systemd.Unit.StartTransient
for varlink. It is similar to the dbus version, but there is a key
difference:
1. Instead of building the unit from key/value properties it
takes a json object in the "service" parameter. It is also
only implementing a minimal set of what can be done with a
service (for now)
2. No aux units (for now)
3. When called with --more the varlink socket can notify about
unit job and state changes controlled via a bool on the
varlink call inputs: notify{Job,Unit}Changes
We use the new io.systemd.Job interface when outputing the
io.systemd.Unit.StartTransient result as it makes the output
nice and mirrors the input.
Note that the property names follow the D-Bus naming to make a
future "systemctl show" transition from D-Bus -> varlink easier.
Because UnitContext is now also used for the inputs we need
to make a bunch of fields `SD_VARLINK_NULLABLE` so that the
input is even accepted. This does not affect the output, it
is still fully populated, just the schema. The ID of UnitContext
is still required.
Thanks to ikruglov and Lennart for their excellent feedback on
this.
Extract the transient unit setup logic from transient_unit_from_message()
in dbus-manager.c into a shared helper.
This prepares for reuse by the varlink StartTransient implementation.
This commit creates a new varlink-io.systemd.Job.c file and puts
the job related varlink types into this file. Those will be
used by the upcoming io.systemd.Unit.StartTransient.
Note that the property names follow the D-Bus naming to make a
future "systemctl show" transition from D-Bus -> varlink easier.
Thanks to @ikruglov for suggesting this preparation commit and
to Lennart for suggesting the D-Bus compatibility considerations.
We have code like `size_t max_size = sizeof(*private)` in three
places. This is evaluated at compile time so its safe to use. However
the new pointer-deref checker in coccinelle is not smart enough to know
this and will flag those as errors. To avoid these false positives
we have some options:
1. Reorder so that we do:
```C
size_t max_size = 0;
assert(private);
max_size = sizeof(*private);
```
2. Use something like `size_t max_size = sizeof(*ASSERT_PTR(private));`
3. Place the assert before the declaration
4. Workaround coccinelle via SIZEOF(*private) that we can then hide
via parsing_hacks.h
5. Fix coccinelle (OCaml, hard)
6. ... somehting I missed?
None of these is very appealing. I went for (4) but happy about
suggestions.
We have a test failure where the testsuite is calling
sd_varlink_idl_parse() with *ret being NULL. This is now an
assert error. So we could either fix the test or fix the code
Given that it seems genuinely useful to run sd_varlink_idl_parse()
without *ret to e.g. just check if the idl is valid I opted to
fix the code.
The updated parsing_hacks.h file uncovered a bunch of extra
things that the check-pointer-deref coccinelle script flags.
This commit fixes them to make the tree check-pointer-deref clean.
This commit runs the check-coccinelle checker scripts with the
parsing_hacks.h. Because this was missing before there were some
issues that did not get flagged.
While at it it also adds some missing cleanup attributes and
iterators to get better results. Its a bit sad that there is no
(easy/obvious) way to detect when new things are needed for
parsing_hacks.h
When there is a potential protocol upgrade we need to be careful that
we do not read beyond our json message as the custom protocol may be
anything. This was archived via a byte-by-byte read. This is of course
very inefficient. So this commit moves to use MSG_PEEK to find the
boundary of the json message instead. This makes the performance hit
a lot smaller.
Thanks to Lennart for suggesting this.
When the server side of a varlink connection supports connection
upgrades we need to go into single byte-read mode to avoid the
risk of a client that sends the json to protocol upgrade and then
immediately the custom protocol payload. This commit implements
this.
The next step is using MSG_PEEK to avoid the single-byte overhead.
This commit tweaks the API of sd_varlink_call_and_upgrade and
sd_varlink_reply_and_upgrade to return two independent fds even
if the internal {input,output}_fd are the same (e.g. a socket).
This makes the external API easier as there is no longer the risk
of double close. The sd_varlink_call_and_upgrade() is not in a
released version of systemd yet so I presume it is okay to update
it still.
This also allowed some simplifications in varlinkctl.c now that
the handling is easier.
With the new protocol upgrade support in varlinkctl client we can
now do the equivalent for the server side. This commit adds a new
`serve` verb that will serve any command that speaks stdin/stdout
via varlink and its protocol upgrade feature. This is the
"inetd for varlink".
This is useful for various reasons:
1. Allows to e.g. provide a heavily sandboxed io.myorg.xz.Decompress
varlink endpoint, c.f. xz CVE-2024-3094)
2. Allow sftp over varlink which is quite useful with the
varlink-http-bridge (that has more flexible auth mechanism than
plain sftp).
3. Makes testing the varlinkctl client protocol upgrade simpler.
4. Because we can.
Extract the fd-handling logic from sd_varlink_call_and_upgrade() into a
shared static helper so that it can be reused by the upcoming server-side
sd_varlink_reply_and_upgrade().
The io.systemd.Manager.{PowerOff,SoftReboot,Halt,Kexec} manager
varlink and bus methods log the peer ID when calling shutdown.
The logind code is missing this, so this commit adds a similar
logging now. The code is quite similar to the one in existing in
src/core/manager.c but its hard to share code so this adds a bit
of duplication.
To properly support a shutdown interface with varlink we need
the functionality of verify_shutdown_creds(). This is currently
dbus only. There are some options:
1. refactor and abstract so that verify_shutdown_creds() is agnostic
2. provide a equivalent function with varlink
3. allow to call it with either a dbus or a varlink message.
The most elegant of course is (1) but it makes reviewing harder
and has a higher regression risk. It will also be more code.
Doing (2) has the risk of drift, i.e. we will need to keep
the two functions in sync and not forget about it ever.
So this commit opts for (3): allowing either dbus or varlink.
This is quite ugly, however the big advantage is that its very
simple to review as the dbus/varlink branches mirror each other.
And there is no risk of drift the dbus/varlink options are
close to each other. It unlikely we get a third bus, so it will
most likely stay this way. It is still ugly though so I can
understand if this is undesired and I can look into (1) if its
too ugly.
With this function avaialble logind-varlink.c is now updated to
use it.
Move verify_shutdown_creds() and its helper have_multiple_sessions()
from logind-dbus.c to logind-shutdown.c so that they can be reused
by the varlink transport. No functional changes.
Also prefix both with `manager_` now that they are public.
The shutdown interface is currently only exposed via dbus. This commit
adds a comparable varlink implementation. It is inspired by the existing
dbus methods and implements PowerOff, Reboot, Halt, Kexec, SoftReboot.
It is (intentional) simpler than the dbus for now, i.e. strictly root
only. To match dbus we will need to the functionality
of verify_shutdown_creds() which is dbus-ish right now and would
need some refactor.
For the same reason it does not do the Can* methods - we will need
the verify_shutdown_creds() equivalent first.
This function operates on generic Manager state and will be needed
by the varlink shutdown interface too. Move it out of logind-dbus.c
into a new logind-shutdown.c, alongside the SHUTDOWN_SCHEDULE_FILE
define and use `manager_reset_scheduled_shutdown() as the new
name.
This commit tweaks the TEST-74-AUX-UTILS.varlinkctl.sh code
to use `systemd-notify --fork $UPGRADE_SERVER` instead of
the (ugly) timeout.
This also fixes a stale comment in around
`Test --upgrade with stdin redirected from a regular file`.
Thanks to Daan for suggesting this!
Instead of exiting in exec_with_listen_fds() just return an error
and do the actual _exit() in the caller. Much nicer this way.
Thanks for Lennart for suggesting this.
This commit fixes an issue with the fd handling in
sd_varlink_call_and_upgrade() when one direction of the output
FDs is unset.
Thanks to Lennart for spotting this and suggesting the fix.
The shared/ dns_packet_read_uint*() functions used to be a bit
inconsistent when it comes to the handling of NULL in *ret. The
helpers dns_packet_read_{uint16,name} accepted NULL but the
dns_packet_read_uint{8,32} did not.
This commit makes them all symmetric in accepting NULL.
Some functions are in the hot path of a sort. While it is more
correct to also use assert() here for the passed pointers it is
also a runtime cost that should never be needed when e.g. qsort()
is used as it will always pass non-NULL pointers.
So for these critical functions use POINTER_MAY_BE_NULL() which
is compiled out and add a comment to them. Not all _compare()
functions are changed, it was a bit of a judgement call because
some (like ether_addr_compare) are not called with that much data.
But happy to get advise here and tweak accordingly.
Add the needed assert changes to make the code clean
for the new check-pointer-deref script.
Note that for `_cmp()` functions its a bit of a question
if we want `assert()` or `POINTER_MAY_BE_NULL()` here. The
former is more correct but it adds a (small) runtime hit. If
we use POINTER_MAY_BE_NULL() it is compiled out but strictly
speaking it is wrong (so at least we would need a comment).
There are some unnecessary() if() checks for variables
that are already guarded by assert()s. Drop those.
Thanks to keszybz for clarifying the direction of this commit.
The previous code used strv_join() when it generated the log
message for `varlinkctl --exec`. However this can lead to
inaccurate logging so use `quote_command_line()` instead.
Thanks to Lennart for suggesting this.
Having support for `--exec` when using `--upgrade` is nice so this
commit adds it. It does it by extracting a shared helper called
`exec_with_listen_fds()` and then use that in the `verb_call()`
and `varlink_call_and_upgrade()` calls.
The varlink spec supports protocol upgrades and they are very
useful to e.g. transfer binary data directly via varlink. So
far varlinkctl/sd-varlink was not supporting this. This commit
adds support for it in varlinkctl by using the new code in
sd-varlink and the generalized socket-forward code.
The SimplexForwader was using the naming of the SocketForwarder
for bi-directional sockets. This was to keep the diff small and
to make it easier to follow what changed and what was reused.
However the name "client/server" for the SimplexForwader does
no longer make much sense. The SimplexForwader is no longer
about client/server but really just read/write. So this commit
adjusts the naming.
Now that the socket forward code is extracted we can
extend it to not just support bidirectional sockets
but also input/output fd-pairs. This will be needed
for e.g. the varlinkctl protocol upgrade support where
one side of the connection is a fd-pair (stdin/stdout).
This is done by creating two half-duplex forwarders
that operate independantly. This also allows to simplify
some state tracking, e.g. because each fd serves only one
direction we don't need to dynamically create the event mask
with EPOLLIN etc, its enough to set it once. It also handles
non-pollable FDs transparently.
Thanks to Lennart for his excellent suggestions here.
The varlink spec supports protocol upgrades and they are very
useful to e.g. transfer binary data directly via varlink. So
far sd-varlink was not supporting this.
This commit adds a new public sd_varlink_call_and_upgrade()
that sends a method call, waits for the reply, then steals
the connection fds for raw I/O. It returns separate input_fd
and output_fd to support both bidirectional sockets and pipe
pairs.
A helper is extracted and shared between sd_varlink_call_full()
and sd_varlink_call_and_upgrade(). A new `protocol_upgrade`
bool in `struct sd_varlink` ensures that on a protocol upgrade
request we only exactly read the varlink protocol bytes and
leave anything beyond that to the caller that speaks the upgraded
protocol.
Note that this is the client side of the library implementation
only for now. The server side needs work but this is already
useful as it allows to talk to varlink servers that speak protocol
upgrades (like the rust implemenations of varlink).
Trivial ordering/modernizing change that got highlighted by
claude and refined by keszybz to move to the modern systemd
style.
Thanks to keszybz for suggesting this.
For some of the directories it makes more sense to keep them
excluded from the coccinelle check. Specifically:
- libc: compatibility, no asserts or systemd headers yet
- test: uses NUL internally to test crashes etc
Our coccinelle/check-pointer-deref.cocci checker has a special
case for `assert(pidref_is_set(param))`. It turns out we can
generalize this and catch the following:
- iovec_is_set
- sd_dhcp_duid_is_set
- sd_dhcp_client_id_is_set
The fix in 8f1751a111 made me wonder if we could automatically detect
when pointers are accessed but when this might not be safe. Systemd
is already using a lot of `assert(dst)` and this change now forces
us to use them.
So this commit (ab)uses coccinelle to flag any pointer parameter
dereference not preceded by assert(param), ASSERT_PTR(param), or an
explicit NULL check. It adds integration into meson as a new "coccinelle"
test suite (just like clang-tidy) and is run in CI. The check is not
perfect but seems a reasonable heuristic.
For this RFC commit it is scoped to a subset, it excludes 25 dirs right
now and includes around 100. About 300 warnings left. Busywork that I am
happy to do if there is agreement that it is worth it.
With this in place we would have caught the bug from 8f1751a111 in CI:
```
FAIL: check-pointer-deref.cocci found issues in systemd/src/boot:
diff -u -p systemd/src/boot/measure.c /tmp/nothing/measure.c
--- systemd/src/boot/measure.c
+++ /tmp/nothing/measure.c
@@ -312,7 +312,6 @@ EFI_STATUS tpm_log_tagged_event(
if (err != EFI_SUCCESS)
return err;
- *ret_measured = true;
return EFI_SUCCESS;
}
```
This also adds a new POINTER_MAY_BE_NULL() for the cases when the
called function will do the NULL check (like `iovec_is_set()`).
The comment about `SD_BUS_VTABLE_CAPABILITY(CAP_SYS_BOOT)` is
incorrect. To quote @YHNdnzj:
```
nah, the capability-based permission model is a legacy from kdbus. We cannot do it race-freely without it. Please simply drop the comment.
```
Thanks to @YHNdnzj for suggesting this.
The manager_do_set_objective() was doing a bunch of work to
check the `root` path that can already be done via
`json_dispatch_path` so instead of duplicating use the helper.
Thanks to @YHNdnzj for suggesting this.
Drop the _manager suffix for vl_method_{poweroff,{soft,}reboot,
halt,kexec}_manager.
They're unambiguous enough and the power mgmt operations are
not manager-wide but system-wide.
Thanks to @YHNdnzj for suggesting this.
External tools that use the systemd varlink ecosystem require
to know if a specific varlink method supports/requires the
"more" flag from the IDL. This is tracked upstream in
https://github.com/varlink/varlink.github.io/issues/26
As an intermediate step systemd adds the (very nice) comments
```
# [Requires 'more' flag]
or
# [Supports 'more' flag]
```
to the various methods.
This commit extends the comment around the code that adds the
comment to clarify that this should be considered API and that
the comment should not be changed as external tools (like e.g.
the varlink-http-bridge) rely on it.
This commit allows unset pidref when calling manager_log_caller().
With that we can log manager calls even if we cannot resolve the
caller. Currently when we cannot resolve the caller we are just
not logging anything. With this commit we at least log the call
(even though we don't know what caller it was).
Thanks to keszybz for the suggestion.
Extract a common helper varlink_log_caller() and use in the varlink
code when logging the caller of a method. It also logs the method
now that was tried (but failed) to be logged with log_notice just
like manager_log_caller() would do.
I was looking into modifying `manager_log_caller` instead and
accept a NULL pidref but could not log more than the method without
pidref and would make the manager_log_caller slightly less nice.
Thanks to keszybz for suggesting this.
This adds the low-level io.systemd.Manager shutdown support. This
is (much) simpler than the logind one. It mimics dbus but uses
a shared helper for the simple cases.
Note that this is more restrictive than the dbus version. The
dbus version uses SD_BUS_VTABLE_CAPABILITY(CAP_SYS_BOOT) but
the varlink version uses varlink_check_privileged_peer(link).
This is mostly because I'm not sure how to do the equivalent
in a race-free way.
Thanks to Daan for suggesting this.
This commit extracts the socket forwarding code from the existing
socket-proxyd into a new shared helper that will be used by the
varlinkctl protocol upgrade support code and is used as is in
the socket-proxyd.c.
It tries to keep the changes as small as possible, its mostly
renaming like:
* connection_create_pipes -> socket_forward_create_pipes
* connection_shovel -> socket_forward_shovel
* connection_enable_event_sources -> socket_forward_enable_event_sources
* traffic_cb -> socket_forward_traffic_cb
and a new socket_forward_new() that creates/starts the forwarding.
All log_error_errno() got downgraded to log_debug_errno().
Currently the varlink interface for hostname is only available
after sysinit. This means it is not available until systemd-firstboot
is finished. But there is information like the boot-id in there that
is useful to get early.
My use-case is to query the system early via the varlink-http-bridge
and currently I can't get data from io.systemd.Hostname until
systemd-firstboot is completed which is a bit limiting.
So to fix it this commit sets DefaultDependencies=no on both the socket
and service units.
It also changes hostnamed.c to use
bus_open_system_watch_bind_with_description() which means we will
reconnect once dbus is available. This mimics what resolved-bus.c
is doing (and which was originally introduced in d7afd945b).
Thanks to Lennart for pointing this out.
With the planned extraction of the socket-forward code its useful
to have a basic way to validate the functionality. So add a basic
test that ensures at least base functionality is intact.
In a project I'm working on I recently observed a boot failure
with the most recent version of systemd. It seems it is triggered
by bb19b61049 which fixed a bug that now leads to the function
being excuted differently. The code is missing a check if
`*ret_version` is actually valid in the `ret_measured = true`
case.
We already have some varlink support for the journal to perform
some actions like `Rotate`. It would be nice to be able to query
the journal via varlink too so this commit adds a new varlinkctl
based journal service that exposes a single GetEntries() call
to retrieve journal entries. Basic filtering is supported and
we can expand the API as needed.
This is a separate `io.systemd.JournalControl` [1] service from the
existing `io.systemd.Journald` to decouple read and write (thanks
to Lennart for suggesting this).
This also extracts some shared helper so that we do not duplicate
code when generating the json or when adding the filters.
[1] The name mirrors the bootctl->io.systemd.BootControl naming.
The CleanPool requires --more to be set and checks that in
`vl_method_clean_pool`. By switching to SD_VARLINK_REQUIRES_MORE
this will automatically be handled and is more clear to
the varlink users.
Based on the comment from Lennart in
https://github.com/systemd/systemd/pull/40650#discussion_r2832378002
and the work done by Mike in 09388a6b9e (thanks!).
When sd_varlink_connect_url() gets an unknown URL we now
check if there is a `$LIBEXECDIR/varlink-bridges/$scheme`
binary and execute it (with the url as the first arguments).
This makes varlink more flexible as it provides a way to
dynamically add "bridges" in LIBEXECDIR/varlink-bridges/. This is
conceptually similar to the libvarlink `varlink --bridge` command
and allows to e.g. call varlink over http{,s} via e.g. the new
varlink-http-bridge.
With a running varlink-http-bridge [0] one can do:
```console
$ varlinkctl call http://localhost:8080/ws/sockets/io.systemd.Hostname \
io.systemd.Hostname.Describe {}
{
"Hostname" : "top",
...
```
Closes: https://github.com/systemd/systemd/issues/40640
[0] https://github.com/mvo5/varlink-http-bridge/pull/1
This patch adds a `--ephemeral` option to vmspawn that will turn
on snapshot mode on the qemu disk passed via `--image` or the
directory passed via `--directory`. For disk images it uses the
native mechanism that qemu provides. For directories it (re)uses
the snapshot mechanism that nspawn is using, i.e. it will create
a btrfs snapshot if possible and if not falls back to a traditional
directory copy.
This commit extracts the creation of the snapshot for
nspawn into a shared helper. Its not a lot of code but its
subtle so having a single place seems beneficial.
While extracting the snapshot code got tweaked/simplified based
on the feedback from Yu (many thanks!).
With that we can simplify the snapshot delete and error handling
as well. The shared helper is then going to be used by vmspawn
when it also gets the --ephemeral option.
Michael Vogt