When PCR is extended during a TPM2 operation, the internal counter of
the TPM2 will detect a new extension. systemd already retry this
operation several times, but sometimes the report will be an error, and
will refer to the wrong PCR.
This situation happends when IMA is enabled (when secure boot is active,
for example). PCR 10 will receive an extension for each new kernel
module that is loaded, logging in the journal errors that are recovered
later after a subsequent retry.
We replace the implicit single callback and wait-future indirection with
the usual slot design so independent fibers and future groups can track
the same future. We also opt to always have callbacks dispatch asynchronously,
including when they are registered after the future has already resolved.
We now require futures to be resolved before their final unref rather than
cancelling them implicitly. The idea is to catch bugs rather than leak futures.
A future should always be cancelled and awaited before it is cleaned up,
otherwise resources might leak if the stack is not unwound properly. We make
exceptions for futures that haven't started executing yet as there's no risk
of leaking resources there since there's no stack to unwind.
One issue with awaiting futures is that sd_fiber_await() returned both
errors and the future's result via the function's return value, meaning it
was impossible to distinguish future failures from interrupted waits (or
waits that failed for whatever other reason). This is generally not an issue
except in the cleanup functions, where we have to retry on interrupted waits
(the best example is a parent fiber waiting for a future being cancelled
itself, which should cancel the future it's waiting for and retry the wait).
So we introduce fiber_wait() which distinguishes between both values via
return value and output argument so the two can be distinguished from each
other.
Cleanup always consumes the caller's reference, including on invalid self or
cross-loop waits. The existing assertion still forbids dropping the last
reference to an unresolved future. The installed header documents this rule.
We introduce a new RUNNING fiber state to distinguish executing fibers from
non-executing fibers. We replace CANCELLED with queued resume results and
preserve interruptions across ordinary wakeups.
We add future groups with wait-all, wait-any and ignore-errors policies to
wait for multiple futures at once and migrate various existing call sites over
to them. Group size is a direct size_t query, treating NULL as an empty group.
Document all four policy combinations, including that IGNORE_ERRORS suppresses
fail-fast and parent cancellation without discarding the group's error result.
We also add defer futures and migrate bus, varlink, QMP and fiber I/O callers to
use them. Bus and QMP futures report -ENOPKG when no event loop is attached,
rather than passing NULL to sd_future_new() and returning -EINVAL.
Parent-await suppression is derived from the active wait's callback slot, so
groups use sd_fiber_await() without a separate await API or additional state.
Waits, group children and captured parents must use the same event loop.
Groups created on another loop have no parent, and a group releases its
parent reference when that fiber finishes.
Resolve SD_EVENT_DEFAULT in sd_future_new() so futures retain the actual
loop. Match group parents against that resolved loop as well; an unavailable
default returns -ENOPKG without creating an event loop implicitly.
Keep the public await entry point in fiber.c, sharing the slot-based wait with
cancellation cleanup through future-internal.h. Keep event/future integration
in event-future.c/h: priority inheritance uses the public fiber accessor, and
sd_future_new_defer() lives alongside the other event-backed constructors.
This avoids sd-event depending on sd-future private headers and removes the
redundant private defer wrapper.
Assert the non-failing slot-disabling and active-loop invariants instead of
returning early without completing callback dispatch.
Regression tests cover group error propagation and NULL-safe size queries,
interrupted waits, partial add-many rollback, groups outliving their parent,
and cancellation or timeout during nested event-loop waits, including
event-loop and descriptor cleanup. They also cover cross-loop parent isolation,
reference release on invalid cleanup calls, mixed WAIT_ANY outcomes, failed
child cancellation and policy changes on resolved groups. Additional tests
verify failed group-priority updates, slot priority during dispatch and exit,
and cancellation, timeout and spurious wakeups during bus and QMP calls. Late
replies must leave subsequent calls unaffected. Default-event tests cover
missing-loop errors, reference lifetime and parent capture by constructors.
Unattached bus and QMP clients are covered by missing-event-loop regressions.
Constructor tests verify inherited and default priorities for I/O, defer and
timer futures, plus defer argument validation and missing-default errors.
Signed-off-by: Daan De Meyer <daan@amutable.com>
When IPv4 ACD is enabled for a DHCPv4 address, the address remains in
the request queue until ACD succeeds. Hence, when ACD detects a
conflict, the address may not be present in link->addresses yet.
on_acd() currently looks up the address with address_get(), which only
searches link->addresses. This makes the callback return early and
prevents dhcp4_address_on_conflict() from sending DHCPDECLINE.
Use address_get_harder(), which also looks up pending address requests.
Fixes#39460
When a DHCP lease is lost or a client is stopped, there may exist
requests not processed yet. We need to cancel them, otherwise an
address or route requested by the lost lease may be configured later.
Currently, in_addr_prefix_range() is the only user of in_addr_prefix_nth(),
and it is too generic and complicated.
Previously, in_addr_prefix_range() calls in_addr_prefix_nth() with 0 and 1.
The function in_addr_prefix_nth(0) is equivalent to in_addr_mask(), and
in_addr_prefix_nth(1) is equivalent to just increment the masked value
with 1.
So, let's reimplement the code without using in_addr_prefix_nth().
Three independent fixes to code on `journalctl`'s per-entry read and
output path. Each of them removes work that was never needed, rather
than trading away anything. There is no behaviour change: the output is
byte for byte identical for every format except `-o json`, where only
the order of the keys within each object can shift, which is already
nondeterministic between two runs of the unpatched binary (see the
correctness section below). Nothing about the formats, tunables or
public API changes.
Measured on a 730 MB journal spread over 100 files (roughly 2M entries):
| command | before | after | |
|---|---:|---:|---:|
| `journalctl -o cat` | 1.93s | 0.77s | **2.51x** |
| `journalctl --grep=<no match>` | 1.97s | 0.82s | **2.40x** |
| `journalctl -o export` | 4.06s | 1.73s | **2.35x** |
| `journalctl -n 100000 -o short` | 0.86s | 0.43s | **2.00x** |
| `journalctl -o json -n 200000` | 5.85s | 3.50s | 1.67x |
| `journalctl -o short` (the default) | 3.32s | 1.96s | **1.69x** |
| `journalctl -o verbose` | 5.99s | 3.61s | 1.66x |
### 1. `utf8_is_printable_newline()` decoded every character twice
`utf8_encoded_valid_unichar()` decodes the character internally in order
to validate it, and then throws the codepoint away. The caller
immediately called `utf8_encoded_to_unichar()` again to get the very
same codepoint back. So for every byte of every field printed, the UTF-8
decoder ran twice, only to answer whether the byte was a control
character.
This is fixed where it originates rather than at the call site:
`utf8_encoded_valid_unichar_full()` optionally returns the character it
had to decode anyway, so a caller that wants both the validity and the
value decodes once. On top of that, eight bytes are tested at once for
plain printable ASCII, which is what almost all journal content is. A
word that does not qualify falls through to the per character path
unchanged, so tabs, newlines and multibyte sequences behave exactly as
before.
On its own, against unmodified main: `-o export` 4.02s to 2.89s.
### 2. sd-journal read the clock once per journal file, per entry
`sd_journal_next()` sweeps every open journal file to find the next
entry, and `next_beyond_location()` rate limits its tail timestamp
refresh per file. Since `ratelimit_below()` reads the clock itself, a
journal spread over a hundred files meant a hundred `clock_gettime()`
calls for every single entry iterated, nearly all of them only to
conclude that the one second interval had not elapsed yet.
This was the single largest cost in `journalctl`. In a profile of `-o
export` it was over a fifth of the entire run, spent deciding not to do
the thing it was guarding.
Add `ratelimit_below_at()`, which takes the timestamp from the caller,
and read the clock once per sweep instead. `ratelimit_below()` keeps its
behaviour and becomes a wrapper.
The timestamp is now sampled once per sweep rather than once per file.
The interval being rate limited is one second and a sweep takes
microseconds, so this makes no practical difference.
On its own: `-o export` 4.02s to 2.93s, and `-o short` 3.30s to 2.14s.
This is the largest single contributor for the default output mode.
### 3. The JSON output built a fresh, unsized hashmap for every entry
`journal_entry_to_json()` creates an empty hashmap per entry and fills
it with that entry's fields, of which there are typically around twenty.
Because it starts empty it has to grow several times per entry, and
every growth rehashes everything inserted so far. `-o json` spent about
a quarter of its time in siphash, a third of that under
`resize_buckets()`.
Reserving room up front is a one line change.
On its own: `-o json -n 200000` 5.71s to 4.64s.
This adds 'add' boolean parameter to sd_nfnl_nft_message_new_table() and
friends, so that we can remove nftables table and so on. In
test-firewall-util, before and after running test cases, remove the
garbage nftables test table.
If an address with IPv4ACD is requested, then sd_ipv4acd instance for
the address is configured. If the request is cancelled, we should
also disable the IPv4ACD instance.
Note, if the address is already configured, the IPv4ACD instance for the
address will be removed when the address is removed. See address_drop().
Otherwise, if the timer event source is enabled, a new query may be
invoked by the already removed service.
This also renames dns_remove_service() to dnssd_discovered_service_detach(),
and drops unused DnssdDiscoveredService.query.
When a DnsServiceBrowser object is detached from the manager,
all queries, event sources, and then the reference to Manager should be
dropped. Otherwise, spurious query may be triggered for already
disconnected varlink.
make_console_stdio() forces KD_TEXT while reopening /dev/console. When
systemd-shutdown enters the exitrd, this can expose the text console over
a retained graphical shutdown splash.
Let callers choose whether to switch to text mode, and preserve the mode
for the exitrd handoff. Keep the existing text-mode behavior for manager
reexecution and the crash shell. Restore text mode if executing /shutdown
fails and before reporting a fatal error and freezing.
Add an unsafe, nonparallel virtual-console regression test requiring
explicit opt-in and an already-owned inactive VT in a disposable VM.
Check graphics preservation and text restoration through a private
/dev/console bind mount, retain stderr diagnostics, and restore the saved
mode in the parent even if the child fails.
Split out of https://github.com/systemd/systemd/pull/42983
## Problem
`dns_scope_announce()` put the entire zone into a single reply packet —
for DNS-SD that is every PTR/SRV/TXT record of every published service.
With a few hundred services that grows far past the interface MTU, and
past what RFC 6762 section 17 allows: a Multicast DNS packet larger than
the MTU is sent using fragments, and such a packet "MUST NOT contain
more than one resource record".
## Fix
Pack an announcement into as many MTU-sized packets as needed. A record
that does not fit the MTU on its own is emitted alone in an oversized
packet relying on IP fragmentation — the case section 17 explicitly
allows — bounded by its hard 9000-byte ceiling including headers, which
also caps jumbo-frame links.
Without a usable interface MTU (no link on the scope, or `IFLA_MTU` not
read yet) packets are cut to the largest datagram a host of the family
must accept unfragmented — RFC 791's 576 bytes for IPv4, RFC 8200
section 5's 1280-byte minimum link MTU for IPv6 — rather than to that
ceiling, so an announcement is never fragmented merely because the MTU
was not known yet, and the lone-oversized-record path stays reachable.
A record that cannot be encoded at any size (an rdata past 64KiB, a
character string past 255 bytes) is skipped like one too large for the
ceiling rather than failing the whole announcement:
`dns_packet_append_rr()` rolls the packet back on every failure, so its
neighbours are unaffected either way.
The packing lives in `resolved-mdns.c` as
`mdns_announcement_packetize()`, taking an answer and the two size
bounds and returning finalized packets, so it is testable without a
scope, a link or a socket. Emission is best effort per packet — a failed
send no longer withholds the rest — and each pass logs the packet and
record counts it actually put on the wire.
1. test-fstab-generator: fix inverted /mnt symlink guard
The test-18-options.fstab.input scenario was guarded by a check
that was always true on a normal system, so it was silently
skipped and its fixtures had drifted out of sync with actual
generator output. This commit fixes the guard and repairs the
stale fixtures it exposed.
2. fstab-generator: support for x-systemd.onfailure
Add support for the x-systemd.onfailure= fstab option in
systemd-fstab-generator. This allows configuring OnFailure=
dependencies for generated mount units directly through /etc/fstab.
For example:
/dev/sda1 /data ext4 defaults,x-systemd.onfailure=emergency.target 0 2
results in the generated mount unit containing:
[Unit]
OnFailure=emergency.target
This makes it possible to attach recovery targets or services to
mount failures directly from fstab configuration.
Add documentation and generator tests.
The default attestation key for the TPM2 report signer is currently
created with an empty policy but with the adminWithPolicy attribute set.
This makes it impossible to authorize use of the key for the ADMIN role,
which is required by TPM2_Certify.
This drops the adminWithPolicy attribute so that it can be authorized
using its (empty) authorization value. This is appropriate for local
style attestation keys ("LAKs"). I've been working on an interface for
managing attestation keys which will support creating an IAK, and this
will have both the adminWithPolicy attribute set and a policy defined as
specified in the TCG "TPM 2.0 Keys for Device Identity and Attestation"
spec.
journal_entry_to_json() builds a fresh hashmap for every entry and then fills
it with that entry's fields, of which there are typically around twenty. Since
the hashmap starts out empty it has to grow several times per entry, and every
growth rehashes everything inserted so far. journalctl -o json spent about a
quarter of its time in siphash, a third of that under resize_buckets().
Reserve room up front. On a 730M journal spread over 100 files,
journalctl -o json -n 200000 goes from 5.71s to 4.64s.
sd_journal_next() sweeps every open journal file to find the next entry, and
next_beyond_location() rate-limits its tail timestamp refresh per file. Since
ratelimit_below() reads the clock itself, iterating a journal spread over a
hundred files meant a hundred clock_gettime() calls for every single entry,
nearly all of them only to conclude that the one second interval had not
elapsed yet. On a 730M journal in 100 files this was the single largest cost in
journalctl, over a fifth of the run.
Add ratelimit_below_at(), which takes the timestamp from the caller, and read
the clock once per sweep in real_journal_next() instead. ratelimit_below()
keeps its behaviour and becomes a wrapper.
The timestamp is now sampled once for the whole sweep rather than once per
file. The interval being rate-limited is a second and a sweep takes
microseconds, so this makes no practical difference.
On that journal, journalctl -o export goes from 4.02s to 2.93s, and the default
journalctl -o short from 3.30s to 2.14s.
This decoded every character twice: utf8_encoded_valid_unichar() decodes
internally to do its validation and then throws the codepoint away, and the
caller immediately called utf8_encoded_to_unichar() again to get the very same
codepoint back.
Fix that where it originates rather than at the call site: add
utf8_encoded_valid_unichar_full(), which optionally returns the character it had
to decode anyway, and make utf8_encoded_valid_unichar() a wrapper that passes
NULL. Callers that want both the validity and the value now decode once.
On top of that add a bulk fast path, since almost everything we are asked about
is plain printable ASCII: test eight bytes at a time for that, i.e. that no byte
is >= 0x80, none is < 0x20 and none is 0x7f. A remainder of less than eight
bytes is copied into a word padded with spaces so that it can be tested the same
way. A word that does not qualify falls through to the per character path, so
tabs, newlines and multibyte sequences are handled exactly as before.
The function sits on journalctl's per-field output path, so this is measurable:
on a 730M journal spread over 100 files, journalctl -o export goes from 4.02s to
about 2.9s.
Also extend the tests, which so far covered neither DEL nor the C1 range nor
allow_newline=false: place each of the interesting bytes at each offset within
the eight byte word, and check a multibyte character straddling the end of one.
The new code was verified to agree with the old one on 41945090 inputs, for both
values of allow_newline: every byte string of length zero to three over all 256
byte values, four million random strings over uniform bytes, over an ASCII-heavy
alphabet and over an alphabet of encoding boundary bytes, and a corpus placing
every byte value at every offset of buffers of every length from eight to
twenty-four bytes, with valid and truncated multibyte sequences straddling the
word boundary.
This is split out of #43058, but makes sense on its own: and makes
strip_tab_ansi() a lore more complete and correct.
ptybroker is going to use for the ansi sequence parser for other
purposes too, but making strip_tab_ansi() work better is enough of a
reason to have this I think.
It's nice to use --attach since it validates our images. So far, it
strictly insisted on checking that the image was a DDI however, which
limits its use for certain usecases.
Add "--relax" to address this limitation.
Fixes: #43605
systemd has two UTF-8 decoders, and the console width path uses the
permissive one.
`utf8_encoded_to_unichar()` performs no overlong check, no
`unichar_is_valid()` check, and still accepts the obsolete five and six
byte forms. Those checks live in the wrapper
`utf8_encoded_valid_unichar()`, which is what `utf8_is_valid()` and
`ellipsize_mem()` use. `utf8_char_console_width()` called the bare
decoder directly.
Checking both against `utf8_is_valid_n()` over all 4,244,897,280 NUL
free byte strings of length one to four, which is the entire UTF-8
encoding space:
| decoder | disagreements with `utf8_is_valid_n()` |
| --- | --- |
| `utf8_encoded_valid_unichar()` | 0 |
| `utf8_encoded_to_unichar()` | 8,835,838, always more permissive |
So `utf8_console_width()` returned a finite width for strings
`utf8_is_valid()` rejects. `format-table.c` measures a cell with
`utf8_console_width()` and, when it does not fit, hands it to
`ellipsize()`, which validates strictly and returns NULL. That is
reported as `-ENOMEM`:
```
lead surrogate valid_utf8=0 utf8_console_width=25 ellipsize=NULL
-> table_format FAILS: Cannot allocate memory
```
A surrogate, overlong, noncharacter or five byte sequence in any cell
failed the whole table with a bogus out of memory error, and only once
that column was narrow enough to need truncation, so whether it happened
depended on the width of the user's terminal. At other widths the
invalid sequence was printed to the terminal instead. Invalid cells are
now refused consistently with `-EINVAL`, which is what the sequences
`utf8_encoded_to_unichar()` already rejected always produced.
The second commit fixes a related disagreement. `ellipsize_mem()` and
`align_string_mem()` open coded character width as `unichar_iswide(c) ?
2 : 1`, while `unichar_console_width()` counts a tab as eight cells.
`format-table.c` uses both: it finds a field too wide with
`utf8_console_width()`, then asks `ellipsize()` to fit that many
columns, and `ellipsize()` thought a tab was one cell, so the result
still overflowed the column that the measurement had just asked it to
fit. A table pinned to `table_set_width(20)` emitted lines of 25, 44 and
39 columns. This uses `unichar_console_width()`, which was split out for
exactly this purpose in 5c0d4b5061.
Both commits add tests that fail before the change and pass after. The
full unit suite is otherwise unaffected.
Cover the size derivation and the packing decisions that the integration
test cannot reach: its veth MTU exceeds every record TEST-89 publishes,
so neither the oversized-record fallback nor the drop of a record beyond
the RFC 6762 section 17 ceiling is ever exercised there.
The cases pin the two size bounds for both families (including the
fallback for an MTU not known yet and the jumbo-frame cap), the split
at max_size, the lone oversized record in a fragmented packet, the
drop of a record that fits nowhere, the skip of a record that cannot be
encoded at all, that
answer-item flags reach the wire (a goodbye must keep TTL 0 and its
cache-flush bit), and that name compression stays within a packet --
the property the split exists for, since every DNS-SD name shares a
suffix and a compression pointer may not cross a packet boundary.
dns_scope_announce() put the entire zone -- for DNS-SD every PTR/SRV/TXT
record of every published service -- into a single reply packet. With a
few hundred published services that grows far beyond the interface MTU,
and beyond what RFC 6762 section 17 permits: a packet larger than the
MTU is sent using fragments, and such a packet "MUST NOT contain more
than one resource record".
Pack the announcement into as many MTU-sized packets as needed instead.
A single record that does not fit the MTU by itself is emitted alone in
an oversized packet, relying on IP fragmentation, which the same section
explicitly allows for that case -- bounded by its hard 9000-byte
ceiling, headers included, which also caps jumbo-frame links. While the
interface MTU is not known yet, packets are cut to the datagram size
every host of the family must be able to accept -- RFC 791 section
3.1's 576 octets for IPv4, whole or reassembled, and RFC 8200 section
5's 1280-byte minimum link MTU for IPv6, which no link fragments --
rather than to that ceiling, so an announcement is not packed for
deliberate fragmentation merely because the MTU was not read yet.
A record that cannot be encoded at any size -- a character string past
255 bytes, say -- is skipped like one too large for the ceiling, rather
than failing the whole announcement: dns_packet_append_rr() rolls the
packet back on every failure, so its neighbours are unaffected either
way. The packetizer returns how many it dropped and the scope warns on
it, since nothing else would distinguish an announcement that carried
everything from one that did not -- and for a withdrawal that matters:
the caller drops the records from the zone either way, so one skipped
here is never withdrawn at all and peers keep it for its full TTL.
The packing itself lives in resolved-mdns.c as mdns_announcement_packetize(),
taking an answer and the two size bounds and returning finalized packets,
so it can be tested without a scope, a link or a socket.
Emission is best effort per packet -- a failed send no longer withholds
the announcement's remaining packets -- and each pass logs the packet and
record counts it actually put on the wire. Each packet takes its own
token from the scope's multicast rate limit: that limit bounds datagrams
on the wire, and an announcement that would exhaust its burst is the
very flood it exists to cap.
dns_scope_send_goodbye(), the other unsolicited-response emitter, goes
through the same packetizer: a withdrawal of many records had the same
oversized-packet problem, and a single record failing to append
abandoned the whole withdrawal. Its return value follows the emitter's
now, so a withdrawal split across packets reports an error when any one
of them fails to go out.
credentials_varlink_error_by_errno() answers with the first row whose errno
matches, so two rows sharing one errno make the second unreachable and the
client is told the wrong error. This series re-keys UnexpectedPCRState to
EPERM and adds PCRsKeptChanging on EUCLEAN, so walk the table by id and check
that the errno lookup comes back to the same row, in both errno signs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Christian Hesse
Yu Watanabe
Luca Boccassi
Daan De Meyer
Heran Yang
Birk Skyum
Chris Coulson
Armaan Sandhu
Roman Perehonchuk
Christian Glombek
Alberto PlanasandClaude Opus 5