As discussed in the PR, those fields can go down, so they cannot be advertised
as type COUNTER. But the type is shared for the whole family, so we need to
create a new family to keep the other ones as COUNTER and add the new ones
as GAUGE.
kernel/sched/cputime.c says above get_cpu_idle_time_us() and get_cpu_iowait_time_us():
> Note that this is partially broken due to the counter of iowait tasks that
> can be remotely updated without any synchronization. Therefore it is possible
> to observe backward values within two consecutive reads.
steal is apparently also unreliable because it is reported by the hyperviser.
So let's be cautious and report it as gauge too.
Extend the io.systemd.Basic metric families with host-level runtime
metrics: cumulative per-state CPU seconds, memory in use (MemTotal minus
MemAvailable), swap in use, the kernel's 10s pressure stall averages
(cpu/memory/io x some/full). Partitions and stacked devices are excluded
from the disk counters, since their traffic is already accounted for in
the parent device.
Also extend the TEST-74 report subtest with checks for the new metrics.
Network metrics are not added, because the service runs with
PrivateNetwork=yes, and it doesn't make sense to report stats for 'lo'.
After some back-and-forth, I settled on "CPUUsage". We use this scheme of
capitalization in other metric family names: "IPv4Public", "IPv6Public",
IPv4AddressState", "IPv6AddressState", "OSRelease" "SMBIOS" "BootID",
"CPUsOnline", "MachineID", "MachineInfo", "OSRelease",
"SMBIOS.SystemManufacturer", "TPM2.Manufacturer", "TPM2.VendorString",
"IOReadBytes", "IOReadOperations", "CGroup". The exception is
"io.systemd.cgroup.CpuUsage". I think we should treat that one as a
mistake and consider changing it to conform.
The networkd metrics interface already reports a lot of interesting
metrics. With this commit it also report the network addresses too.
Each ready address is emitted as one record per (interface, address)
pair:
- object: ifname
- value: address in CIDR notation
- fields: { family: "ipv4"|"ipv6", scope: "global"|"link"|"host"|... }
The loopback addresses are not reported as its just noise.
Example output:
```
root@localhost:~# varlinkctl --more --json=short call /run/systemd/report/io.systemd.Network io.systemd.Metrics.List '{}'
{"name":"io.systemd.Network.Address","object":"enp0s1","value":"fe80::5054:ff:fe12:3456/64","fields":{"family":"ipv6","scope":"link"}}
{"name":"io.systemd.Network.Address","object":"enp0s1","value":"fec0::5054:ff:fe12:3456/64","fields":{"family":"ipv6","scope":"site"}}
{"name":"io.systemd.Network.Address","object":"enp0s1","value":"10.0.2.15/24","fields":{"family":"ipv4","scope":"global"}}
```
This commit adds the boot timeline (MANAGER_TIMESTAMP_KERNEL/USERSPACE/FINISH) as
metrics. The kernel CLOCK_MONOTONIC value is 0 by definition, so only its
.Realtime is reported. For userspace and finish report both .Realtime and
.Monotonic. The naming follows D-Bus.
Add io.systemd.Basic.OSRelease metric families that reports select fields
from os-release.
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="45 (Cloud Edition Prerelease)"
RELEASE_TYPE=development
ID=fedora
VERSION_ID=45
VERSION_CODENAME=""
PRETTY_NAME="Fedora Linux 45 (Cloud Edition Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:45"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/rawhide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=rawhide
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=rawhide
SUPPORT_END=2027-11-24
VARIANT="Cloud Edition"
VARIANT_ID=cloud
$ varlinkctl call --more ./build/systemd-report-basic io.systemd.Metrics.List {} | jq --seq -c
...
{"name":"io.systemd.Basic.OSRelease.NAME","value":"Fedora Linux 44 (Workstation Edition)"}
{"name":"io.systemd.Basic.OSRelease.ID","value":"fedora"}
{"name":"io.systemd.Basic.OSRelease.CPE_NAME","value":"cpe:/o:fedoraproject:fedora:44"}
{"name":"io.systemd.Basic.OSRelease.VARIANT_ID","value":"workstation"}
{"name":"io.systemd.Basic.OSRelease.VERSION_ID","value":"44"}
{"name":"io.systemd.Basic.OSRelease.SUPPORT_END","value":"2027-05-19"}
I picked the fields that contain useful information about the specific
version/image/variant/experiment/flavour of the system. Also, either
NAME or PRETTY_NAME is included. This one is intended for human readers
to be able to identify the OS version easily.
SUSE uses a different preset, so don't just assert in the test,
instead just start the socket in case it is not enabled
TEST-74-AUX-UTILS.sh[1594]: ++ systemctl is-enabled systemd-report-basic.socket
TEST-74-AUX-UTILS.sh[1540]: + [[ disabled == enabled ]]
TEST-74-AUX-UTILS.sh[120]: + echo 'Subtest /usr/lib/systemd/tests/testdata/units/TEST-74-AUX-UTILS.report.sh failed'
Follow-up for 4409e52494
In https://github.com/systemd/systemd/pull/41688 we merged metrics
and facts for systemd-report. However while some metric sources
are enabled by default (like `io.systemd.{Manager,Network}`) the
`io.systemd.Basic` service is not enabled by default.
This commit changes this and enables it by default.
We could also enable the systemd-report-cgroup.socket but that sends
a lot more data not sure that is a good default.
We removed the concept of facts, so we need to update the existing
io.systemd.Basic facts provider to metrics. This commit does just
that. Its mostly mechanical.
This also means that facts.{c,h} and varlink-io.systemd.Facts.{c,h}
are gone now.
Two new verbs are added: "generate" and "upload". The first one just
creates a "report", i.e. puts the metrics into a structured JSON object
that in the future is intended to carry additional data like a
signature:
$ build/systemd-report generate io.systemd.Manager.UnitsTotal
{
"mediaType" : "application/vnd.io.systemd.report",
"timestamp" : "Tue 2026-04-28 22:30:09 UTC",
"metrics" : [
{
"name" : "io.systemd.Manager.UnitsTotal",
"value" : 520
}
]
}
The second verb can be used to upload or otherwise process the report.
It builds on the code added in 0a8560eed8.
In /run/systemd/metrics-upload/ we expect a set of sockets. We'll call
out to each one of them. This allows the data to be processed in custom
ways, incl. writing to storage or sending over the network.
Each socket must provide a single interface:
io.systemd.Metrics.Upload {"report":$data}
Extend io.systemd.CGroup.CpuUsage from a single per-unit nanosecond
counter to three rows distinguished by a "type" field of "total",
"user", or "system". The values come from cpu.stat's usage_usec,
user_usec and system_usec keys, read in a single keyed-attribute
fetch and cached on each CGroupInfo so each scrape only opens
cpu.stat once per cgroup.
This gets rid of the duality in the cmdline interface and various
APIs. The general plan is collect both "facts" and "metrics" in a
single list. We have various producers which respond on the
io.systemd.Facts endpoint. Those will need to be adjusted to respond
to io.systemd.Metrics.
Cmdline interface:
'metrics' (unchanged)
'describe-metrics' → 'describe'
'facts' → merged into 'metrics'
'describe-facts' → merged into 'describe'
Extend fake-report-server.py with optional --cert, --key, --port
arguments for TLS support. Add a test case that generates a
self-signed certificate and tests HTTPS upload of metrics and facts.
Also exercise the --header param.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Add a fake HTTP server (fake-report-server.py) that accepts JSON POST
requests and validates the report structure, and test cases in
TEST-74-AUX-UTILS.report.sh that exercise plain HTTP upload of both
metrics and facts.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Add CpuUsage, MemoryUsage, IOReadBytes, IOReadOperations, and
TasksCurrent in a standalone socket-activated varlink service.
The new systemd-report-cgroup service listens at
/run/systemd/report/io.systemd.CGroup and exposes:
- io.systemd.CGroup.CpuUsage
- io.systemd.CGroup.IOReadBytes
- io.systemd.CGroup.IOReadOperations
- io.systemd.CGroup.MemoryUsage (with type=current/available/peak)
- io.systemd.CGroup.TasksCurrent
The collection of facts is entirely unprivileged and has very little to
do with PID1. PID1 is privileged and single-threaded and a point of
contention, so we shouldn't put things in PID1 that don't need to be
there. A separate service can be enabled/disabled/started/stopped at
will, is easy to sandbox, etc. If it turns out to be necessary to
collect some facts through PID1 in the future, we can always add a
smaller facts endpoint to PID1 again.
The tool should not be considered stable, and those things we usually
place in /usr/lib/systemd, and not in $PATH.
We can move that to $PATH once we are confident it's gonna stay the way
it is.
Lennart Poettering
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
Michael Vogt
Yaping Li