Documented cf-reactor wakeup sources

content/reference/components/cf-reactor.markdown

@@ -88,17 +88,82 @@ The daemon is then signaled to handle the event and run the bundles of the `then
 
 **Notes:**
 
-- Event detection is based on polling, but `cf-reactor` does not use busy-waiting.
+- Event detection is based on polling, sped up by [wakeup sources](#wakeup-sources), but `cf-reactor` does not use busy-waiting.
   Instead, the polling thread sleeps between polling cycles and remains dormant when no events are registered.
 - `cf-reactor` does not guarantee that an event will be handled immediately after it occurs.
   The delay depends on factors such as the polling interval, policy reloads, and slow code execution.
   However, `cf-reactor` guarantees that changes will be detected as fast as possible and the corresponding code executed, provided that execution does not fail.
+- After a wakeup source wakes it up, `cf-reactor` waits 200 milliseconds before checking, so that a burst of changes is checked once.
+- An event that fires more than 10 times in 60 seconds is delayed, starting at 10 seconds and doubling up to 5 minutes, until it calms down.
+  The changes it holds back are handled in one run when the delay ends.
+  Meanwhile, and while its bundles run, the event is not checked again: it is checked once when it can run.
+
+### Wakeup sources
+
+A wakeup source wakes an event's watcher up as soon as the operating system reports a change to the watched file or its directory, so that `cf-reactor` checks the event right away instead of at its next [`interval`][events#interval].
+Each event uses the first available wakeup source of its [`wakeup_sources`][events#wakeup_sources] attribute.
+Wakeup sources only decide _when_ an event is checked, never _whether_ it happened, and polling continues underneath all of them.
+Changing the wakeup source never changes which events fire, only how soon.
+
+- `ebpf` (Linux, Enterprise) loads a small program into the kernel that matches changes against the watched files' inodes before telling `cf-reactor`.
+  It needs kernel 5.8 or later (or the RHEL 8 kernel) with BTF, and `CAP_SYS_ADMIN` (or `CAP_BPF` and `CAP_PERFMON`).
+  It has no per-directory watch limits, and watches up to 65,536 files; further events use their next wakeup source, and `cf-reactor` logs a warning.
+- `fanotify` and `inotify` (Linux, Enterprise) watch each file for writes and changes of its permissions or owner, and its parent directory for files being created, deleted and renamed.
+  Files created and deleted in that directory are reported to `cf-reactor`, which ignores the ones for other files.
+  `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`.
+  When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval.
+  Raise the limit to watch more than about half as many files as it allows.
+- `usn` (Windows, Enterprise) reads the change journal of NTFS and ReFS volumes, waiting for new records as they are written.
+  It sees changes made while `cf-reactor` was stopped, and doesn't lose changes during bursts, so it comes before `rdcw`.
+  It needs administrator rights and a change journal on the volume; `cf-reactor` never creates one.
+  If the journal is deleted, recreated or overwritten past the last record read, every event on that volume is checked.
+- `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW` on the watched files' directories, and works on every local file system, including FAT32 and exFAT.
+  When its buffer overflows during a burst, the events of that directory are checked.
+  It isn't told when a watched directory, or one above it, is renamed or moved; `cf-reactor` notices within 5 seconds and watches the new location.
+  Changes made while `cf-reactor` was stopped are found by the first check after it starts.
+- `poll` only checks at the interval. It is the only wakeup source in CFEngine Community.
+
+`package_installed` and `package_uninstalled` events always use `poll`: the installed packages are listed every 10 seconds whatever the wakeup source, so no wakeup source can make them faster.
+
+**Cost:** away from watched files, no wakeup source adds measurable CPU time to the rest of the system.
+In a directory that holds a watched file, each file operation costs about 1 µs more with `ebpf`, `fanotify` or `inotify` (about 10%), counting the process making the change and `cf-reactor`.
+At 1,000 file operations per second in one watched directory this is about 0.1% of a CPU core, so the choice of wakeup source makes no practical difference to CPU time.
+Where the wakeup sources differ:
+
+| Source     | Wakes the watcher after a change | Kernel resources                                                         |
+| ---------- | -------------------------------- | ------------------------------------------------------------------------ |
+| `ebpf`     | under 2 ms                       | up to 65,536 files, about 0.7 MB of locked memory for 1,000              |
+| `fanotify` | about 20 ms                      | one mark per file and one per directory (`fs.fanotify.max_user_marks`)   |
+| `inotify`  | about 20 ms                      | one watch per file and one per directory (`fs.inotify.max_user_watches`) |
+| `usn`      | under 2 ms                       | one reader per volume, none per file                                     |
+| `rdcw`     | under 3 ms                       | one directory handle per watched directory                               |
+| `poll`     | at the next interval             | none                                                                     |
+
+`cf-reactor` then waits 200 milliseconds before checking, so in practice the native wakeup sources react equally fast.
+`fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup.
+Measured on Ubuntu 24.04 (kernel 6.8) and Windows Server 2019, in virtual machines.
+
+`file_content_modified` hashes the whole file when its size or modification time changed.
+For a large file that is written to all the time, such as a log, this costs about as much with every wakeup source, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core.
+Use `file_modified` for such files, which only compares the modification time.
+
+**Changes only seen at the next interval:**
+
+- writes through a memory mapping (`mmap`) that stays open, with every wakeup source
+- with `ebpf`, writes on file systems that don't report them to the kernel's generic write path (for example some network file systems) until the file is closed
+- with every wakeup source, changes on network file systems made by other hosts
+
+**Notes:**
+
+- On CFEngine Enterprise, the wakeup source each event uses is reported to the hub, in the `reactor_watchers` table of the reporting database.
+- SELinux or other mandatory access control may deny `cf-reactor` a wakeup source; it then falls back to the next one in the list.
 
 ## Performance
 
 `cf-reactor` uses these threads:
 
 - A watcher thread checks for events, and queues the events it detects.
+  Events about files on network file systems are checked by a second watcher thread, so that a server that stops responding doesn't hold up the other events.
 - The main thread takes the events from the queue and runs their [`then`][events#then] bundles, one event at a time.
 - For `package_installed` and `package_uninstalled` events, a listing thread lists the installed packages every 10 seconds.
   There is one listing thread per [package module][Package modules] (e.g. `apt_get`), shared by all the `events` promises using it, whatever the number of packages they watch.
@@ -109,9 +174,10 @@ How many events per second it can handle depends on two things:
 - The cost of `cf-reactor` itself: checking for the event, which depends on the event type, and setting up and running the `then` bundles.
 - The work done by the `then` bundles, and anything outside of `cf-reactor`, such as how fast the package manager lists the installed packages.
 
-Each `events` promise fires at most once per check, so at most once per [`interval`][events#interval].
-With `N` `events` promises and an interval of `I` seconds, `cf-reactor` has at most `N / I` events per second to handle.
+Each `events` promise fires at most once per check.
+Without a [wakeup source](#wakeup-sources), it is checked once per [`interval`][events#interval]; a wakeup source can check it sooner.
 Changes happening between two checks are seen as one event.
+Either way, an `events` promise that fires more than 10 times in 60 seconds is delayed (see [Under the hood](#under-the-hood)), so with `N` `events` promises that keep firing, `cf-reactor` has at most about `N / 6` events per second to handle.
 
 The figures below were measured on an Amazon EC2 `t2.micro` instance (1 vCPU, Intel Xeon E5-2686 v4 at 2.30 GHz, 1 GB of memory) running Ubuntu 24.04.
 Each `events` promise watched its own file, user or package, with the default `interval` of 1 second, and ran this `then` bundle:
@@ -151,6 +217,8 @@ The limit is the most events per second `cf-reactor` can handle for each event t
 - Above the limit, `cf-reactor` doesn't lose events: they pile up in the queue and are handled late.
   The queue has no size limit: if events keep coming faster than `cf-reactor` handles them, the delay keeps growing, and so does the memory used by the queue, until `cf-reactor` runs out of memory.
 - Slower `then` bundles lower these limits.
+- These figures were measured with polling only, before an `events` promise was limited to 10 runs in 60 seconds.
+  They are totals across many `events` promises, each watching its own file, user or package: no single `events` promise reaches these rates, since one that fires more than 10 times a minute is delayed.
 
 ## Enterprise hub tasks
 

content/reference/promise-types/events.markdown

@@ -119,6 +119,7 @@ bundle reactor main
 
 Each event has its own interval.
 A lower value means the event is noticed sooner but checked more often.
+A [wakeup source](#wakeup_sources) can wake the check up earlier, so with one the interval mostly bounds how late a missed change is noticed.
 For how polling works, see [Under the hood][cf-reactor#Under the hood].
 
 ```cf3
@@ -135,6 +136,50 @@ body when deleted_slow(file)
 
 **Default value:** `1`
 
+#### wakeup_sources
+
+**Description:** The wakeup sources that may wake this event's watcher up before its next interval, in order of preference.
+
+A wakeup source is a way for the operating system to tell `cf-reactor` that the watched file, or its directory, changed; `poll` means none, only the interval.
+It makes `cf-reactor` check the event right away instead of at the next interval.
+It never decides whether the event happened: the check does, so the same events fire whatever the wakeup source.
+Polling at the [`interval`](#interval) continues underneath every wakeup source, and catches what a wakeup source misses.
+
+`cf-reactor` uses the first wakeup source in the list that is available on the host and can watch the event's target.
+Without `wakeup_sources`, it tries `ebpf`, `fanotify`, `inotify`, `usn`, `rdcw` and `poll`, in that order.
+CFEngine Community only has `poll`.
+If none of the listed wakeup sources is available, the event is not watched and `cf-reactor` logs an error; list `poll` last to fall back to polling instead.
+
+| Source     | Platform | Edition    |
+| ---------- | -------- | ---------- |
+| `ebpf`     | Linux    | Enterprise |
+| `fanotify` | Linux    | Enterprise |
+| `inotify`  | Linux    | Enterprise |
+| `usn`      | Windows  | Enterprise |
+| `rdcw`     | Windows  | Enterprise |
+| `poll`     | All      | All        |
+
+For how the wakeup sources differ, see [Wakeup sources][cf-reactor#Wakeup sources].
+
+```cf3
+body when passwd_modified
+{
+  file_content_modified => "/etc/passwd";
+  interval => 300;
+  wakeup_sources => { "inotify", "poll" };
+}
+```
+
+**Type:** `slist`
+
+**Allowed input range:** `poll`, `inotify`, `fanotify`, `ebpf`, `usn`, `rdcw`
+
+**Notes:**
+
+- changing `wakeup_sources` does not reset the event: on policy reload, a watcher keeps its state when only its wakeup sources change.
+- with `wakeup_sources => { "poll" }` the event is only checked every `interval` seconds.
+- no wakeup source can watch `package_installed` and `package_uninstalled` events: they use `poll`, so a list without `poll` leaves them unwatched.
+
 #### file_deleted
 
 **Description:** Creates an event when the file at the given path is deleted.