[seks99x]

This PR introduces --stream-source, enabling rsync to perform full rolling-checksum delta transfers directly from non-seekable named pipes (FIFOs) and process substitutions (when combined with -L / --copy-links) into local or remote regular destination files.

Previously, passing a FIFO to rsync either copied the special FIFO device node itself (--specials) or skipped it entirely. --stream-source disguises the source FIFO as a regular file inside the file list, reads the incoming stream sequentially through map_ptr(), computes rolling block/file checksums against the receiver's basis file on the fly, and seamlessly truncates the logical file length if the stream reaches EOF before the configured ceiling.

Key Features & Options

• --stream-source: Instructs the local sender to treat a single local source FIFO as a regular file and stream its contents through the delta-transfer engine.
• --stream-max-size=SIZE (Required): Sets the initial virtual file size ceiling advertised in the file list before the stream is consumed.
• --stream-timeout=SECONDS: Configures a select() read timeout on the FIFO descriptor inside map_ptr() (default: 30 seconds; <= 0 disables the timeout, matching --timeout behavior).
• --stream-pipe-sz=SIZE: Adjusts the kernel pipe ring-buffer capacity on Linux via fcntl(fd, F_SETPIPE_SZ, ...) within [64K, 16M] (default: 1M) to reduce context-switch overhead between the writer process and rsync.

Why --stream-max-size is Required as an Initial Ceiling

Named pipes report st_size == 0 via stat(), and the true byte length of a live stream cannot be known until the writer closes the pipe. However, rsync's protocol and delta-transfer engine require a file size during the flist phase (make_file()) and before entering match_sums() so that:

1. The sender can advertise a valid regular-file entry to the generator/receiver without changing the wire protocol (preserving full compatibility with older remote rsync peers).
2. The receiver can compute and transmit basis block checksums (s->sums) covering the destination file.
3. map_file() and hash_search() can initialize the sliding window bounds (len and end).

--stream-max-size acts strictly as an upper-bound ceiling, not an exact size requirement:

• Early EOF (Stream < --stream-max-size): Users can safely pass a generous ceiling (e.g., --stream-max-size=10G for a database dump that turns out to be 42M or even 0 bytes). When read() in map_ptr() hits 0 (EOF), map->file_size and map->p_len shrink to the exact byte count read (map->p_fd_offset). Both hash_search() and match_sums() detect buf->file_size < len, clamp len dynamically, and flush the remaining literals/tokens up to the real EOF so the reconstructed destination file (both temporary-file and --inplace modes) has the exact stream byte length. On the sender side, sender.c reconciles stats.total_size and stats.total_transferred_size to the actual bytes read. Because the initial flist entry transmits stream_max_size without a wire protocol change, the advertised ceiling still controls %l format logging, transfer progress percentages, and receiver-side (or daemon-side) total-size statistics.
• Hard Cap (Stream > --stream-max-size): If a producer writes more bytes than --stream-max-size, rsync stops reading at the exact limit, probes the pipe with a 1-byte read in sender.c to detect unconsumed data, logs a warning, and completes the transfer cleanly at the ceiling size without overflowing.

Assigning Virtual File Metadata

Because make_file() converts the FIFO's mode from S_IFIFO to S_IFREG while preserving the rest of the stat structure, standard rsync metadata options work normally on the resulting destination file:

• Permissions & Mode: Controlled directly via --perms (-p) and --chmod (e.g., --perms --chmod=F600 to lock down a streamed backup).
• Ownership: Controlled via --owner (-o), --group (-g), and --chown=USER:GROUP.
• Timestamps: Controlled via --times (-t), applying the FIFO's mtime or custom modification times to the target regular file.
• Symlinked Streams: Combining with -L / --copy-links allows streaming from symlinked pipes such as /dev/stdin or shell process substitutions (<(cmd)), using the target file's name on the destination.

Option Restrictions & Automatic Overrides

Because a FIFO can only be read once sequentially and its true size is unknown until EOF, --stream-source is restricted to a single local sender stream and blocks options that require directory traversal, multiple passes, or a stable pre-known source size:

• Local sender only (no remote pull or daemon mode): --stream-source is only accepted on the local client acting as the sender (!am_server && !am_daemon && am_sender). Pushing to a local path, remote shell (HOST:DEST), or remote rsync daemon (HOST::DEST / rsync://) works normally without requiring --stream-source support on the remote peer.
• No directory recursion (-r, -a, -d, --files-from): Stream mode operates on a single explicit FIFO source. While -a is rejected because it implies -r and --specials, individual metadata flags (-p, -t, -o, -g, -L) can still be passed normally.
• No multi-pass modes (-c / --checksum, --append, --append-verify, batch mode): --checksum would consume the pipe during flist generation before the transfer starts, and --append / batch modes conflict with single-pass stream semantics.
• No file-size-dependent options (--size-only, --max-size, --min-size, -y / --fuzzy, --preallocate): Blocked explicitly because the generator only sees the transmitted stream_max_size ceiling in the flist rather than the final stream length.
• No --remove-source-files: Prevents unlinking persistent named pipes or /dev/fd/* descriptors after transfer.

Additionally, --stream-source automatically forces --ignore-times (ignore_times = 1) and clears update_only, preserve_links, preserve_specials, and preserve_devices so the generator never skips the transfer based on quick checks or copies the FIFO node itself instead of its data.

Implementation Details

• flist.c (make_file): Validates that the source is S_ISFIFO(st.st_mode) when is_stream() is active, rewrites st.st_mode from S_IFIFO to S_IFREG, and sets st.st_size = stream_max_size.
• fileio.c (map_file, map_ptr):
  • Applies fcntl(fd, F_SETPIPE_SZ, stream_pipe_sz) in map_file() under #ifdef F_SETPIPE_SZ.
  • Refuses backward seeks (read_start != map->p_fd_offset) on streams in map_ptr().
  • Waits on select() when stream_timeout > 0, retries on EINTR, and handles early EOF (nread == 0) by truncating map->file_size = map->p_fd_offset, setting map->p_len = read_offset, and zeroing the unused buffer tail.
• match.c (hash_search, match_sums):
  • Updates len, end, and k whenever stream_early_EOF(buf, len) triggers across the initial read, the hash-collision skip path, and the rolling-window advance path.
  • Passes offset - backup (last_match) to map_ptr() during hash-collision skips (matching the normal byte-by-byte path) so unsent literal bytes (last_match .. offset) stay in map_ptr()'s sliding buffer instead of being evicted and triggering a backward lseek() when matched() flushes them.
  • Pre-reads each chunk in match_sums() when s->count == 0 so early EOF shrinks len before matched() flushes literal chunks, and guards map_ptr(buf, len-1, 1) at finish_search: with if (len > 0) to handle 0-byte streams safely.
• sender.c (send_files): Re-verifies S_ISFIFO, checks whether data remains in the pipe after reaching stream_max_size, and reconciles sender-side stats.total_size and stats.total_transferred_size with mbuf->file_size.
• Documentation & Tests: Documents the stream options in rsync.1.md and usage.c, and adds two comprehensive test suites (testsuite/stream-source_test.py and testsuite/stream-early-eof_test.py).
  Feature Proposal: Native Stream with UNIX Pipe discussion (--stream-source) #1117

[seks99x]

I think XNU/NetBSD would require extra/different pipe handling for EOF. Will look into that.

[steadytao]

FD_SET is undefined when the descriptor reaches FD_SETSIZE -- With 1100 =< inherited descriptors there can be out-of-bounds access under UBSan and a stack-buffer-overflow under ASan. Use poll() or ppoll() and add the high-descriptor regression.

[steadytao]

This read happens after transfer finalisation and does not use the stream timeout. An exact-limit stream can hang here and an oversized stream commits truncated output with exit status 0. Detect excess with the same bounded wait before finalisation and fail the transfer when extra data exists.

[steadytao]

Docs say non-positive values disable the timeout but -1 restores the 30-second default. Keep one documented meaning and test the actual stalled-stream behaviour.

[steadytao]

Fails because the second source is a regular file not because multiple sources were rejected. Two FIFO sources are currently accepted and both copied. Reject the source count before opening either FIFO and test two FIFOs in both orders.

[seks99x]

I think XNU should have different handling with poll also.

[seks99x]

@steadytao poll would introduce inconsistency and weird behavior with XNU due to how it handles EOF. I think I might fall back to select only in macOS and make bounds check for the fd number.

[seks99x]

@steadytao Also, XNU supports expanding FD_SETSIZE (Linux do not) at compile time via _DARWIN_UNLIMITED_SELECT, I'll bump FD_SETSIZE in fileio.c alongside the select() fallback and bounds check so high file descriptors still work cleanly on macOS without dynamic allocations or complex handling.