rsync

CI Interop Validation Release

oc-rsync

rsync re-implemented in Rust. Wire-compatible with upstream rsync 3.4.2 (and back-compat with 3.4.1, protocol 32), works as a drop-in replacement.

Binary name: oc-rsync - installs alongside system rsync without conflict.

Status

Release: 0.6.2 (alpha) - Wire-compatible drop-in replacement for rsync 3.4.2 (and 3.4.1, protocols 28-32).

All transfer modes (local, SSH, daemon), delta algorithm, metadata preservation, incremental recursion, and compression are complete. Interop tested against upstream rsync 3.0.9, 3.1.3, and 3.4.1.

Component Status
Transfer Local, SSH, daemon push/pull
Delta Rolling + strong checksums, block matching
Metadata Permissions, timestamps, ownership, ACLs (-A), xattrs (-X)
File handling Sparse, hardlinks, symlinks, devices, FIFOs
Deletion --delete (before/during/after/delay), --delete-excluded
Compression zlib, zstd, lz4 with level control and auto-negotiation
Checksums MD4, MD5, XXH3/XXH128 with SIMD (AVX2, SSE2, NEON)
Incremental recursion Pull and push directions, enabled by default
Batch --write-batch / --read-batch roundtrip
Daemon Negotiation, auth, modules, chroot, syslog, pre/post-xfer exec
Filtering --filter, --exclude, --include, .rsync-filter, --files-from
Reference dirs --compare-dest, --link-dest, --copy-dest
Options --delay-updates, --inplace, --partial, --iconv, fuzzy matching
I/O io_uring (Linux 5.6+), copy_file_range, clonefile (macOS), adaptive buffers
Platforms Linux, macOS (full); Windows (NTFS DACL partial, xattrs via NTFS ADS, IOCP socket I/O; no symlinks/devices)

Platform Support

The primary platform is Linux. macOS is well-supported with parity for all metadata, ACL, and xattr features, including AppleDouble (._foo) resource-fork preservation. Windows builds and runs core transfer modes with NTFS DACL preservation (via windows-rs GetNamedSecurityInfoW/SetNamedSecurityInfoW, currently Tier 1C partial - see docs/platform-notes.md for the Windows ACL behavior summary, the –acls entry in docs/oc-rsync.1.md, and docs/design/windows-ntfs-acl-support.md for the documented lossy cases), xattrs (via NTFS Alternate Data Streams), and IOCP socket I/O (WSARecv/WSASend); symlinks and POSIX device nodes remain stubbed.

Feature Linux macOS Windows Notes
Permissions (-p) ✓ ✓ ⚠ Windows preserves only the read-only flag; POSIX mode bits are not applicable.
Times (-t) ✓ ✓ ✓ Nanosecond precision via the filetime crate on all platforms.
File ownership (-o/-g, uid/gid) ✓ ✓ ✗ apply_ownership_from_entry is a no-op on non-Unix; uid/gid mapping is Unix-only.
ACLs (-A) ✓ ✓ ⚠ Uses exacl on Linux/macOS/FreeBSD. Windows uses windows-rs GetNamedSecurityInfoW/SetNamedSecurityInfoW for NTFS DACL round-trip (Tier 1C partial): deny ACEs, inherited ACEs, the SACL, non-rwx access bits, and unresolvable SIDs are dropped with a one-time warning. SDDL fidelity payload, --audit-acls, --fail-on-windows-acl-loss, and --windows-acls are planned. See docs/platform-notes.md for the Windows ACL behavior summary, docs/design/windows-ntfs-acl-support.md for the full mapping matrix, and the –acls entry in docs/oc-rsync.1.md.
Extended attributes (-X) ✓ ✓ ✓ Linux/macOS via the xattr crate (macOS adds AppleDouble resource-fork support); Windows stores xattrs as NTFS Alternate Data Streams.
Hardlinks (-H) ✓ ✓ ✓ Uses portable std::fs::hard_link; works on NTFS.
Symbolic links ✓ ✓ ✗ create_symlinks is cfg(not(unix)) no-op; symlink entries are skipped on Windows.
Devices/specials (-D) ✓ ✓ ✗ create_fifo and create_device_node are no-ops on non-Unix.
Sparse files (-S) ✓ ✓ ⚠ Uses portable seek + set_len; depends on filesystem (NTFS supports sparse but is not explicitly marked via FSCTL_SET_SPARSE).
Async I/O backend ✓ io_uring ⚠ standard I/O ⚠ IOCP compiled, not wired io_uring runtime-detected on Linux 5.6+; IOCP is implemented in fast_io but not yet consumed by the transfer pipeline (#1868).
Reflink / clone copy ✓ FICLONE ✓ clonefile ⚠ ReFS reflink Linux Btrfs/XFS/bcachefs via FICLONE; macOS via clonefile; Windows via FSCTL_DUPLICATE_EXTENTS_TO_FILE (ReFS only).
Optimized file copy ✓ copy_file_range ✓ fcopyfile ✓ CopyFileExW All three are wired into the local-copy executor with standard-I/O fallback.

Legend: ✓ supported, ⚠ partial or not yet wired, ✗ not implemented.

What’s New (v0.6.2)

Bug fixes

--info producer emissions (upstream parity)

--debug producer emissions (upstream parity)

INC_RECURSE instrumentation (sender-side)

Compression

Checksums

Upstream interop

Code quality

Daemon & metadata

Documentation

Interop Testing

Tested against upstream rsync 3.0.9, 3.1.3, 3.4.1, and 3.4.2 in CI across protocols 28-32. Both push and pull directions verified for 30+ scenarios covering transfer modes, deletion, compression, metadata, reference dirs, file selection, batch roundtrip, path handling, device nodes, and daemon auth.

Performance

Benchmark: oc-rsync vs upstream rsync

Threaded architecture replaces upstream’s fork-based pipeline while keeping full protocol compatibility, reducing syscall overhead and context switches. Adaptive I/O buffers scale from 8KB to 1MB based on file size. Optional io_uring on Linux 5.6+ with three policies: auto (default; probe kernel and fall back to standard I/O), --io-uring (require io_uring; error if unavailable), --no-io-uring (always use standard buffered I/O). The active backend is reported by --version and -vv output. See oc-rsync(1) for details.

Known Limitations / Architectural Trade-offs

oc-rsync is wire-compatible with upstream rsync 3.4.1, but a few architectural choices and unfinished surfaces are worth calling out for operators planning a deployment:

Installation

Homebrew

brew tap oferchen/rsync https://github.com/oferchen/rsync
brew install oferchen/rsync/oc-rsync

Prebuilt packages

Download from the Releases page:

Platform Formats
Linux (x86_64, aarch64) .deb, .rpm (with OpenSSL), static musl .tar.gz
macOS (x86_64, aarch64) .tar.gz
Windows (x86_64) .tar.gz, .zip

Linux static tarballs are available in two checksum variants:

Variant Filename Description
Pure Rust (recommended) *-musl.tar.gz Pure-Rust checksums, zero system dependencies
OpenSSL *-musl-openssl.tar.gz OpenSSL-accelerated MD4/MD5 checksums (vendored, statically linked)

Each release also includes three toolchain variants: stable (recommended, no suffix), beta (-beta), and nightly (-nightly).

Build from source

Requires Rust 1.88+.

git clone https://github.com/oferchen/rsync.git
cd rsync
cargo build --workspace --release

Cargo features

The workspace exposes the following opt-in features. Defaults are tuned for the release binary on modern Linux/macOS/Windows hosts; everything marked experimental is wired but not yet promoted to the default set.

Feature Crate(s) Default Purpose Status
zstd workspace, core, engine, transfer, compress, protocol, batch yes Enables zstd compression codec and wire negotiation. stable
lz4 workspace, core, engine, transfer, compress, protocol yes Enables LZ4 compression codec. stable
zlib-ng workspace, core, engine, transfer, compress, protocol no Selects the SIMD-accelerated zlib-ng C backend instead of pure-Rust zlib. stable
xattr workspace, cli, core, transfer, daemon, metadata yes Preserves extended attributes (-X) on Unix and NTFS ADS on Windows. stable
acl workspace, cli, core, engine, transfer, daemon, metadata yes Preserves POSIX/NFSv4 ACLs (-A) via exacl, NTFS ACLs via windows-rs. stable
iconv workspace, cli, core, transfer, daemon, protocol yes Filename and symlink-target charset transcoding (--iconv). stable
parallel workspace, cli, engine, checksums yes Rayon-based multi-core file and checksum operations. stable
io_uring workspace, transfer, fast_io yes Linux 5.6+ batched async I/O with runtime fallback. stable
iocp workspace, transfer, fast_io yes Windows I/O Completion Ports for overlapped file and socket I/O. stable
copy_file_range workspace, fast_io yes Compat alias; the copy_file_range syscall is now always compiled with runtime detection. stable
async workspace, core, engine, transfer, daemon yes Brings in tokio for async I/O paths across the orchestrator stack. stable
openssl workspace, checksums no Routes MD4/MD5 through the system OpenSSL build. stable
openssl-vendored workspace, checksums no Same as openssl but statically links a vendored OpenSSL. stable
embedded-ssh workspace, core, rsync_io no Pure-Rust SSH client via russh; removes the runtime dependency on system ssh. stable
sd-notify workspace, core, daemon no systemd sd-notify integration for the daemon. stable
incremental-flist transfer yes Incremental file-list processing with failed-directory tracking. stable
lazy-metadata engine yes Defers stat() calls until metadata is needed. stable
multi-producer engine no Relaxes the single-producer compile-time invariant on WorkQueueSender. experimental
thread-slab-pool engine no Per-thread bounded LIFO slab in front of BufferPool; pays off above ~32 workers (#1271, #1370). experimental
vmsplice fast_io, transfer no Linux vmsplice(2) + splice(2) zero-copy writer for large page-aligned chunks. experimental
async-ssh core, rsync_io no Wires AsyncSshTransport into the client remote path; opt-in at runtime via OC_RSYNC_ASYNC_SSH=1 (#1593, #1796, #1805, #1806). experimental
ssh-socketpair-stderr rsync_io no Constructs the SSH child’s stderr over a socketpair(AF_UNIX, SOCK_STREAM) instead of an anonymous pipe, enabling epoll/kqueue-integrated async drain and a larger default buffer to absorb chatty remote shells (Linux recommended; Windows shim pending SSE-5). See docs/ssh-transport.md and docs/design/socketpair-stderr-channel.md (#2371, #2372). experimental
async-daemon daemon no Hybrid tokio accept loop dispatching sync workers via spawn_blocking (#1935). experimental
concurrent-sessions daemon no Shared dashmap session state for multi-session daemons. experimental
tracing core, engine, transfer, daemon no Structured tracing instrumentation for diagnostics. stable

Receiver memory tuning: SpillPolicy

The concurrent-delta receiver bounds its in-memory ReorderBuffer through a process-wide SpillPolicy knob. The default policy keeps everything in memory (byte-equivalent to prior releases). Opt in to disk-backed spill by setting OC_RSYNC_SPILL_THRESHOLD_BYTES (e.g. 64M) and, optionally, OC_RSYNC_SPILL_DIR to point at a fast scratch directory. CLI flags --spill-dir and --spill-threshold-bytes are planned for STN-11 and will shadow the env vars when present. Full surface (env vars, reclaim mode, granularity, compression, validation rules, defaults table) is in docs/design/spill-policy-public-api.md; the underlying buffer is documented in docs/design/reorderbuffer-spill-to-tempfile.md. Operator-facing tuning guidance is in docs/operator-migration-guide-vNEXT.md under Receiver spill tunability.

Choosing features for your build

# Default release build (recommended starting point).
cargo build --workspace --release

# Enable the async network paths end-to-end. async-ssh and async-daemon are
# crate-level features, so build the affected crates directly.
cargo build --release -p core --features async-ssh
cargo build --release -p daemon --features async-daemon

# Squeeze out maximum parallelism on a fat machine: keep the workspace
# defaults and add the experimental engine slab and vmsplice writer.
cargo build --workspace --release \
    --features "parallel async io_uring" \
  && cargo build --release -p engine --features thread-slab-pool \
  && cargo build --release -p fast_io --features vmsplice

Usage

Works like rsync - drop-in compatible:

# Local sync
oc-rsync -av ./source/ ./dest/

# Remote pull (SSH)
oc-rsync -av user@host:/remote/path/ ./local/

# Remote push (SSH)
oc-rsync -av ./local/ user@host:/remote/path/

# Daemon pull
oc-rsync -av rsync://host/module/ ./local/

# Daemon push
oc-rsync -av ./local/ rsync://host/module/

# Run as daemon
oc-rsync --daemon --config=/etc/oc-rsyncd/oc-rsyncd.conf

# Delta transfer with compression
oc-rsync -avz --compress-level=3 ./source/ ./dest/

# Checksum-based sync with deletion
oc-rsync -avc --delete ./source/ ./dest/

# Batch mode (record and replay)
oc-rsync -av --write-batch=changes ./source/ ./dest/
oc-rsync -av --read-batch=changes ./other-dest/

For supported options: oc-rsync --help

Development

Prerequisites

Build and test

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features --no-deps -D warnings
cargo nextest run --workspace --all-features

Project layout

src/bin/oc-rsync.rs     # Entry point
crates/cli/             # CLI flags, help, output formatting
crates/core/            # Orchestration facade, session management, config
crates/protocol/        # Wire protocol (v28-32), multiplex framing
crates/transfer/        # Generator, receiver, delta transfer pipeline
crates/engine/          # Local copy executor, sparse writes, temp-file commit
crates/daemon/          # Daemon mode, module access control, systemd
crates/checksums/       # Rolling and strong checksums (MD4, MD5, XXH3, SIMD)
crates/filters/         # Include/exclude pattern engine, .rsync-filter
crates/metadata/        # Permissions, uid/gid, mtime, ACLs, xattrs
crates/platform/        # Platform-specific unsafe code isolation (signals, chroot)
crates/rsync_io/        # SSH stdio, rsync:// TCP transport, handshake
crates/fast_io/         # Platform I/O (io_uring, copy_file_range, sendfile)
crates/compress/        # zstd, lz4, zlib compression codecs
crates/bandwidth/       # Bandwidth limiting and rate control
crates/signature/       # Signature layout and block-size calculations
crates/matching/        # Delta matching and block search
crates/flist/           # File list generation and traversal
crates/logging/         # Logging macros and verbosity control
crates/logging-sink/    # Message sink and output formatting
crates/batch/           # Batch file read/write support
crates/branding/        # Binary naming and version metadata
crates/embedding/       # Programmatic entry points for library usage
crates/apple-fs/        # macOS filesystem operations (clonefile, FSEvents)
crates/windows-gnu-eh/  # Windows GNU exception handling shims
crates/test-support/    # Shared test utilities (dev-dependency only)

See cargo doc --workspace --no-deps --open for API documentation.

Architecture

cli -> core -> engine, daemon, rsync_io, logging
                core -> protocol -> checksums, filters, compress, bandwidth -> metadata
                                                                            -> platform

Key crates: cli (Clap v4), core (orchestration facade), protocol (wire v28-32, multiplex framing), transfer (generator/receiver, delta pipeline), engine (local copy, sparse writes, buffer pool), checksums (MD4/MD5/XXH3, SIMD), daemon (TCP, auth, modules), platform (unsafe code isolation).

Security

All crates enforce #![deny(unsafe_code)]. Targeted #[allow(unsafe_code)] is permitted only in crates that wrap platform FFI or SIMD intrinsics:

Not vulnerable to known upstream rsync CVEs (CVE-2024-12084 through CVE-2024-12088, CVE-2024-12747). OS-level race conditions (TOCTOU) remain possible at filesystem boundaries.

For security issues, see SECURITY.md.

Contributing

  1. Fork and create a feature branch
  2. Run cargo fmt, cargo clippy, and cargo nextest run
  3. Open a PR describing behavioral changes and interop impact

License

GNU GPL v3.0 or later. See LICENSE.

Acknowledgements

Inspired by rsync by Andrew Tridgell and the Samba team. Thanks to Pieter for his heroic patience in enduring months of my rsync commentary. Thanks to Elad for his endless patience hearing rsync protocol commentary as I’m introduced to it.

This site is open source. Improve this page.