Skip to main content

Using rsync for Data Migration on TrueNAS

rsync is a powerful utility for synchronizing files and directories, ideal for data migration on TrueNAS. It efficiently copies data, only transferring changes and offering robust options for integrity checks.

Why rsync is superior to cp for large transfers​

While cp can copy files, rsync offers significant advantages when dealing with many files or large files:

  • Delta Transfer (Differential Copying): rsync only transfers the differences between source and destination files after the initial copy. If you interrupt a transfer or need to update an existing destination, rsync will only send the changed blocks, making subsequent transfers much faster than cp, which always copies the entire file.
  • Resume Capability: Due to its delta transfer mechanism, if an rsync transfer is interrupted, you can restart the command, and it will pick up largely where it left off, avoiding redundant work. cp lacks this capability.
  • Robust Error Handling: rsync is designed for network transfers and includes more robust error handling and verification, making it more reliable for critical data moves.
  • Integrity Verification: Options like --checksum provide a strong guarantee that the copied data is identical to the source, a feature not inherently available with cp.

How to use rsync on TrueNAS​

You can execute rsync commands directly via the TrueNAS shell. This allows you to precisely control data transfer between your storage devices.

tip

The web shell in TrueNas sucks for this task because you'll most likely get auto-logged out after n period of time. I recommend using an SSH connection from powershell/terminal on your main PC, and also prevent your main PC from sleeping to prevent a disconnect.

Example Command Breakdown​

Consider the following rsync command for migrating data:

rsync -avh --info=progress2 --no-compress --checksum "/from_dir" "/to_dir"

Let's break down the options:

  • -a (archive mode): This is a combination of several options (-rlptgoD). It ensures recursive copying, preserves symbolic links, permissions, modification times, group, owner, and device files. Essential for maintaining data integrity during migration.
  • -v (verbose): Provides detailed output, showing which files are being transferred and other progress information.
  • -h (human-readable): Displays numbers in a human-readable format (e.g., "1G" instead of "1073741824").
  • --info=progress2: Shows overall progress during the transfer, including percentage complete and estimated time remaining.
  • --no-compress: Disables data compression. Since you're likely transferring data between local drives, compression is usually unnecessary and can even slow down the process.
  • --checksum: Forces rsync to verify file integrity using checksums rather than just relying on file size and modification times. This is crucial for ensuring every bit of data is copied accurately, especially when moving important data.
  • "/from_dir": This is the source directory. The trailing slash is important as it tells rsync to copy the contents of the from_dir directory, not the directory itself.
  • "/to_dir": This is the destination directory.

Other Helpful rsync Flags​

  • --remove-source-files: Deletes the source files after a successful transfer. Use with extreme caution.
  • --delete: Deletes extraneous files from the destination directory that are not present in the source directory. This makes the destination an exact mirror of the source. Use with caution.
  • --dry-run or -n: Performs a trial run without making any changes. This is invaluable for verifying your rsync command will do what you expect before actually executing it.
  • --exclude='PATTERN': Excludes files or directories matching the specified pattern. Useful for skipping temporary files or specific subdirectories.

This command provides a robust and verifiable method for migrating data on your TrueNAS system.