File Sync & Backup Tools (FreeFileSync TrueNAS SyncThing Synolog QNap NAS)

NextCloud vs. Seafile vs. SyncThing #

• Cons (issues) of OwnCloud / Nextcloud: uploading big files in one time using HTTP is not a good choice; each WebDAV connection per file which is a bad practice for small files (10GB files requies 10 hours); 255 bytes per filename due to Linux (usually enought, better than Win).
• Pros of owncloud/nextcloud: it stores files in files.
• Cons (issues) of Seafile: stores in its own format, canNOT access files directly in OS.
• Pros of Seafile: fast; pro. version is free for 3 users including full-text document search (elasticsearch-based).
• OwnCloud supports diff sync, which is still not in nextcloud.

more about MAX_PATH: doc, nextcloud help.
ssdnodes (bak1: 4systems.ru, bak2: ever)

SyncThing on Synology #

con: case sensitivity conflicts with windows, will not know which file wins.

use a docker container #

1. install the docker package;
2. picka a source image by seaching in “registry” (OFFICIAL distribution is recommended);
3. modify settings & advanced settings as seen in this ref, then lunch.
   1. auto-restart; shortcut on desktop > web > “http://<nas_ip>:8384”;
   2. volumes, e.g.: /config (for linuxserver.io), /var/syncthing (for official), /<other_data_folders>
   3. port-forwarding; PUID, PGID.

WARN:

1. mount a Synology folder to “/config” (for linuxserver.io) or “/var/syncthing” (for official) as indicated by “Environment” -> $HOME, which is the configuration folder (also includes the default “Sync” folder);
2. if only one Synology user will use SyncThing, please set PUID and PGID env. variables using that Synology user’s puid & puid, so root’s high priviledge is not needed. (note: an irrelated UID and GID is used by default and will cause permission problems).
3. recommended: add “(?d)” at the beginning of each ignore patterns/rules, so the ignored itmes can be deleted if ignored things are the only remaining things in a folder.

ref.original

Tip:

1. SyncThing file change watcher may complain about inotify limit. A solution to change it is here, but note that 1 watcher uses 1kB RAM, thus the Syncthing recommended 204800 uses 220M RAM (110M in 32bit sys);
2. SyncThing (docker) uses about 0.8~1.2 GB RAM (excluding reserved virtual RAM for caching);
3. timing a scan: a 800GB sized folder with 100k subfolders and 800k files take 8 minutes to scan in a laptop with i7-11800H @ 2.30GHz and a NVMe SSD.

use a community package (only for DSM < v7.) #

First, add SynoCommunity to package source.
Second, install SyncThing in package centre.

SyncThing in TrueNAS (FreeNAS) Jail #

(the reason to use a container: SyncThing cannot support Synology DSM v7)

WARN: FreeBSD does NOT provide an easy way to limit jail resources (except legancy versions). As SyncThing may use a lot more than usual and cause TrueNAS (GUI & CLI) to be extremely slow (not due to CPU), users may need to limit the resources using FreeBSD CLI.

Install FreeNAS/TrueNAS (dual boot / mirrored boot is suggested)
Install & Config TrueNAS SyncThing plugin

ref

Tip: enabling pkg may be needed.

sed -i.bak 's/enabled: yes/enabled: no/' /usr/local/etc/pkg/repos/local.conf && \
sed -i.bak 's/enabled: no/enabled: yes/' /usr/local/etc/pkg/repos/FreeBSD.conf && \
pkg update &&\
pkg install -y cmdwatch

Note 1: -i: in-place replacing with backup file extention;
Note 2: For CN users (tsinghua does not have a FreeBSD mirror):

cat <<EOF > /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: {
  url: "pkg+http://mirrors.ustc.edu.cn/freebsd-pkg/${ABI}/quarterly",
}
EOF

Tip: pkg -vv | grep ABI | head -1 | tr -d ' '

SyncThing FAQ #

Q: Where is the GUI?
A: localhost:8384

Q: What is the port number?
A: if set to 80/443, it will automatically change to 22000.

Q: What is a “folder marker”?
A: It is a hidden folder named “.stfolder” in the shared folder root which is created before the first sync. If it is missing, it could mean that the folder has been moved etc. It can be caused by remounting/ mount-point changing.

Issue: Win GUI SyncTrayzor ID conflicts with CLI
Not exactly. The windows GUI version (SyncTrayzor) includes a CLI version which is intended to be lunched by the GUI (SyncTrayzor). It turns out that the origianl CLI version and the SyncTrayzor’s CLI version share the same device ID (and config), but the GUI (SyncTrayzor) uses another ID (and config).

Issue: Stuck in “Syncing” or “Out of Sync” even if the files are synced (i.e. working, but shows not working).
Solution: none, a bug. ref: e.g., github issue.
It is claimed to be fixed in 1.18.0, but I still got it in 1.18.2.

Issue: “unexpected device ID, expected … got …” in Remote Devices.
Solution: restart the remote peer (e.g. the jail in NAS).

FreeFileSync #

Together with is RealTimeSync, this open source solution works well.

See: RealTimeSync

Pro: can handle long file/folder names; support SMB/mounted net. drive/SFTP/FTP/Google Drive; free; donate any for parallel and portable version.
Con: file change monitorning is limited to OS notification (Win10 sends notification with an interval about 15s).

UGreen (绿联) NAS (nonprofessional) #

DH2100 provides remote access w/o speed limit (2022), but does not support Docker yet.

DISM++ #

It contains a lot of great tools and useful & handy system modification functions, including system backup.

Con: it can be used for system backup & reinstall, but not for real-time tasks such as file syncing.

Note: DISM is CLI tool developed by MicroSoft, but DISM++ is just using the name.

Iperius Backup (Free Version) #

Paid version is needed to sync deletion.

Pros: can handle long path and filenames without any limit; support multiple destinations; support smb path and/or mounted partions from NAS as dest.; export/import backup configs; frequency (interval) can be 1 min.

Cons: versioning is not suitable for manual restore, so versioning should be done on NAS; canNOT avoid creating the root folder in the destination (bad news for sync).

Explaination of backup options for each destination:
Op1 - “Always full backup”: always copy all files;
Op2 - “First time full, then only new or changed files 保留一个完整备份然后使用新文件或修改后的文件” (actually incremental/delta backups/transfers indeed, recommended and default): the 1st time will copy all files, later backups will (try to?) make copies inside the destination, only changes (including deletion) will be transferred, the result is always a full copy but only changed content is transferred; WARN: for “including deletion” (to delete dest. files if source deleted), multi versioning should be used (number of copies >= 2) and only available for newly created backups i.e. if number of versions reaches the MAX (e.g. 2), the oldest backup will be re-used so the previously deleted files are back … so, paid version is needed …
Op3 - “Full backup + incremental copies 保留一个完整备份和几个增量副本”: the 1st time will copy all files, later backups will compare with the previous backup and copy the new files and updated content (excluding deletion) and only those files/dirs are saved in the folder, which saves spaces.
Op4 - “Full backup + differential copies 保留一个完整备份和几个差异副本”: similar to op3, but the comparison is to compre current status with the last full backup.

Tip: make sure to clear the desination folder if backup type is changed, otherwise could cause some issues.

ref: official blog

Other recommended tips:

• in the source options, uncheck “rebuild full path” (as we just want to “sync”).
• in the desination options, uncheck “use backup task name to create folders” (same reason as above).
• in the backup task options > advanced options > error and warning, check “no warning even if no files are copies” (no copy if no change); > files and folders, check “include hidden files” and “include sys. files”.

NextCloud #

(tested on TrueNAS + win10)

See also: ref NextCloud vs. OwnCloud (bak)

cons: max_path 255 due to php limit thus canNOT be avoid; need to reboot windows to install and uninstall the client; no delta sync.

pros: many plugins, including online office tools.

NextCloud on truenas tip: (to find php confi of trusted domain)

find / -name "config.php"
/mnt/<pool_name>/iocage/jails/<nextcloud_jail>/root/usr/local/www/nextcloud/config/config.php

Note: edit using nano, vim canNOT work inside the webpage.

WD Backup #

cons: long path/filename issue: it reports cannot backing up specific files, but indeeded backed up some of them while no clue how the software could pick very long names (260+), but just ignored the middle-long (~255) names.

pros: very nice and smooth user experience for beginners.

Note: WD officially says it is replaced by AcronisTrueImage for WD, but it failed to install. Also, Acronis is very big (~1GB installer).

Synology #

sync folder #

cd /volume1/homes/<user_name>/Drive/

make S.M.A.R.T. smart #

Storage Manager > HDD/SSD > Test Scheduler:
edit the default “Auto S.M.A.R.T. Test” and set “S.M.A.R.T. Test Type” to “extended”.

snapshot replication app (file version control) #

Storage Manager > Volume > Action > Configure:
set “Record file access time frequency” to “never”.

Next, open “Snapshot Replication” > Snapshots:
select the folder to be time-machined (e.g. homes) and open “settings”:
set settings (sunny recommended):

• Schedule > every 2 hours or daily snapshot at early morning
• Retention > advanced retention policy: 12, 6, 3, 3, 0.
  • choose when to delete snapshots: e.g. several hours before taking
• Advanced > check “make snapshot visible”

user home (for drive client) #

Control Panel (with advanced) > User > Advanced > User Home > Enable user home service.

ups #

Control Panel (with advanced) > Hardware & Power > UPS

nfs #

//TODO

tips #

enable block expiration: #

Control Panel (with advanced) > Security -> Account -> Enable block expiration (e.g. x days).
Otherwise the file /temp/var/lib/synosmartblock/db.sqlite-wal may grow too big and eat all space which causes ““You cannot login to the system because the disk space is full.” [ref]

recycle emptying schedule: #

Control Panel > Shared Folder > Right click > Action > “Create recycle bin emptying schedule”.

pro #

global indexed search (Universal Search) includes Drives by default, can add other folders.

problem/cons #

CanNOT handle pathes/filenames longer than 255 (the same as OneDrive among others).

expand capacity by replace a disk #

(tested under the default SHR config)

Storage Manager > HDD > select a disk > Action > Deactivate.

Plug out the disk (alarm will occur), and plug in a new one.

Storage Manager > Storage pool > select the pool > Action > Repair.

It may take 2 days to restore a 10T drive.

expand capacity by inserting new disks #

Insert disks and add to an existing volume or create a new volume.
Note that expanding will check parity consistancy, and the speed is influenced a lot by “RAID resync speed”, max speed ~2T/day.

migrate to a new synology #

Many migration pathes are available ref (bak).

direct disk moving: #

Before powering off the old:

• Check the compatibility;
• Make sure both stations are in the supported models list;
• Update the old station & write down the DSM version (i did not upgrade the major version as it is kinda beta).
• Backup settings.

After plugging in the new:

• Boot without disks so the system can boot (also, now it is possible to make IP static).
• Go to the IP in a desktop browser and it asks to insert disks and “reconnect”.
• Insert disks with the same slot order as on the old station.
• “Reconnect”
• “Manual Installation” when asked.
• Use the same DSM version (update version, if any) to install.
• It takes seconds to install (cuz already installed the same version).
• It will detect the old system and ask if to migrate.
• Select migrate (keep settings by default).
• “Manual Installation” again when asked.
• It takes 10 minutes to install.
• Create a new user (i created the same, but whatever, will be restored).
• Login and install needed packages.
• Restore settings, select “overwrite when conflicts”, re-login.
• Config other settings that not backed-up, e.g. UPS

QNap #

sync folder #

cd /share/homes/<user_name>/.Qsync

pro: can handle long path/filenames.

problem/con:

1. Mobile to NAS connection via Internet (relay?) is extremely slow (slow enough that always trying to avoid), while both Synology and UGreen are fast;
2. global indexed search is very expensive and is an annual subscription (Synology is free), while non-indexed global search is disgusting.

Sync Between Synology and QNap #

[ref zhihu, bak]

RAID on Ubuntu #

Install OS & RAID at the same time (note: no nested 1+0 or 5+0 ).
Using mdadm after installing OS.
5295c230c0b99)]

RAID on Ubuntu #

Install OS & RAID at the same time (note: no nested 1+0 or 5+0 ).
Using mdadm after installing OS.