What is the problem you are having with rclone?
I'm copying a large amount of data from a QNAP NAS to a Nextcloud instance backed by a remote object storage accessed via FTP)
The path is QNAP -> Nextcloud -> FTP.
Of course you could tell I can go directly from QNAP to FTP, but this backup is something I need to run regularly so I'd like to use the definitive procedure form the beginning.
Using the command below for copying, I noticed several runs (disconnections, power loss) generate a lot of versions on Nextcloud. I wasn't able to investigate on the NC side, but it looks like running multiple times the same command generate a reupload of some files. I know it's an unhelpful sentence when asking for support but I wasn't able to identify the cause so far.
I remember in the past I tried to rsync/robocopy from a windows host to Nextcloud (mounted with raidrive) and I failed miserably because NC via webdav wasn't storing file modification time. I checked some files transferred now with rclone and in Nextcloud I see the correct (original) timestamp of the file, so it doesn't seem the case.
I was wondering if I'm using the correct params for avoiding duplication of data on the storage and duplication of transferred data.
I'm now trying with --size-only and it seems to behave better, that means transferring only files which haven't been uploaded before. But I fear this test can be weak as it wouldn't detect partial or incorrect transfers.
Once I've copied most of the data (2TB) with size only, I'd like to regularly run with -c to check by size and checksum. Will this increase the traffic between Nextcloud and the FTP backend? Being this an object storage I will be charged if NC does a lot of download operations on the files.
Thanks.
What is your rclone version (output from rclone version)
rclone installed via QNAP club repository (link).
rclone v1.56.1
- os/version: qts 4.4.1 (20191206) (64 bit)
- os/kernel: 4.14.24-qnap (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.8
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Nextcloud 22.2.0
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy /share/Video/record_nvr/ nextcloud:/Video/QNAP/ -v --exclude ".nvr/" --exclude ".@__thumb/" --bwlimit 3M -P --no-check-certificate
Despite the excludes it's still trying to copy some dirs, like:
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/adefault2021-04-15 13-00-00~13-05-00.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/a1002021-04-15 13-00-00~13-05-00.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/adefault2021-04-15 13-05-00~13-10-00.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/a1002021-04-15 13-05-00~13-10-00.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/adefault2021-04-15 13-15-00~13-19-56.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/a1002021-04-15 13-15-00~13-19-56.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/adefault2021-04-15 13-10-00~13-15-00.avi: Can't follow symlink without -L/--copy-links
2021-11-03 11:16:54 NOTICE: channel1/2021-04-15/13D/.@__thumb/a1002021-04-15 13-10-00~13-15-00.avi: Can't follow symlink without -L/--copy-links
The rclone config contents with secrets removed.
[nextcloud]
type = webdav
url = https://files-video.domain.it/remote.php/webdav/
vendor = nextcloud
user = sync
pass = 12345
