Koofr and Server Side Copying

What is the problem you are having with rclone?

I want to create snapshots of a local folder and store them on my Koofr using Sync. When I run the following

"D:\Downloads\rclone\rclone.exe" sync "D:\Test" KoofrEncrypted:\Backup_This_Week -v

the files are uploaded to the remote as expected. However if I then Sync the same folder to a different folder using

"D:\Downloads\rclone\rclone.exe" sync "D:\Test" KoofrEncrypted:\Backup_Last_Week -v

It looks like (if I'm understanding the log) that the files are still being uploaded from the local folder to the remote whereas I was expecting the Backup_Last_Week folder would be populated by a server side copy from Backup_This_Week as the files are already on the remote. The time taken for the two Syncs is approx the same, whereas a manual copy on the remote via a web browser took approx 20s.
The overview of storage systems https://rclone.org/overview/#optional-features indicates that Koofr supports copying so am I misunderstanding the log file or am I missing something?

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
os/kernel: 10.0.19045.2486 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.20.6
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Koofr

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

D:\Downloads\rclone\rclone.exe" sync "D:\Test" KoofrEncrypted:\Backup_This_Week -v
and
D:\Downloads\rclone\rclone.exe" sync "D:\Test" KoofrEncrypted:\Backup_Last_Week -v

The rclone config contents with secrets removed.

Paste config here

[Koofr]
type = koofr
provider = koofr
user =
password =
endpoint = "https://app.koofr.net"

[KoofrEncrypted]
type = crypt
remote = Koofr:/Encrypted
password =
password2 =

A log from the command with the -vv flag

Log for initial SYNC

2023/08/02 13:21:05 DEBUG : rclone: Version "v1.63.1" starting with parameters ["D:\\Downloads\\rclone\\rclone.exe" "sync" "D:\\Test" "KoofrEncrypted:\\Backup_This_Week" "-vv"]
2023/08/02 13:21:05 DEBUG : Creating backend with remote "D:\\Test"
2023/08/02 13:21:05 DEBUG : Using config file from "D:\\Downloads\\rclone\\rclone.conf"
2023/08/02 13:21:05 DEBUG : fs cache: renaming cache item "D:\\Test" to be canonical "//?/D:/Test"
2023/08/02 13:21:05 DEBUG : Creating backend with remote "KoofrEncrypted:\\Backup_This_Week"
2023/08/02 13:21:05 DEBUG : Creating backend with remote "Koofr:/Encrypted/706bi8417s1gcvjlf21r1he7qepupqdhio7lk0heub2i2gtfinjg"
2023/08/02 13:21:09 DEBUG : fs cache: renaming cache item "KoofrEncrypted:\\Backup_This_Week" to be canonical "KoofrEncrypted:/Backup_This_Week"
2023/08/02 13:21:11 DEBUG : EPG/7DEPGDATA.xml: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/New Text Document.txt: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/TVGUIDE.xml: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/blacklist: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/bouquets.radio: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/bouquets.tv: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/freeguide-0.11.1-with-xmltv-win32.exe: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml.chlold: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml.dat: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml.exe: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml.pdf: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml_14D.bat: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/mc2xml_7D.bat: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/nxtvepg-3.0.0-win-setup.exe: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/rytec.channels.xlsx: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/rytec.channels.xml: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/sky - xmltv channels.txt: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/xmltv.csv: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : EPG/xmltv.xml: Need to transfer - File not found at Destination
2023/08/02 13:21:11 DEBUG : Encrypted drive 'KoofrEncrypted:/Backup_This_Week': Waiting for checks to finish
2023/08/02 13:21:11 DEBUG : Encrypted drive 'KoofrEncrypted:/Backup_This_Week': Waiting for transfers to finish
2023/08/02 13:21:25 DEBUG : EPG/New Text Document.txt: md5 = e1f2dd172b3b9af95e373239ca8d7564 OK
2023/08/02 13:21:25 INFO  : EPG/New Text Document.txt: Copied (new)
2023/08/02 13:21:26 DEBUG : EPG/blacklist: md5 = e777ded933751546efa20ab17c5b2e0a OK
2023/08/02 13:21:26 INFO  : EPG/blacklist: Copied (new)
2023/08/02 13:21:32 DEBUG : EPG/bouquets.tv: md5 = 683dce1546d0fd2024664c51467c2794 OK
2023/08/02 13:21:32 DEBUG : EPG/bouquets.radio: md5 = 927fe585b590e0fda67b5e75b164d4ea OK
2023/08/02 13:21:32 INFO  : EPG/bouquets.tv: Copied (new)
2023/08/02 13:21:32 INFO  : EPG/bouquets.radio: Copied (new)
2023/08/02 13:21:38 DEBUG : EPG/mc2xml.chlold: md5 = 0d70963a9b61efd70bc38a66710633ce OK
2023/08/02 13:21:38 INFO  : EPG/mc2xml.chlold: Copied (new)
2023/08/02 13:21:43 DEBUG : EPG/mc2xml.dat: md5 = 7676e24275ea91fc92e25cd1f570242f OK
2023/08/02 13:21:43 INFO  : EPG/mc2xml.dat: Copied (new)
2023/08/02 13:21:53 DEBUG : EPG/mc2xml.exe: md5 = 9274111aceaa02f6d6c5d120000e0435 OK
2023/08/02 13:21:53 INFO  : EPG/mc2xml.exe: Copied (new)
2023/08/02 13:22:01 DEBUG : EPG/mc2xml.pdf: md5 = c6ac6a41db1fc4c42f97a995417171f1 OK
2023/08/02 13:22:01 INFO  : EPG/mc2xml.pdf: Copied (new)
2023/08/02 13:22:09 INFO  :
Transferred:       55.604 MiB / 125.496 MiB, 44%, 1.071 MiB/s, ETA 1m5s
Transferred:            8 / 19, 42%
Elapsed time:       1m3.8s
Transferring:
 *                             EPG/7DEPGDATA.xml: 40% /52.575Mi, 370.192Ki/s, 1m26s
 *                               EPG/TVGUIDE.xml: 46% /43.580Mi, 370.331Ki/s, 1m4s
 *     EPG/freeguide-0.11.1-with-xmltv-win32.exe: 69% /17.928Mi, 353.665Ki/s, 16s
 *                            EPG/mc2xml_14D.bat:136% /133, 25/s, -

2023/08/02 13:22:11 DEBUG : EPG/mc2xml_14D.bat: md5 = 6804bd8f2a6d0600439678fe4eb1e7fa OK
2023/08/02 13:22:11 INFO  : EPG/mc2xml_14D.bat: Copied (new)
2023/08/02 13:22:18 DEBUG : EPG/mc2xml_7D.bat: md5 = e082f05c5b325689c48421d57fed4397 OK
2023/08/02 13:22:18 INFO  : EPG/mc2xml_7D.bat: Copied (new)
2023/08/02 13:22:29 DEBUG : EPG/freeguide-0.11.1-with-xmltv-win32.exe: md5 = 080f4fe823ade7b975ef913f64c9488c OK
2023/08/02 13:22:29 INFO  : EPG/freeguide-0.11.1-with-xmltv-win32.exe: Copied (new)
2023/08/02 13:22:32 DEBUG : EPG/nxtvepg-3.0.0-win-setup.exe: md5 = 99c7ef55529bf7e047b01fc283c06696 OK
2023/08/02 13:22:32 INFO  : EPG/nxtvepg-3.0.0-win-setup.exe: Copied (new)
2023/08/02 13:22:36 DEBUG : EPG/rytec.channels.xlsx: md5 = 97c163a72a616b57dcb9b05cdb8ec56a OK
2023/08/02 13:22:37 INFO  : EPG/rytec.channels.xlsx: Copied (new)
2023/08/02 13:22:43 DEBUG : EPG/rytec.channels.xml: md5 = 4ae449e3cfe412c35ccbda196d842a23 OK
2023/08/02 13:22:43 INFO  : EPG/rytec.channels.xml: Copied (new)
2023/08/02 13:22:44 DEBUG : EPG/sky - xmltv channels.txt: md5 = 921b0eaa5f19f848654634b5113e3f66 OK
2023/08/02 13:22:44 INFO  : EPG/sky - xmltv channels.txt: Copied (new)
2023/08/02 13:22:50 DEBUG : EPG/xmltv.xml: md5 = e9114812a3ba8a46313b37a2a47b5dbf OK
2023/08/02 13:22:50 INFO  : EPG/xmltv.xml: Copied (new)
2023/08/02 13:22:54 DEBUG : EPG/xmltv.csv: md5 = a1e8e0da07934598c86fab0ddafd9916 OK
2023/08/02 13:22:54 INFO  : EPG/xmltv.csv: Copied (new)
2023/08/02 13:23:09 INFO  :
Transferred:      122.014 MiB / 125.504 MiB, 97%, 1.104 MiB/s, ETA 3s
Transferred:           17 / 19, 89%
Elapsed time:       2m3.9s
Transferring:
 *                             EPG/7DEPGDATA.xml: 93% /52.575Mi, 630.677Ki/s, 5s
 *                               EPG/TVGUIDE.xml:100% /43.580Mi, 388.374Ki/s, -

2023/08/02 13:23:10 DEBUG : EPG/TVGUIDE.xml: md5 = e97f8aa18e3a3013b4dc704c59a2ac23 OK
2023/08/02 13:23:10 INFO  : EPG/TVGUIDE.xml: Copied (new)
2023/08/02 13:23:17 DEBUG : EPG/7DEPGDATA.xml: md5 = 5d13e9ed86de669913e7596f1d701390 OK
2023/08/02 13:23:17 INFO  : EPG/7DEPGDATA.xml: Copied (new)
2023/08/02 13:23:17 DEBUG : Waiting for deletions to finish
2023/08/02 13:23:17 INFO  :
Transferred:      125.527 MiB / 125.527 MiB, 100%, 893.676 KiB/s, ETA 0s
Transferred:           19 / 19, 100%
Elapsed time:      2m11.5s

Log from 2nd SYNC

2023/08/02 13:30:08 DEBUG : rclone: Version "v1.63.1" starting with parameters ["D:\\Downloads\\rclone\\rclone.exe" "sync" "D:\\Test" "KoofrEncrypted:\\Backup_Next_Week" "-vv"]
2023/08/02 13:30:08 DEBUG : Creating backend with remote "D:\\Test"
2023/08/02 13:30:08 DEBUG : Using config file from "D:\\Downloads\\rclone\\rclone.conf"
2023/08/02 13:30:08 DEBUG : fs cache: renaming cache item "D:\\Test" to be canonical "//?/D:/Test"
2023/08/02 13:30:08 DEBUG : Creating backend with remote "KoofrEncrypted:\\Backup_Next_Week"
2023/08/02 13:30:08 DEBUG : Creating backend with remote "Koofr:/Encrypted/mqskkfvejhfu8jn4hepook4e9p4ontka1p68m140k9l948gjlt40"
2023/08/02 13:30:12 DEBUG : fs cache: renaming cache item "KoofrEncrypted:\\Backup_Next_Week" to be canonical "KoofrEncrypted:/Backup_Next_Week"
2023/08/02 13:30:14 DEBUG : EPG/7DEPGDATA.xml: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/New Text Document.txt: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/TVGUIDE.xml: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/blacklist: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/bouquets.radio: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/bouquets.tv: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/freeguide-0.11.1-with-xmltv-win32.exe: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml.chlold: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml.dat: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml.exe: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml.pdf: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml_14D.bat: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/mc2xml_7D.bat: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/nxtvepg-3.0.0-win-setup.exe: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/rytec.channels.xlsx: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/rytec.channels.xml: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/sky - xmltv channels.txt: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/xmltv.csv: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : EPG/xmltv.xml: Need to transfer - File not found at Destination
2023/08/02 13:30:14 DEBUG : Encrypted drive 'KoofrEncrypted:/Backup_Next_Week': Waiting for checks to finish
2023/08/02 13:30:14 DEBUG : Encrypted drive 'KoofrEncrypted:/Backup_Next_Week': Waiting for transfers to finish
2023/08/02 13:30:28 DEBUG : EPG/New Text Document.txt: md5 = ec4deff129af299e25f4ffa21a3fdcc7 OK
2023/08/02 13:30:28 INFO  : EPG/New Text Document.txt: Copied (new)
2023/08/02 13:30:29 DEBUG : EPG/blacklist: md5 = 1e07fa1adb66692f46bb57006bc1a6f0 OK
2023/08/02 13:30:29 INFO  : EPG/blacklist: Copied (new)
2023/08/02 13:30:33 DEBUG : EPG/bouquets.radio: md5 = 0f946f0dc9ec8810f5a82ae03121ad59 OK
2023/08/02 13:30:33 INFO  : EPG/bouquets.radio: Copied (new)
2023/08/02 13:30:33 DEBUG : EPG/bouquets.tv: md5 = 4b3c321b750941060e89c13ffe16d2cb OK
2023/08/02 13:30:33 INFO  : EPG/bouquets.tv: Copied (new)
2023/08/02 13:30:38 DEBUG : EPG/mc2xml.chlold: md5 = e8ae6f6ed4c2ff1f3bd37c32c721cac9 OK
2023/08/02 13:30:38 INFO  : EPG/mc2xml.chlold: Copied (new)
2023/08/02 13:30:44 DEBUG : EPG/mc2xml.dat: md5 = 52954dfe85192c185ed918237e480483 OK
2023/08/02 13:30:44 INFO  : EPG/mc2xml.dat: Copied (new)
2023/08/02 13:30:53 DEBUG : EPG/mc2xml.exe: md5 = 3ce347637f1b0834da490577801b022a OK
2023/08/02 13:30:53 INFO  : EPG/mc2xml.exe: Copied (new)
2023/08/02 13:30:58 DEBUG : EPG/mc2xml.pdf: md5 = 77510dc04e0c80872113921528b5a70d OK
2023/08/02 13:30:58 INFO  : EPG/mc2xml.pdf: Copied (new)
2023/08/02 13:31:03 DEBUG : EPG/mc2xml_14D.bat: md5 = af7227d9dcd05825014d3d69f506fc51 OK
2023/08/02 13:31:03 INFO  : EPG/mc2xml_14D.bat: Copied (new)
2023/08/02 13:31:08 DEBUG : EPG/mc2xml_7D.bat: md5 = 2a1f80a00b6646fba549c1937389ece3 OK
2023/08/02 13:31:08 INFO  : EPG/mc2xml_7D.bat: Copied (new)
2023/08/02 13:31:12 INFO  :
Transferred:       55.542 MiB / 125.497 MiB, 44%, 1.072 MiB/s, ETA 1m5s
Transferred:           10 / 19, 53%
Elapsed time:       1m4.1s
Transferring:
 *                             EPG/7DEPGDATA.xml: 38% /52.575Mi, 350.275Ki/s, 1m34s
 *                               EPG/TVGUIDE.xml: 44% /43.580Mi, 353.757Ki/s, 1m9s
 *     EPG/freeguide-0.11.1-with-xmltv-win32.exe: 73% /17.928Mi, 345.677Ki/s, 14s
 *               EPG/nxtvepg-3.0.0-win-setup.exe: 27% /3.352Mi, 216.157Ki/s, 11s

2023/08/02 13:31:23 DEBUG : EPG/nxtvepg-3.0.0-win-setup.exe: md5 = a255445563ff1473f52b065d97bb6677 OK
2023/08/02 13:31:23 INFO  : EPG/nxtvepg-3.0.0-win-setup.exe: Copied (new)
2023/08/02 13:31:32 DEBUG : EPG/freeguide-0.11.1-with-xmltv-win32.exe: md5 = cbdccde12994839dcf36297f252b210e OK
2023/08/02 13:31:32 INFO  : EPG/freeguide-0.11.1-with-xmltv-win32.exe: Copied (new)
2023/08/02 13:31:33 DEBUG : EPG/rytec.channels.xlsx: md5 = bdc9b5a376a4fa1c0e9cfcba781b7dea OK
2023/08/02 13:31:33 INFO  : EPG/rytec.channels.xlsx: Copied (new)
2023/08/02 13:31:38 DEBUG : EPG/sky - xmltv channels.txt: md5 = 75f30dff16c79ef98c82ab7a9d11faf9 OK
2023/08/02 13:31:38 INFO  : EPG/sky - xmltv channels.txt: Copied (new)
2023/08/02 13:31:41 DEBUG : EPG/rytec.channels.xml: md5 = 2960e1cbd4e0a9b659b4c512c718a172 OK
2023/08/02 13:31:41 INFO  : EPG/rytec.channels.xml: Copied (new)
2023/08/02 13:31:49 DEBUG : EPG/xmltv.csv: md5 = b4e41b3b32403ad118e0c7f50e2b9031 OK
2023/08/02 13:31:49 INFO  : EPG/xmltv.csv: Copied (new)
2023/08/02 13:31:50 DEBUG : EPG/xmltv.xml: md5 = b38d7266a69aafa972b0d2f7adeaa3db OK
2023/08/02 13:31:50 INFO  : EPG/xmltv.xml: Copied (new)
2023/08/02 13:32:09 DEBUG : EPG/TVGUIDE.xml: md5 = 19044c6284bcb3cf4c069e1963eaa869 OK
2023/08/02 13:32:09 INFO  : EPG/TVGUIDE.xml: Copied (new)
2023/08/02 13:32:12 INFO  :
Transferred:      121.951 MiB / 125.514 MiB, 97%, 1.105 MiB/s, ETA 3s
Transferred:           18 / 19, 95%
Elapsed time:       2m4.1s
Transferring:
 *                             EPG/7DEPGDATA.xml: 93% /52.575Mi, 640.167Ki/s, 5s

2023/08/02 13:32:17 DEBUG : EPG/7DEPGDATA.xml: md5 = 6b19380999d8476fcdb189894dc02de2 OK
2023/08/02 13:32:17 INFO  : EPG/7DEPGDATA.xml: Copied (new)
2023/08/02 13:32:17 DEBUG : Waiting for deletions to finish
2023/08/02 13:32:17 INFO  :
Transferred:      125.527 MiB / 125.527 MiB, 100%, 1003.908 KiB/s, ETA 0s
Transferred:           19 / 19, 100%
Elapsed time:       2m9.4s

2023/08/02 13:32:17 DEBUG : 4 go routines active

the way you have it now, two separate commands, so you will have two copies of the files in koofr

you might want something like this, which should use server-side
rclone sync D:\Test KoofrEncrypted:\Backup_This_Week --backup-dir=KoofrEncrypted:\Backup_Last_Week --dry-run

Yes it does but you do not copy anything but sync from local to remote.

rclone copy remote:folder1 remote:folder2

would happen server side.

but when you sync then sync is not checking all your remote for already existing files - it can do it though for your sync destination folder.

So workaround I would use (unless you decide to use much space efficient @asdffdsa suggestion):

  1. populate New_Backup with Old_Backup - this should happen server side

rclone copy KoofrEncrypted:Old_Backup KoofrEncrypted:New_Backup

  1. sync with track-renames flag

rclone sync "D:\Test" KoofrEncrypted:New_Backup --track-renames

This combo will spare you from uploading anything what is already in the cloud.

And in general for this type of versioning backup it is much better to use specialised backup tools like restic (which actually uses rclone to transfer files)

Tx for the quick reply. I tried your suggestion and it did indeed do a server side copy - I can see now the flaw in my logic!

Tx for the quick quick reply, I'll have a look at restic. Other than track-renames are there any other flags you would suggest to increase the speed of the sync - if all goes to plan the local source will contain 20K-30K files and as many of them will be small I can see the first run taking a long time.

For sync obvious candidates would be:

--checkers int   Number of checkers to run in parallel (default 8)
--transfers int    Number of file transfers to run in parallel (default 4)

flags. You can experiment with increasing default values.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.