I run rclone copy as per below and it stopped halfway through at 20+Gb (can't recall exactly) out of 44 Gb; the rclone process was nowhere to be seen. I executed the same command relying on the fact that it shouldn't copy unchanged files.
That command has now finished properly, but the destination folder is now 63 Gb instead of 44 Gb. Judging by the numbers it could be a failure ; but when I look into the folder on pcloud, I can't see any! not sure where the fault here lies, but rclone itself shows the discrepancy (see below)
What is your rclone version (output from rclone version)
rclone v1.52.2
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 18 LTS amd64 (headless server, logged into through SSH)
Which cloud storage system are you using? (eg Google Drive)
OneDrive to pcloud
The command you were trying to run (and the test of size on original and destination folders)
[1]+ Done rclone copy unimelbod:Imágenes/ mypc:Images/
ubuntu@procesador:/data$ rclone size mypc:Images
Total objects: 24858
Total size: 63.160 GBytes (67817945602 Bytes)
ubuntu@procesador:/data$ rclone size unimelbod:Imágenes
Total objects: 14454
Total size: 44.240 GBytes (47502084172 Bytes)
The rclone config contents with secrets removed.
[unimelbod]
type = onedrive
token = {"access_token":"XXXXX$drive_id = XXXX
drive_type = business
[mypc]
type = pcloud
token = {"access_token":"XXXX","token_type":"bearer","expiry":"0001-$
@asdffdsa, yes, the point is that there were files in the destination; but they were identical to a portion of the ones in the original folder; the destination folder was created by the first copy command that failed halfway through and I simply restarted it.
@Animosity
2020/07/23 14:33:35 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "-vv" "copy" "unimelbod:Imá$2020/07/23 14:33:35 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/07/23 14:33:35 DEBUG : fs cache: renaming cache item "unimelbod:Imágenes/" to be canonical "unimelbod:Imágenes"
2020/07/23 14:33:37 DEBUG : fs cache: renaming cache item "mypc:Imagenes/" to be canonical "mypc:Imagenes"
2020/07/23 14:33:43 INFO : file1.jpg: Copied (new)
2020/07/23 14:33:44 INFO : 20151125_093802.jpg: Copied (new)
2020/07/23 14:33:44 INFO : file3.zip: Copied (new)
(snip)
Transferred: 15.068G / 44.237 GBytes, 34%, 1.934 MBytes/s, ETA 4h17m25s
Transferred: 8719 / 14454, 60%
Elapsed time: 2h12m58.9s
Transferring:
* Álbum de cámara/20151123_113000.jpg:100% /2.239M, 638.030k/s, 0s
* Álbum de cámara/20151123_113130.jpg: 93% /2.964M, 494.031k/s, 0s
* Álbum de cámara/20151123_113145.jpg: 13% /2.309M, 91.981k/s, 22s
* Álbum de cámara/20151123_113227.jpg: transferring
2020/07/23 16:46:38 INFO : Álbum de cámara/20151123_113130.jpg: Copied (new)
2020/07/23 16:46:39 INFO : Álbum de cámara/20151123_113145.jpg: Copied (new)
2020/07/23 16:46:41 INFO : Álbum de cámara/20151123_113227.jpg: Copied (new)
2020/07/23 16:46:45 INFO : Álbum de cámara/20151123_114002.jpg: Copied (new)
2020/07/23 16:46:47 INFO : Álbum de cámara/20151123_113501.jpg: Copied (new)
2020/07/23 16:46:49 INFO : Álbum de cámara/20151123_114013.jpg: Copied (new)
2020/07/23 16:46:51 INFO : Álbum de cámara/20151123_114047.jpg: Copied (new)
2020/07/23 16:46:52 INFO : Álbum de cámara/20151123_114235.jpg: Copied (new)
############## rclone stopped without explanation here (just like the first time) and I restarted the transfer #########################
2020/07/23 22:43:43 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "-vv" "copy" "unimelbod:Imá$2020/07/23 22:43:43 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/07/23 22:43:43 DEBUG : fs cache: renaming cache item "unimelbod:Imágenes/" to be canonical "unimelbod:Imágenes"
2020/07/23 22:43:44 DEBUG : fs cache: renaming cache item "mypc:Imagenes/" to be canonical "mypc:Imagenes"
2020/07/23 22:43:45 DEBUG : llhkadf.mp4: Size and modification time the same (differ by 0s, within $2020/07/23 22:43:45 DEBUG : llhkadf.mp4: Unchanged skipping
2020/07/23 22:43:45 DEBUG : 20151125_093802.jpg: Size and modification time the same (differ by 0s, within tolerance$2020/07/23 22:43:45 DEBUG : 20151125_093802.jpg: Unchanged skipping
(snip)
It's looking like it's doing the right thing; I'll report on the final outcome in a few hours. I thought I'd catch the earlier log messages now before the log file became unwieldy.
thank you @asdffdsa for the prompt reply! The files shouldn't be different from the source because they could only have come from the source... If they are, again, there must be something NQR about rclone copy.
At this stage, I was simply interested in testing whether rclone copy really ignores identical files; whether pcloud itself was playing up or what.. (I thought maybe people would have experience with this).
Transferring a different/smaller folder (and without the process dying) I got the same sizes, so I'm guessing it's not pcloud, although I'm still mesmerised by the fact that I can't seem to find any duplicates in the 63Gb folder copied from the 44 Gb one.. In fact, there are a couple of missing folders...
I didn't know about the --dry-run flag, how would it differ from me just copying to a temp destination and deleting if something doesn't seem right (which essentially what I'm doing)? Does it take much less time?
Sorry I'm new to the world of large cross-cloud-service file transfers, so I have a lot of questions...xD
--dry-run is a good flag to know about, but if you are using a temp dest folder, that is good.
maybe pcloud does versioning, and that is what the extra folder size is about.
rclone sync with --dry-run will let you know the differences between the source and dest.
there is a beta of rclone that creates a file with a list of files that are missing on the source and missing on the dest but i cannot find it.