Rclone copy to pcloud yields larger dest folder then original

What is the problem you are having with rclone?

I run rclone copy as per below and it stopped halfway through at 20+Gb (can't recall exactly) out of 44 Gb; the rclone process was nowhere to be seen. I executed the same command relying on the fact that it shouldn't copy unchanged files.

That command has now finished properly, but the destination folder is now 63 Gb instead of 44 Gb. Judging by the numbers it could be a failure ; but when I look into the folder on pcloud, I can't see any! not sure where the fault here lies, but rclone itself shows the discrepancy (see below)

What is your rclone version (output from rclone version)

rclone v1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18 LTS amd64 (headless server, logged into through SSH)

Which cloud storage system are you using? (eg Google Drive)

OneDrive to pcloud

The command you were trying to run (and the test of size on original and destination folders)

[1]+  Done                    rclone copy unimelbod:Imágenes/ mypc:Images/
ubuntu@procesador:/data$ rclone size mypc:Images
Total objects: 24858
Total size: 63.160 GBytes (67817945602 Bytes)
ubuntu@procesador:/data$ rclone size unimelbod:Imágenes
Total objects: 14454
Total size: 44.240 GBytes (47502084172 Bytes)

The rclone config contents with secrets removed.

type = onedrive
token = {"access_token":"XXXXX$drive_id = XXXX
drive_type = business

type = pcloud
token = {"access_token":"XXXX","token_type":"bearer","expiry":"0001-$

As in the template, you'd have to run that and share a log to look at the issue further.

was mypc:Images empty before you started the copy?
it might have had some files in it

@asdffdsa, yes, the point is that there were files in the destination; but they were identical to a portion of the ones in the original folder; the destination folder was created by the first copy command that failed halfway through and I simply restarted it.


2020/07/23 14:33:35 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "-vv" "copy" "unimelbod:Imá$2020/07/23 14:33:35 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/07/23 14:33:35 DEBUG : fs cache: renaming cache item "unimelbod:Imágenes/" to be canonical "unimelbod:Imágenes"
2020/07/23 14:33:37 DEBUG : fs cache: renaming cache item "mypc:Imagenes/" to be canonical "mypc:Imagenes"
2020/07/23 14:33:43 INFO  : file1.jpg: Copied (new)
2020/07/23 14:33:44 INFO  : 20151125_093802.jpg: Copied (new)
2020/07/23 14:33:44 INFO  : file3.zip: Copied (new)


Transferred:       15.068G / 44.237 GBytes, 34%, 1.934 MBytes/s, ETA 4h17m25s
Transferred:         8719 / 14454, 60%
Elapsed time:   2h12m58.9s
 *           Álbum de cámara/20151123_113000.jpg:100% /2.239M, 638.030k/s, 0s
 *           Álbum de cámara/20151123_113130.jpg: 93% /2.964M, 494.031k/s, 0s
 *           Álbum de cámara/20151123_113145.jpg: 13% /2.309M, 91.981k/s, 22s
 *           Álbum de cámara/20151123_113227.jpg: transferring

2020/07/23 16:46:38 INFO  : Álbum de cámara/20151123_113130.jpg: Copied (new)
2020/07/23 16:46:39 INFO  : Álbum de cámara/20151123_113145.jpg: Copied (new)
2020/07/23 16:46:41 INFO  : Álbum de cámara/20151123_113227.jpg: Copied (new)
2020/07/23 16:46:45 INFO  : Álbum de cámara/20151123_114002.jpg: Copied (new)
2020/07/23 16:46:47 INFO  : Álbum de cámara/20151123_113501.jpg: Copied (new)
2020/07/23 16:46:49 INFO  : Álbum de cámara/20151123_114013.jpg: Copied (new)
2020/07/23 16:46:51 INFO  : Álbum de cámara/20151123_114047.jpg: Copied (new)
2020/07/23 16:46:52 INFO  : Álbum de cámara/20151123_114235.jpg: Copied (new)

############## rclone stopped without explanation here (just like the first time) and I restarted the transfer #########################

2020/07/23 22:43:43 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "-vv" "copy" "unimelbod:Imá$2020/07/23 22:43:43 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/07/23 22:43:43 DEBUG : fs cache: renaming cache item "unimelbod:Imágenes/" to be canonical "unimelbod:Imágenes"
2020/07/23 22:43:44 DEBUG : fs cache: renaming cache item "mypc:Imagenes/" to be canonical "mypc:Imagenes"
2020/07/23 22:43:45 DEBUG : llhkadf.mp4: Size and modification time the same (differ by 0s, within $2020/07/23 22:43:45 DEBUG : llhkadf.mp4: Unchanged skipping
2020/07/23 22:43:45 DEBUG : 20151125_093802.jpg: Size and modification time the same (differ by 0s, within tolerance$2020/07/23 22:43:45 DEBUG : 20151125_093802.jpg: Unchanged skipping

It's looking like it's doing the right thing; I'll report on the final outcome in a few hours. I thought I'd catch the earlier log messages now before the log file became unwieldy.

if the dest already has files and some of those files are different from the source
then the dest will be larger than the source.
that is expected

seems like you want to sync the source to the dest,
to make the dest an exact copy of the source.
perhaps you want rclone sync, not rclone copy.

and either way you should test first using

thank you @asdffdsa for the prompt reply! The files shouldn't be different from the source because they could only have come from the source... If they are, again, there must be something NQR about rclone copy.

At this stage, I was simply interested in testing whether rclone copy really ignores identical files; whether pcloud itself was playing up or what.. (I thought maybe people would have experience with this).

Transferring a different/smaller folder (and without the process dying) I got the same sizes, so I'm guessing it's not pcloud, although I'm still mesmerised by the fact that I can't seem to find any duplicates in the 63Gb folder copied from the 44 Gb one.. In fact, there are a couple of missing folders...

I didn't know about the --dry-run flag, how would it differ from me just copying to a temp destination and deleting if something doesn't seem right (which essentially what I'm doing)? Does it take much less time?

Sorry I'm new to the world of large cross-cloud-service file transfers, so I have a lot of questions...xD

--dry-run is a good flag to know about, but if you are using a temp dest folder, that is good.

maybe pcloud does versioning, and that is what the extra folder size is about.
rclone sync with --dry-run will let you know the differences between the source and dest.

there is a beta of rclone that creates a file with a list of files that are missing on the source and missing on the dest but i cannot find it.

I've had reports that pcloud can create duplicate files which is unexpected (and possibly a bug at pcloud).

So can you try rclone dedupe --dry-run remote: and see if it finds any duplicates?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.