Jeah
October 21, 2024, 7:41pm
1
Copy/Sync of large files i.e. 1 Gb always fails with the following message:
Failed to copy: multi-thread copy: failed to close object after copy: incorrect size after upload: got 0, want 1073299456
It's reproducible. Tried with different large files. Always the same result. The uploaded file is moved to the pCloud trash. The trashed files are completely ok. Obviously there is a delay on the pCloud side in updating the file size so that rClone is getting a zero file size returned.
2024/10/21 21:15:46 ERROR : AVeryLargeFile.dvdmedia/VIDEO_TS/VTS_04_1.VOB: Failed to copy: multi-thread copy: failed to close object after copy: incorrect size after upload: got 0, want 1073299456
Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
rclone v1.68.1
os/version: raspbian 12.7 (64 bit)
os/kernel: 6.6.51+rpt-rpi-2712 (aarch64)
os/type: linux
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.1
go/linking: static
go/tags: none
pCloud
rclone copy /tmp remote:tmp)flock /tmp/syncMedien.lock --command "rclone sync -P '/home/xxx/yyy/zzz' 'pCloud:Medien/zzz' --filter-from ignore.txt --no
[Jo]
type = crypt
remote = pCloud:Jo
password = XXX
filename_encryption = off
directory_name_encryption = false
[pCloud]
type = pcloud
hostname = eapi.pcloud.com
token = XXX
-vv flagif necessary I will submit the log later
asdffdsa
October 21, 2024, 7:44pm
2
welcome to the forum,
how did you confirm that?
please copy a single file and post the full debug log?--retries=1
Jeah
October 21, 2024, 7:50pm
3
I tried also the option --inplace - but makes no difference.
I will post -vv log tomorrow as i dont have the time now to wait for the copy to finish.
I also wanted to check from which size on the problem occurs. A few hundred Mb seem not to be a problem.
asdffdsa
October 21, 2024, 7:58pm
4
what i find a little confusing is that the error is about multi-threadrcone docs , pcloud does not support MultithreadUpload
edit: i just noticed from the changelog that multi-thread uploads was just enabled recentlyImplement OpenWriterAt feature to enable multipart uploads
in the past, did you use an older version of rclone, that did not have this issue or you just started using rclone with v1.68.1 ?
so, i might test a version of rclone, older than v1.68.0
Jeah
October 21, 2024, 8:08pm
5
I just startet using rclone and pCloud.
As the transfer has been already finished - here the -vv output.
flock /tmp/syncMedien.lock --command "rclone sync -Pvv '/home/jojo/Medien/Filme/Shaun das Schaf/Shaun das Schaf (Waschtag).dvdmedi
2024/10/21 22:06:15 DEBUG : 7 go routines active
Jeah
October 21, 2024, 8:09pm
6
By the way - thanks for looking into this... !!!
asdffdsa
October 21, 2024, 8:18pm
7
weclome.
going forward, use --inplace. just one less potential issue to deal with
Jeah
October 21, 2024, 8:33pm
8
Ok - here the log with --inplace.
If you move the transfered file from the trash to a different location and remove the xyzabc.partial ending of the trashed file it is always fully functional.
Good night - i have to leave now. And thanks again for digging into this problem.
flock /tmp/syncMedien.lock --command "rclone sync -Pvv '/home/jojo/Medien/Filme/Shaun das Schaf/Shaun das Schaf (Waschtag).dvdmedia/VIDEO_TS/VTS_04_1.VOB' 'pCloud:Medien/Filme/Shaun das Schaf/Shaun das Schaf (Waschtag).dvdmedia/VIDEO_TS/' --filter-from ignore.txt --no-update-modtime --retries=1 --inplace"
2024/10/21 22:28:25 DEBUG : 7 go routines active
Jeah
October 22, 2024, 9:57am
9
To tear down the problem i have created test files with sizes between 300 and 900 Mb using dd.
Tried the transfer 3 times. Result is a bit surprising!
In the first attempt the 700 and 800 Mb files had problems, in the second attempt is was the 500 Mb file and in the third the 800 Mb file.
Here are the log of the first attempt (only this one because of the character limit of the post).
...
2024/10/22 11:18:43 DEBUG : 7 go routines active
Jeah
October 22, 2024, 10:36am
10
Setting --low-level-retries to 10 seems to cure the problem.
Update:
Tried this with files they always failed to transfer correctly.
Even set to 10 - rclone always tries only 5 times and is giving up then. See log. Why? Thought this will instruct rclone to retry every single pCloud command x times.
Here the relevant part of the log of copying a 1 Gb file.
2024/10/22 12:56:25 DEBUG : VTS_04_1.VOB: multi-thread copy: chunk 16/16 (1006632960-1073299456) size 63.578Mi finished
2024/10/22 12:56:26 DEBUG : VTS_04_1.VOB: multi-thread copy: chunk 13/16 (805306368-872415232) size 64Mi finished
2024/10/22 12:56:26 DEBUG : VTS_04_1.VOB: multi-thread copy: chunk 14/16 (872415232-939524096) size 64Mi finished
2024/10/22 12:56:26 DEBUG : VTS_04_1.VOB: multi-thread copy: chunk 15/16 (939524096-1006632960) size 64Mi finished
2024/10/22 12:56:26 DEBUG : VTS_04_1.VOB.fa329ff8.partial: checking file size: try 0/5
2024/10/22 12:56:27 DEBUG : VTS_04_1.VOB.fa329ff8.partial: checking file size: try 1/5
2024/10/22 12:56:28 DEBUG : VTS_04_1.VOB.fa329ff8.partial: checking file size: try 2/5
2024/10/22 12:56:29 DEBUG : VTS_04_1.VOB.fa329ff8.partial: checking file size: try 3/5
2024/10/22 12:56:30 DEBUG : VTS_04_1.VOB.fa329ff8.partial: checking file size: try 4/5
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 13/42 (805306368-872415232) size 64Mi finished
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 17/42 (1073741824-1140850688) size 64Mi starting
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4.ddba4246.partial: writing chunk 16
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 14/42 (872415232-939524096) size 64Mi finished
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 18/42 (1140850688-1207959552) size 64Mi starting
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4.ddba4246.partial: writing chunk 17
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 15/42 (939524096-1006632960) size 64Mi finished
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 19/42 (1207959552-1275068416) size 64Mi starting
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4.ddba4246.partial: writing chunk 18
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 16/42 (1006632960-1073741824) size 64Mi finished
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4: multi-thread copy: chunk 20/42 (1275068416-1342177280) size 64Mi starting
2024/10/22 12:56:31 DEBUG : THE_RIGHT_STUFF.mp4.ddba4246.partial: writing chunk 19
2024/10/22 12:56:31 DEBUG : VTS_04_1.VOB: multi-thread copy: cancelling transfer on exit
2024/10/22 12:56:31 ERROR : VTS_04_1.VOB: Failed to copy: multi-thread copy: failed to close object after copy: incorrect size after upload: got 0, want 1073299456
dinosm
October 22, 2024, 10:40am
11
I wonder what the cause of this is. Timeouts by your ISP somehow? I have moved files much bigger than that to pCloud without issues.
Jeah
October 22, 2024, 10:53am
12
German Telekom Fiber - usually quite reliable.
dinosm
October 22, 2024, 11:23am
13
Not so much about reliability but maybe some throttling going on for large uploads?
Jeah
October 22, 2024, 12:37pm
14
Dont think that this is a problem of throttling.
The files are complete! after transfer. rclone justvis not able to verify the file size.
So there must be a problem in.interacting of rclone and pCloud.
relfoh
October 22, 2024, 1:18pm
15
I am facing the same problem from a completely different place, looking forward to your inputs on the matter!
For what is worth, in my case the file size fetched from pcloud gets to the proper value only after some time.. So, if I was able to postpone the file.size check (or avoid it completely), it think it would be a step in the right direction.
Jeah
October 22, 2024, 1:27pm
17
I agree.
Thats why i tought that the option --low-level-retries would help. But unfotunately it is ignored for getting the file size.
And i didn't find any other option that would do this.
Jeah
October 22, 2024, 1:33pm
18
Why,do you think.that is a problem of the recent release?
I have found a discussion somewhere else (forgotten where) whete it was stated that there is a problem with files bigger than 200 Mb.
So it seems to me this is an older issue.
asdffdsa
October 22, 2024, 1:36pm
19
i mentioned that up above...
Jeah
October 22, 2024, 1:45pm
20
Will try,in the evening...