sizeDiffers incorrect after transfer

hahanonym · March 15, 2021, 1:07pm

What is the problem you are having with rclone?

sync/sync file not overwrite if src content modify or dest conten modify .
I debug the code find that ,sizeDiffers maybe Incorrect.

What is your rclone version (output from `rclone version`)

latest master code

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

local system

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)


rclone.exe rcd --rc-addr=":5572" --ignore-case --ignore-case-sync --progress --log-file=rclone.log --log-level DEBUG --rc-user=abc --rc-pass=abc --rc-allow-origin=":5572" --rc-web-gui-no-open-browser --rc-files=webgui --rc-job-expire-duration=10s --rc-job-expire-interval=1s --local-no-check-updated
remote control mode : sync/sync
_async=true
_config=null

The rclone config contents with secrets removed.

Paste config here

A log from the command with the `-vv` flag

After transfer ,before removeFailedCopy,the file is overwrite correct,
but after differ,the application get file size is incorrect.

asdffdsa · March 15, 2021, 1:51pm

hello and welcome to the forum,

does this happen every time you run rclone, or just this one time?

about that file, do you have that it open in a file editor?

perhaps there is a background process modifying whilst rclone is running?

hahanonym · March 15, 2021, 3:19pm

every time

no ,I have test many times.
the repeat path is
1.normal sync file
2.modify src or dest file
3.sync again then failed （dest will be remove）

ncw · March 16, 2021, 2:16pm

Can you show an rclone command to replicate this please?

Please show a sequence of commands to replicate the problem.

Thank you

hahanonym · March 18, 2021, 10:07am

1.Create folder "D:\Data\sync\" ,then create "test.txt" in folder "sync"
2.Normal start remote contorl rclone
3. Use postman send

4.Modify D:\Data\sync\test.txt or G:\testSync\test.txt,then postman post again

If you run rclone with code,you can set breakpoint at

The break point will be triggered on step4.
At the moment,you can check src and dest manully.
then you can get the same bug~

ncw · March 18, 2021, 5:00pm

I managed to replicate this (in a simpler way)

$ echo hello >> /tmp/src/test.txt
$ rclone copy -vv /tmp/src/ /tmp/dst/ --local-no-check-updated
2021/03/18 16:54:49 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2021/03/18 16:54:49 DEBUG : rclone: Version "v1.55.0-beta.5317.9ee3ad70e" starting with parameters ["rclone" "copy" "-vv" "/tmp/src/" "/tmp/dst/" "--local-no-check-updated"]
2021/03/18 16:54:49 DEBUG : Creating backend with remote "/tmp/src/"
2021/03/18 16:54:49 DEBUG : local: detected overridden config - adding "{yKcAd}" suffix to name
2021/03/18 16:54:49 DEBUG : fs cache: renaming cache item "/tmp/src/" to be canonical "local{yKcAd}:/tmp/src/"
2021/03/18 16:54:49 DEBUG : Creating backend with remote "/tmp/dst/"
2021/03/18 16:54:49 DEBUG : local: detected overridden config - adding "{yKcAd}" suffix to name
2021/03/18 16:54:49 DEBUG : fs cache: renaming cache item "/tmp/dst/" to be canonical "local{yKcAd}:/tmp/dst/"
2021/03/18 16:54:49 DEBUG : test.txt: Sizes differ (src 42 vs dst 36)
2021/03/18 16:54:49 DEBUG : Local file system at /tmp/dst/: Waiting for checks to finish
2021/03/18 16:54:49 DEBUG : Local file system at /tmp/dst/: Waiting for transfers to finish
2021/03/18 16:54:49 ERROR : test.txt: corrupted on transfer: sizes differ 42 vs 36
2021/03/18 16:54:49 INFO  : test.txt: Removing failed copy
2021/03/18 16:54:49 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 42 vs 36
2021/03/18 16:54:49 DEBUG : Local file system at /tmp/dst/: Waiting for checks to finish
2021/03/18 16:54:49 DEBUG : Local file system at /tmp/dst/: Waiting for transfers to finish
2021/03/18 16:54:49 DEBUG : test.txt: MD5 = 7080d7306b89ac74926f3b446d955007 OK
2021/03/18 16:54:49 INFO  : test.txt: Copied (new)
2021/03/18 16:54:49 ERROR : Attempt 2/3 succeeded
2021/03/18 16:54:49 INFO  : 
Transferred:   	        84 / 84 Bytes, 100%, 112.242 kBytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/18 16:54:49 DEBUG : 2 go routines active

It is the --local-no-check-updated flag which is causing the problem - it is applying to both the source and the destination.

If you create a remote for the source like this

[src]
type = local
no_check_updated = true

Then you will not apply no_check_updated = true on the destination which will fix the problem.

This works OK

$ echo hello >> /tmp/src/test.txt
$ rclone copy -vv src:/tmp/src/ /tmp/dst/
2021/03/18 16:57:44 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2021/03/18 16:57:44 DEBUG : rclone: Version "v1.55.0-beta.5317.9ee3ad70e" starting with parameters ["rclone" "copy" "-vv" "src:/tmp/src/" "/tmp/dst/"]
2021/03/18 16:57:44 DEBUG : Creating backend with remote "src:/tmp/src/"
2021/03/18 16:57:44 DEBUG : Creating backend with remote "/tmp/dst/"
2021/03/18 16:57:44 DEBUG : test.txt: Sizes differ (src 48 vs dst 42)
2021/03/18 16:57:44 DEBUG : Local file system at /tmp/dst/: Waiting for checks to finish
2021/03/18 16:57:44 DEBUG : Local file system at /tmp/dst/: Waiting for transfers to finish
2021/03/18 16:57:44 DEBUG : test.txt: MD5 = 755e352d2753d546754f90f35d2d0a1e OK
2021/03/18 16:57:44 INFO  : test.txt: Copied (replaced existing)
2021/03/18 16:57:44 INFO  : 
Transferred:   	        48 / 48 Bytes, 100%, 146.318 kBytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/18 16:57:44 DEBUG : 2 go routines active

If using the latest beta you can write this like this without creating a new remote

$ echo hello >> /tmp/src/test.txt
$ rclone copy -vv :local,no_check_updated:/tmp/src/ /tmp/dst/
2021/03/18 16:58:38 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2021/03/18 16:58:38 DEBUG : rclone: Version "v1.55.0-beta.5317.9ee3ad70e" starting with parameters ["rclone" "copy" "-vv" ":local,no_check_updated:/tmp/src/" "/tmp/dst/"]
2021/03/18 16:58:38 DEBUG : Creating backend with remote ":local,no_check_updated:/tmp/src/"
2021/03/18 16:58:38 DEBUG : :local: detected overridden config - adding "{yKcAd}" suffix to name
2021/03/18 16:58:38 DEBUG : fs cache: renaming cache item ":local,no_check_updated:/tmp/src/" to be canonical ":local{yKcAd}:/tmp/src/"
2021/03/18 16:58:38 DEBUG : Creating backend with remote "/tmp/dst/"
2021/03/18 16:58:38 DEBUG : test.txt: Sizes differ (src 54 vs dst 48)
2021/03/18 16:58:38 DEBUG : Local file system at /tmp/dst/: Waiting for checks to finish
2021/03/18 16:58:38 DEBUG : Local file system at /tmp/dst/: Waiting for transfers to finish
2021/03/18 16:58:38 DEBUG : test.txt: MD5 = 4a963125957329343b9a89d85dbb57a0 OK
2021/03/18 16:58:38 INFO  : test.txt: Copied (replaced existing)
2021/03/18 16:58:38 INFO  : 
Transferred:   	        54 / 54 Bytes, 100%, 180.642 kBytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2021/03/18 16:58:38 DEBUG : 4 go routines active

hahanonym · March 19, 2021, 5:56am

sorry,I'm confused.
Is there a 'start commandLine parameter' or 'options set' to solve my problem?
I have test ":local,no_check_updated:" as start commandLine occured Error.

ncw · March 21, 2021, 4:47pm

You will need the latest beta to use that.

Post the log if you can't make it work.

hahanonym · March 22, 2021, 1:31am

Sorry，the start parameter worked。
But I also need GitHub - rclone/rclone at fix-5116-rc-stats
Can you merge this branch？

ncw · March 22, 2021, 10:13am

I've merged fix-5116-rc-stats now too

hahanonym · March 23, 2021, 5:38am

RC can use this parameter to start，but access webUI get “ 500 Internal Server Error“。
and other api return correct。please help

ncw · March 23, 2021, 3:33pm

Can you send a log please? I can't see what is wrong. I tried the GUI but it seems to work.

hahanonym · March 23, 2021, 4:18pm

2021/03/24 00:13:19 DEBUG : Using config file from "C:\\Users\\zhoufeng1\\.config\\rclone\\rclone.conf"
2021/03/24 00:13:19 DEBUG : rclone: Version "v1.55.0-DEV" starting with parameters ["e:\\CodeBase\\Rclone\\__debug_bin" "rcd" "--rc-addr=:5573" "--rc-user=abc" "--rc-pass=abc" "--rc-web-gui-no-open-browser" ":local,no_check_updated:" "--rc-files=E:\\CodeBase\\T7\\tools\\DataManageTool\\rclone\\win\\webgui" "--log-file=rclonelogfile.log" "-vv"]
2021/03/24 00:13:19 NOTICE: Serving files from ":local,no_check_updated:"
2021/03/24 00:13:19 INFO  : Using --user abc --pass XXXX as authenticated user
2021/03/24 00:13:19 NOTICE: Serving remote control on http://[::]:5573/
2021/03/24 00:13:19 DEBUG : login_token "YWJjOmFiYw=="
2021/03/24 00:13:19 NOTICE: Web GUI is not automatically opening browser. Navigate to http://abc:abc@[::]:5573/?login_token=YWJjOmFiYw%3D%3D to use.
2021/03/24 00:16:55 DEBUG : rc: "core/version": with parameters map[jobid:19]
2021/03/24 00:16:55 DEBUG : rc: "core/version": reply map[arch:amd64 decomposed:[1 55 0] goTags:none goVersion:go1.15.3 isBeta:true isGit:true linking:dynamic os:windows version:v1.55.0-DEV]: <nil>
2021/03/24 00:17:04 DEBUG : rc: "core/stats": with parameters map[jobid:19]
2021/03/24 00:17:04 DEBUG : rc: "core/stats": reply map[bytes:0 checks:0 deletedDirs:0 deletes:0 elapsedTime:225.044953 errors:0 eta:<nil> fatalError:false renames:0 retryError:false speed:0 totalBytes:0 totalChecks:0 totalTransfers:0 transferTime:0 transfers:0]: <nil>

ncw · March 23, 2021, 5:03pm

I don't see the 500 internal error in rclone's logs?

hahanonym · March 24, 2021, 6:36am

I use “:local,no_check_updated:” as an independent parameter
I debug the code,found some differences with no paramter.
Then I delete line (42/43/44) ，the error is gone。
Is this a historical bug？

ncw · March 24, 2021, 9:31am

I don't know what is going on here!

It seems to work for me.

Can you describe how to replicate the problem?

Thanks

hahanonym · March 24, 2021, 9:33am

This is my visual code start parameter。On windows10 64 bit。
Just this。

ncw · March 24, 2021, 9:59am

What did you do to cause the error?

What does the rclone log look like? You should be able to see the 500 error in the log.

Can you replicate with just the rclone rc command instead of the web gui?

hahanonym · March 24, 2021, 10:23am

No other operations，just start RCD with web gui，then Enter the URL in the browser
There is no 500 error in the log
The “500 error ”No impact on other APIs. And this parameter is also effective

2021/03/24 00:13:19 DEBUG : rclone: Version "v1.55.0-DEV" starting with parameters ["e:\CodeBase\Rclone\__debug_bin" "rcd" "--rc-addr=:5573" "--rc-user=abc" "--rc-pass=abc" "--rc-web-gui-no-open-browser" ":local,no_check_updated:" "--rc-files=E:\CodeBase\T7\tools\DataManageTool\rclone\win\webgui" "--log-file=rclonelogfile.log" "-vv"]

ncw · March 24, 2021, 10:54am

You are using your own web gui files?

Can you try the latest ones from the project?