Some files mysteriously do not transfer with rclone copy, including an AVCHD folder?

dropboxgoogledrive · September 16, 2023, 3:38am

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I am trying to run rclone copy but certain folders do not copy completely, with no error or debug showing. In fact, it seems the checksum also passes check. What happens is I download an AVCHD folder online, which is 1.61 GB, but using rclone copy only gives me 5 Kb. What could be the reason?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: rocky 8.7 (64 bit)
- os/kernel: 4.18.0-425.10.1.el8_7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -P gd:/ "/Volumes/My Large Hard Drive" --exclude "/Folder 1 to exclude/**" --exclude "/Folder 2 to exclude/**"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gd]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
acknowledge_abuse = true
token = XXX
root_folder_id = XXX
team_drive =

A log from the command that you were trying to run with the `-vv` flag

The below is with --dry-run option

2023/09/15 18:03:11 DEBUG : Creating backend with remote "gd:/"
2023/09/15 18:03:11 DEBUG : Using config file from "/Users/home/.config/rclone/rclone.conf"
2023/09/15 18:03:11 DEBUG : fs cache: renaming cache item "gd:/" to be canonical "gd:"
2023/09/15 18:03:11 DEBUG : Creating backend with remote "/Volumes/My Large Hard Drive/"
2023/09/15 18:03:11 DEBUG : fs cache: renaming cache item "/Volumes/My Large Hard Drive/" to be canonical "/Volumes/My Large Hard Drive"
2023-09-15 18:03:11 DEBUG : gd: Loaded invalid token from config file - ignoring
2023-09-15 18:03:11 DEBUG : Saving config "token" in section "gd" of the config file
2023-09-15 18:03:11 DEBUG : gd: Saved new token in config file
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.5s^C

kapitainsky · September 16, 2023, 7:36am

Post results of:

rclone lsd gd:/ --exclude "/Folder 1 to exclude/**" --exclude "/Folder 2 to exclude/**"

and:

rclone copy -P gd:/ "/Volumes/My Large Hard Drive" --exclude "/Folder 1 to exclude/**" --exclude "/Folder 2 to exclude/**" -vv

dropboxgoogledrive · September 16, 2023, 9:39am

Thanks, these are the results below.

For the second command I had to truncate and only show portions of it due to how large it is. I can output all of it if necessary. Basically, Private/ACVHD should be 1.61 GB. However, each time I run copy it seems to sometimes only capture a portion (1.48 GB) or none of it (5 Kb). It is almost as if it is random in how much it captures, with no real errors outputs.

rclone lsd gd:"Files1/ex"
          -1 2016-02-24 18:02:01        -1 PRIVATE
          -1 2016-02-24 18:02:01        -1 STREAM

and then

rclone copy -P gd:"Files1/ex" "/Volumes/Media/test 2" -vv
2023/09/16 02:15:08 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "-P" "gd:Files1/ex" "/Volumes/Media/test 2" "-vv"]
2023/09/16 02:15:08 DEBUG : Creating backend with remote "gd:Files1/ex"
2023/09/16 02:15:08 DEBUG : Using config file from "/Users/user/.config/rclone/rclone.conf"
2023/09/16 02:15:09 DEBUG : Creating backend with remote "/Volumes/Media/test 2"
2023-09-16 02:15:11 DEBUG : DSC04489.JPG: Need to transfer - File not found at Destination
2023-09-16 02:15:11 DEBUG : DSC04490.JPG: Need to transfer - File not found at Destination
2023-09-16 02:15:11 DEBUG : DSC04492.JPG.zelocog0.partial: renamed to: DSC04492.JPG
2023-09-16 02:15:11 INFO  : DSC04492.JPG: Copied (new)
2023-09-16 02:15:11 DEBUG : DSC04489.JPG.tojeyud8.partial: renamed to: DSC04489.JPG
2023-09-16 02:15:11 INFO  : DSC04489.JPG: Copied (new)
2023-09-16 02:15:11 DEBUG : PRIVATE/SONY/SONYCARD(2).IND: Need to transfer - File not found at Destination
2023-09-16 02:15:11 DEBUG : PRIVATE/SONY/SONYCARD.IND: Need to transfer - File not found at Destination
2023-09-16 02:15:11 DEBUG : DSC04490.JPG: md5 = 832d607c6604db2776482445e8e8d7d4 OK
2023-09-16 02:15:11 DEBUG : DSC04490.JPG.neginug4.partial: renamed to: DSC04490.JPG
2023-09-16 02:15:11 INFO  : DSC04490.JPG: Copied (new)
2023-09-16 02:15:12 DEBUG : DSC04491.JPG: md5 = 18e2017a6a97965b537d47f3363afca6 OK
2023-09-16 02:15:12 DEBUG : DSC04491.JPG.cufeway2.partial: renamed to: DSC04491.JPG
2023-09-16 02:15:12 INFO  : DSC04491.JPG: Copied (new)
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX(2).BDM: Need to transfer - File not found at Destination
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX.BDM: Need to transfer - File not found at Destination
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ(2).BDM: Need to transfer - File not found at Destination
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM: Need to transfer - File not found at Destination
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00000(2).MPL: Need to transfer - File not found at Destination
2023-09-16 02:15:12 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00000.MPL: Need to transfer - File not found at Destination
....

2023-09-16 02:16:11 DEBUG : PRIVATE/SONY/SONYCARD(2).IND: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2023-09-16 02:16:11 DEBUG : PRIVATE/SONY/SONYCARD(2).IND.nexuhoz8.partial: renamed to: PRIVATE/SONY/SONYCARD(2).IND
2023-09-16 02:16:11 INFO  : PRIVATE/SONY/SONYCARD(2).IND: Copied (new)
2023-09-16 02:16:11 DEBUG : DSC05031.JPG: md5 = ec9936f4af2973803b6454009c0e3cb5 OK
2023-09-16 02:16:11 DEBUG : DSC05031.JPG.dusumup9.partial: renamed to: DSC05031.JPG
2023-09-16 02:16:11 INFO  : DSC05031.JPG: Copied (new)
2023-09-16 02:16:11 DEBUG : PRIVATE/SONY/SONYCARD.IND: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2023-09-16 02:16:11 DEBUG : PRIVATE/SONY/SONYCARD.IND.ratoror8.partial: renamed to: PRIVATE/SONY/SONYCARD.IND
2023-09-16 02:16:11 INFO  : PRIVATE/SONY/SONYCARD.IND: Copied (new)
2023-09-16 02:16:12 DEBUG : DSC05032.JPG: md5 = a3a83f84e7ab7fca9f615f8e5c7931ea OK
2023-09-16 02:16:12 DEBUG : DSC05032.JPG.gohixux9.partial: renamed to: DSC05032.JPG
2023-09-16 02:16:12 INFO  : DSC05032.JPG: Copied (new)
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX(2).BDM: md5 = 7d9e20a991675b86170b4a3318bc768c OK
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX(2).BDM.watefac3.partial: renamed to: PRIVATE/AVCHD/BDMV/INDEX(2).BDM
2023-09-16 02:16:12 INFO  : PRIVATE/AVCHD/BDMV/INDEX(2).BDM: Copied (new)
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX.BDM: md5 = 7d9e20a991675b86170b4a3318bc768c OK
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/INDEX.BDM.peximod5.partial: renamed to: PRIVATE/AVCHD/BDMV/INDEX.BDM
2023-09-16 02:16:12 INFO  : PRIVATE/AVCHD/BDMV/INDEX.BDM: Copied (new)
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00000(2).MPL: md5 = 00e7dd3a05d1c8244c09b49b92d90731 OK
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM: md5 = 108579aaa981ad6a8b6b2ce034046d7b OK
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00000(2).MPL.kemafeb8.partial: renamed to: PRIVATE/AVCHD/BDMV/PLAYLIST/00000(2).MPL
2023-09-16 02:16:12 INFO  : PRIVATE/AVCHD/BDMV/PLAYLIST/00000(2).MPL: Copied (new)
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM.huqiyos6.partial: renamed to: PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM
2023-09-16 02:16:12 INFO  : PRIVATE/AVCHD/BDMV/MOVIEOBJ.BDM: Copied (new)
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ(2).BDM: md5 = 108579aaa981ad6a8b6b2ce034046d7b OK
2023-09-16 02:16:12 DEBUG : PRIVATE/AVCHD/BDMV/MOVIEOBJ(2).BDM.recalov1.partial: renamed to: PRIVATE/AVCHD/BDMV/MOVIEOBJ(2).BDM
2023-09-16 02:16:12 INFO  : PRIVATE/AVCHD/BDMV/MOVIEOBJ(2).BDM: Copied (new)
2023-09-16 02:16:13 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00001.MPL: md5 = 79329d05d240a286f3ffe73e8fec21ef OK
2023-09-16 02:16:13 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00001.MPL.suropup2.partial: renamed to: PRIVATE/AVCHD/BDMV/PLAYLIST/00001.MPL
2023-09-16 02:16:13 INFO  : PRIVATE/AVCHD/BDMV/PLAYLIST/00001.MPL: Copied (new)
2023-09-16 02:16:13 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00001(2).MPL: md5 = 79329d05d240a286f3ffe73e8fec21ef OK
2023-09-16 02:16:13 DEBUG : PRIVATE/AVCHD/BDMV/PLAYLIST/00001(2).MPL.liduvem5.partial: renamed to: PRIVATE/AVCHD/BDMV/PLAYLIST/00001(2).MPL
2023-09-16 02:16:13 INFO  : PRIVATE/AVCHD/BDMV/PLAYLIST/00001(2).MPL: Copied (new)
....
2023-09-16 02:17:12 DEBUG : PRIVATE/AVCHD/BDMV/CLIPINF/00148.CPI: md5 = 0cbe9e33fb015d0bc87da3c8a21be2fe OK
2023-09-16 02:17:12 DEBUG : PRIVATE/AVCHD/BDMV/CLIPINF/00148.CPI.qenegiw4.partial: renamed to: PRIVATE/AVCHD/BDMV/CLIPINF/00148.CPI
2023-09-16 02:17:12 INFO  : PRIVATE/AVCHD/BDMV/CLIPINF/00148.CPI: Copied (new)
Transferred:   	    1.851 GiB / 1.851 GiB, 100%, 7.905 MiB/s, ETA 0s
Transferred:          826 / 826, 100%
Elapsed time:       2m3.7s
2023/09/16 02:17:12 INFO  :
Transferred:   	    1.851 GiB / 1.851 GiB, 100%, 7.905 MiB/s, ETA 0s
Transferred:          826 / 826, 100%
Elapsed time:       2m3.7s

kapitainsky · September 16, 2023, 9:43am

And what:

rclone size gd:Files1/ex/Private/ACVHD

says?

also on your local disk:

du -sh "/Volumes/Media/test 2/Private/ACVHD"

kapitainsky · September 16, 2023, 9:51am

Any reason you are using end of life OS?

dropboxgoogledrive · September 16, 2023, 9:57am

Ah, very interesting, I have that

rclone size gd:Files1/ex/Private/ACVHD
Total objects: 326 (326)
Total size: 1.496 GiB (1605943520 Byte)

and on local disk

du -sh "/Volumes/Media/test 2/Private/ACVHD"
1.5G

I also did

du -sb "/Volumes/Media/test 2/Private/ACVHD"
1605955080

What is most strange is that in finder, it is only reported as 5 Kb below

Regarding end of life OS, that was my test on linux. I re-did the same thing on my MacOS and saw the same thing. The MacOS information is

rclone v1.64.0
- os/version: darwin 13.5.2 (64 bit)
- os/kernel: 22.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.21.1
- go/linking: dynamic
- go/tags: cmount

Thank you.

kapitainsky · September 16, 2023, 9:59am

What is this? AVCHD directory icon looks like file.

du command in terminal shows real size.

Why program you posted screenshot from shows 5k is not rclone problem really...

dropboxgoogledrive · September 16, 2023, 10:05am

It is a folder structure where you can click "Show Package Contents" to go inside. I just tested something where I disconnected the hard drive from my computer and then remounted/connected it. It seems the AVCHD file is now reporting as 1.61 GB.

However, I repeated this 5 times previously and 2 of the tries still show 5 Kb for the folder even though du -sh shows 1.5G. Do you know if there is some Finder updating that is going on? Thanks

kapitainsky · September 16, 2023, 10:07am

for sure rclone does not interfere with Finder. What Finder does and how is Apple's secret:)

You can use different files' browser - myself for example I use ForkLift as I was tired of Finder random behaviour.

dropboxgoogledrive · September 16, 2023, 10:12am

Thanks very much, this seems to solve it. Just wondering, is it expected for du -sb to not exactly match the byte size in rclone size?

kapitainsky · September 16, 2023, 10:14am

I guess it might be some apple extra on local disk? Like .DS_Store? As local size is larger by few KB.

You could run:

rclone check src: dst: -vv

to identify the culprit

system · September 19, 2023, 10:14am

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.