RClone Copy to Gdrive failing to identify already uploaded files

So, I decided to start uploading my Plex folder to Gdrive but I’ve hit the 750GB daily limit.

Ok, no big deal, I’ve waited 24 hours and just issue the same command. I was hopping rclone would identify the already uploaded files but it seems I’m wrong.

This is the command I’m running

rclone copy --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --stats 1s /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Is there something wrong I`m doing?

It would be really helpful I could just type the same command once a week without having to remember what is actually new on the server.

It will do just that. It’ll compare mod time and size or checksum depending on the remote type. If you turn on debug -vv you’ll see why it’s deciding to re-upload.

Based on that:

  -c, --checksum                                     Skip based on checksum (if available) & size, not mod-time & size

This will not check mod-time and check only size and checksum.

Do you know if checksum is available with gdrive encrypted with crypt?

Checksum is not available for crypt

Ok tried again and it IS copying everything again.

I can’t see from the verbose why it’s copying.

If you can share the -vv output, it should write in there why it is being copied.

> root@Tower:/mnt/user/PLEX# rclone copy -vv --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10  --no-traverse --dry-run  /mnt/user/PLEX/TV\ Shows/ secure:STREAM
> 2019/04/09 07:48:13 DEBUG : rclone: Version "v1.46.0-112-g1c301f9f-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "-vv" "--transfers" "10" "--checkers" "5" "--contimeout" "60s" "--timeout" "300s" "--retries" "6" "--low-level-retries" "15" "--tpslimit" "10" "--no-traverse" "--dry-run" "/mnt/user/PLEX/TV Shows/" "secure:STREAM"]
> 2019/04/09 07:48:13 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
> 2019/04/09 07:48:13 INFO  : Starting HTTP transaction limiter: max 10 transactions/s with burst 1
> 2019/04/09 07:48:15 NOTICE: Band of Brothers/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Avatar - The Last Airbender/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Aqua Teen Hunger Force/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Betty Boop/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Batman - The Animated Series/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Bobby's World/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Cardcaptor Sakura/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Carnivàle/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Chapolin Colorado/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Comedy Central Presents/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:17 NOTICE: Courage the Cowardly Dog/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:17 NOTICE: Cow and Chicken/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/Betty Boop - S1930E01 - Dizzy Dishes.mkv: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/season.nfo: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/Betty Boop - S1930E01 - Dizzy Dishes.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E01 - Currahee.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E02 - Day of Days.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E03 - Carentan.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E05 - Crossroads.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E06 - Bastogne.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E07 - The Breaking Point.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E08 - The Last Patrol.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E08 - The Last Patrol.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E09 - Why We Fight.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E10 - Points.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E10 - Points.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E09 - Why We Fight.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E07 - The Breaking Point.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E03 - Carentan.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E06 - Bastogne.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E05 - Crossroads.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E02 - Day of Days.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E01 - Currahee.mkv: Not copying as --dry-run

I’m doing a dry run now but it’s the same output, just instead it will copy the files over and over again.

For the first time I’ve seen Rclone crash

goroutine 60 [select]:
net/http.(*persistConn).roundTrip(0xc0002ad9e0, 0xc003e34ff0, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/transport.go:2174 +0x75c
net/http.(*Transport).roundTrip(0xc00014a240, 0xc00010ed00, 0x137b524, 0xa, 0xc003d09150)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/transport.go:481 +0xa1b
net/http.(*Transport).RoundTrip(...)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/roundtrip.go:17
github.com/ncw/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0001271d0, 0xc00010ed00, 0xc00010ed00, 0x0, 0x10)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:324 +0x1e3
github.com/ncw/rclone/vendor/golang.org/x/oauth2.(*Transport).RoundTrip(0xc00011a5a0, 0xc00010ec00, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/golang.org/x/oauth2/transport.go:56 +0xfc
net/http.send(0xc00010ec00, 0x15de4c0, 0xc00011a5a0, 0x0, 0x0, 0x0, 0xc00000e350, 0xc003d08dc0, 0x1, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc00011a5d0, 0xc00010ec00, 0x0, 0x0, 0x0, 0xc00000e350, 0x0, 0x1, 0x5a25a75a1d51beec)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc00011a5d0, 0xc00010ec00, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:509
github.com/ncw/rclone/backend/drive.(*resumableUpload).transferChunk(0xc000125440, 0x800000, 0x15ee640, 0xc000609480, 0x800000, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:167 +0xaa
github.com/ncw/rclone/backend/drive.(*resumableUpload).Upload.func1(0x0, 0x0, 0x8)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:214 +0x13e
github.com/ncw/rclone/fs.pacerInvoker(0x1, 0xf, 0xc0006094c0, 0xc0006094c0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fs.go:1189 +0x3c
github.com/ncw/rclone/lib/pacer.(*Pacer).call(0xc000118240, 0xc0006094c0, 0xf, 0x40, 0x12e30a0)
        /home/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:197 +0x8b
github.com/ncw/rclone/lib/pacer.(*Pacer).Call(0xc000118240, 0xc0006094c0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:216 +0x78
github.com/ncw/rclone/backend/drive.(*resumableUpload).Upload(0xc000125440, 0xc000125440, 0x8, 0xc003ef2000)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:212 +0x2a0
github.com/ncw/rclone/backend/drive.(*Fs).Upload(0xc000177800, 0x15dd1e0, 0xc00031b080, 0x494e0a0e7, 0x138f561, 0x18, 0xc0013d2030, 0x21, 0xc003ecc960, 0x9d, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:113 +0x5bf
github.com/ncw/rclone/backend/drive.(*baseObject).update(0xc000100100, 0xc0047fe300, 0x138f561, 0x18, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0x0, 0x0, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:2669 +0x2d4
github.com/ncw/rclone/backend/drive.(*Object).Update(0xc000100100, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0xc000388870, 0x1, 0x1, 0x18, 0x128f060)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:2683 +0x181
github.com/ncw/rclone/backend/drive.(*Fs).Put(0xc000177800, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0xc000388870, 0x1, 0x1, 0x20, 0x128f060, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:1641 +0x201
github.com/ncw/rclone/backend/crypt.(*Fs).put(0xc0002fa140, 0x15dd1c0, 0xc00050a780, 0x147644a0d698, 0xc00064c9c0, 0xc000388870, 0x1, 0x1, 0xc0013e9c30, 0x0, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:357 +0x195
github.com/ncw/rclone/backend/crypt.(*Fs).Put(0xc0002fa140, 0x15dd1c0, 0xc00050a780, 0x147644a0d698, 0xc00064c9c0, 0xc000388870, 0x1, 0x1, 0x1611f20, 0xc000346e20, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:389 +0xd5
github.com/ncw/rclone/fs/operations.Copy(0x1611ea0, 0xc0002fa140, 0x0, 0x0, 0xc00003bbd0, 0x43, 0x16128a0, 0xc00064c9c0, 0x1611f20, 0xc000346e20, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/operations/operations.go:317 +0x14c4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc0000aa000, 0xc0001209c0, 0x1611ea0, 0xc0002fa140, 0xc0000aa0d8)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:295 +0x251
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startTransfers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:321 +0x9f

Just did this:
rclone copy --verbose --transfers 7 --checkers 4 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --bwlimit 8700K --stats 1s --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

And got that

EDIT:
Ok, I can see what you are talking about. It says new besides the copy process.

Like so:
2019/04/09 08:13:48 INFO : Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.nfo: Copied (new)

The thing is I’m running like this now:
rclone copy -vv --stats 2s --no-update-modtime --size-only --fast-list --no-traverse --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries6 --low-level-retries 15 --tpslimit 5 --bwlimit 8700K --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

I’m using every option to avoid it to compare dates and time, to ignore existing files but it doesn’t matter.
Rclone is doing whatever it thinks it should.

Instead of trying to conquer the world, just pick 1 file that you want to test with and copy that and share the output:

[felix@gemini ~]$ rclone copy -vv /etc/hosts GD:
2019/04/09 07:18:29 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:"]
2019/04/09 07:18:29 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:18:30 DEBUG : hosts: Size and modification time the same (differ by -56.293µs, within tolerance 1ms)
2019/04/09 07:18:30 DEBUG : hosts: Unchanged skipping
2019/04/09 07:18:30 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 0, -
Elapsed time:       500ms

2019/04/09 07:18:30 DEBUG : 4 go routines active
2019/04/09 07:18:30 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:"]

and my dry-run output:

[felix@gemini ~]$ rclone copy -vv /etc/hosts GD: --dry-run
2019/04/09 07:19:28 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:" "--dry-run"]
2019/04/09 07:19:28 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:19:28 DEBUG : hosts: Size and modification time the same (differ by -56.293µs, within tolerance 1ms)
2019/04/09 07:19:28 DEBUG : hosts: Unchanged skipping
2019/04/09 07:19:28 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 0, -
Elapsed time:       600ms

2019/04/09 07:19:28 DEBUG : 4 go routines active
2019/04/09 07:19:28 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:" "--dry-run"]

Unless there is a specific fix you are testing, just use the stable version as it makes thing harder to debug by using a beta.

I use the beta because it’s what people at unraid forum’s recommend, it has been working fine till now.

I think I found the reason for my problem.

If you use --no-traverse, rclone will not list the directory, so it can’t know what files are present or not.
removed it and it skiped tons of files :slight_smile:

“Why” did they recommend though? What’s the fix in the beta that is there? That’s the key.

Can you also share an example of your command and no-traverse not working? I don’t think that should be the case as it can’t list a directory but still shouldn’t copy the file. I’m not sure what your command and output looks like though as I can’t seem to replicate it.

I put a directory in another one and ran with no-traverse and it properly skips the files already present.

[felix@gemini dir1]$ rclone copy -vv /home/felix/dir1 GD:dir1 --no-traverse
2019/04/09 07:35:30 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/home/felix/dir1" "GD:dir1" "--no-traverse"]
2019/04/09 07:35:30 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:35:31 DEBUG : dir2/1: Size and modification time the same (differ by -485.807µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/2: Size and modification time the same (differ by -828.82µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/2: Unchanged skipping
2019/04/09 07:35:31 INFO  : Google drive root 'dir1': Waiting for checks to finish
2019/04/09 07:35:31 DEBUG : dir2/hosts: Size and modification time the same (differ by -804.361µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/hosts: Unchanged skipping
2019/04/09 07:35:31 DEBUG : dir2/1: Unchanged skipping
2019/04/09 07:35:31 INFO  : Google drive root 'dir1': Waiting for transfers to finish
2019/04/09 07:35:31 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 3 / 3, 100%
Transferred:            0 / 0, -
Elapsed time:        1.4s

2019/04/09 07:35:31 DEBUG : 4 go routines active
2019/04/09 07:35:31 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/home/felix/dir1" "GD:dir1" "--no-traverse"]

I’m not sure why it’s recommended.
One of the places I see it is here:

As far as I understand this Spaceinvader guy is a veteran in the unraid forum.

And looking at your output, it’s WAY better to understand compared to the beta. Maybe I should go to the stable release indeed.

Right now everything is working proplerly with this command:
rclone copy -P --no-update-modtime --size-only --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 9–bwlimit 9500K /mnt/user/PLEX/TV\ Shows/ secure:STREAM

-P is way more elegant display in the shell compared to --stats 1s.

By the way, one thing I noticed is that files don’t get overwritten in Google Drive except the upload is finished.
When rclone was starting the transfer to those Band of Brothers episodes and I would cancel with ctrl+c. The files would still be present and playable.

So, if I understand rclone settings if you set it as default

  • it will check for size AND mod time. If size is exactly the same but mod time is different it will replace the file.
  • using --size-only will check only the size and if that matches will skip files.
  • using --checksum is not possible with crypt (that’s a shame it would be very handy to have it) and will use size only.

Since I use emby for streaming instead of Plex it creates .nfo files for each episode and tv show.
For some reason those are still copied no matter what. I guess emby updates it constantly?
I have no idea.

Eitherway, right now, from a folder with tons of TV Shows it’s copying some that starts with R and T.
So I guess it worked as I was hopping.

Emby doesn’t create nfo files by deafult. That’s an option you turn on in Emby yourself.

Dashboard-Library-select each library-enable Show Advanced settings-scroll to Metadata Savers

I personally always kept that off when I was tesitng Emby.

You can’t resume files in Google Drive. If a transfer aborts, you have to re-upload it so that’s why you don’t see anything until it is finished.

So you aren’t able to share a use case of --no-traverse not working with some output as I still can’t understand your issue or why it wouldn’t work as I can’t reproduce it.

I’m not sure either. I started using rclone copy with rclone browser and copying the commands from it and adapting it to my unraid shell.

This setting would cause files to still be copied:
rclone copy --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --stats 1s /mnt/user/PLEX/TV\ Shows/ secure:STREAM

This also:
rclone copy -vv --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --dry-run /mnt/user/PLEX/TV\ Shows/ secure:STREAM

rclone copy --verbose --transfers 7 --checkers 4 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --bwlimit 8700K --stats 1s --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

rclone copy -vv --stats 2s --no-update-modtime --size-only --fast-list --no-traverse --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries6 --low-level-retries 15 --tpslimit 5 --bwlimit 8700K --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Every attempt I did caused files to still be copied.
When I switched to this:
rclone copy -P --no-update-modtime --size-only --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 9–bwlimit 9500K /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Everything worked.
I don’t know if it’s a bug or if I screwed something up somewhere (which is way more likely).

Can you run that command with -vv and share the output? If you let it copy a few files, that should be fine as it doesn’t need to run to completion.

Ok, will do it, I just want to finish the transfers I’m doing at the moment and will get back to this after it’s finished.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.