RClone Copy to Gdrive failing to identify already uploaded files

#1

So, I decided to start uploading my Plex folder to Gdrive but I’ve hit the 750GB daily limit.

Ok, no big deal, I’ve waited 24 hours and just issue the same command. I was hopping rclone would identify the already uploaded files but it seems I’m wrong.

This is the command I’m running

rclone copy --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --stats 1s /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Is there something wrong I`m doing?

It would be really helpful I could just type the same command once a week without having to remember what is actually new on the server.

0 Likes

#2

It will do just that. It’ll compare mod time and size or checksum depending on the remote type. If you turn on debug -vv you’ll see why it’s deciding to re-upload.

0 Likes

#3

Based on that:

  -c, --checksum                                     Skip based on checksum (if available) & size, not mod-time & size

This will not check mod-time and check only size and checksum.

Do you know if checksum is available with gdrive encrypted with crypt?

0 Likes

#4

Checksum is not available for crypt

0 Likes

#5

Ok tried again and it IS copying everything again.

I can’t see from the verbose why it’s copying.

0 Likes

#6

If you can share the -vv output, it should write in there why it is being copied.

0 Likes

#7
> root@Tower:/mnt/user/PLEX# rclone copy -vv --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10  --no-traverse --dry-run  /mnt/user/PLEX/TV\ Shows/ secure:STREAM
> 2019/04/09 07:48:13 DEBUG : rclone: Version "v1.46.0-112-g1c301f9f-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "-vv" "--transfers" "10" "--checkers" "5" "--contimeout" "60s" "--timeout" "300s" "--retries" "6" "--low-level-retries" "15" "--tpslimit" "10" "--no-traverse" "--dry-run" "/mnt/user/PLEX/TV Shows/" "secure:STREAM"]
> 2019/04/09 07:48:13 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
> 2019/04/09 07:48:13 INFO  : Starting HTTP transaction limiter: max 10 transactions/s with burst 1
> 2019/04/09 07:48:15 NOTICE: Band of Brothers/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Avatar - The Last Airbender/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Aqua Teen Hunger Force/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Betty Boop/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Batman - The Animated Series/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Bobby's World/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Cardcaptor Sakura/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Carnivàle/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Chapolin Colorado/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:16 NOTICE: Comedy Central Presents/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:17 NOTICE: Courage the Cowardly Dog/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:17 NOTICE: Cow and Chicken/tvshow.nfo: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/Betty Boop - S1930E01 - Dizzy Dishes.mkv: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/season.nfo: Not copying as --dry-run
> 2019/04/09 07:48:19 NOTICE: Betty Boop/Season 1930/Betty Boop - S1930E01 - Dizzy Dishes.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E01 - Currahee.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E02 - Day of Days.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E03 - Carentan.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E05 - Crossroads.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E06 - Bastogne.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E07 - The Breaking Point.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E08 - The Last Patrol.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E08 - The Last Patrol.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E09 - Why We Fight.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E10 - Points.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E10 - Points.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E09 - Why We Fight.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E07 - The Breaking Point.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E03 - Carentan.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E06 - Bastogne.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E05 - Crossroads.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.nfo: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E02 - Day of Days.mkv: Not copying as --dry-run
2019/04/09 07:48:27 NOTICE: Band of Brothers/Season 01/Band of Brothers - S01E01 - Currahee.mkv: Not copying as --dry-run

I’m doing a dry run now but it’s the same output, just instead it will copy the files over and over again.

0 Likes

#8

For the first time I’ve seen Rclone crash

goroutine 60 [select]:
net/http.(*persistConn).roundTrip(0xc0002ad9e0, 0xc003e34ff0, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/transport.go:2174 +0x75c
net/http.(*Transport).roundTrip(0xc00014a240, 0xc00010ed00, 0x137b524, 0xa, 0xc003d09150)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/transport.go:481 +0xa1b
net/http.(*Transport).RoundTrip(...)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/roundtrip.go:17
github.com/ncw/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0001271d0, 0xc00010ed00, 0xc00010ed00, 0x0, 0x10)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:324 +0x1e3
github.com/ncw/rclone/vendor/golang.org/x/oauth2.(*Transport).RoundTrip(0xc00011a5a0, 0xc00010ec00, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/golang.org/x/oauth2/transport.go:56 +0xfc
net/http.send(0xc00010ec00, 0x15de4c0, 0xc00011a5a0, 0x0, 0x0, 0x0, 0xc00000e350, 0xc003d08dc0, 0x1, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc00011a5d0, 0xc00010ec00, 0x0, 0x0, 0x0, 0xc00000e350, 0x0, 0x1, 0x5a25a75a1d51beec)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc00011a5d0, 0xc00010ec00, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
        /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/net/http/client.go:509
github.com/ncw/rclone/backend/drive.(*resumableUpload).transferChunk(0xc000125440, 0x800000, 0x15ee640, 0xc000609480, 0x800000, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:167 +0xaa
github.com/ncw/rclone/backend/drive.(*resumableUpload).Upload.func1(0x0, 0x0, 0x8)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:214 +0x13e
github.com/ncw/rclone/fs.pacerInvoker(0x1, 0xf, 0xc0006094c0, 0xc0006094c0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fs.go:1189 +0x3c
github.com/ncw/rclone/lib/pacer.(*Pacer).call(0xc000118240, 0xc0006094c0, 0xf, 0x40, 0x12e30a0)
        /home/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:197 +0x8b
github.com/ncw/rclone/lib/pacer.(*Pacer).Call(0xc000118240, 0xc0006094c0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:216 +0x78
github.com/ncw/rclone/backend/drive.(*resumableUpload).Upload(0xc000125440, 0xc000125440, 0x8, 0xc003ef2000)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:212 +0x2a0
github.com/ncw/rclone/backend/drive.(*Fs).Upload(0xc000177800, 0x15dd1e0, 0xc00031b080, 0x494e0a0e7, 0x138f561, 0x18, 0xc0013d2030, 0x21, 0xc003ecc960, 0x9d, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/upload.go:113 +0x5bf
github.com/ncw/rclone/backend/drive.(*baseObject).update(0xc000100100, 0xc0047fe300, 0x138f561, 0x18, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0x0, 0x0, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:2669 +0x2d4
github.com/ncw/rclone/backend/drive.(*Object).Update(0xc000100100, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0xc000388870, 0x1, 0x1, 0x18, 0x128f060)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:2683 +0x181
github.com/ncw/rclone/backend/drive.(*Fs).Put(0xc000177800, 0x15dd1e0, 0xc00031b080, 0x160bac0, 0xc00031b0a0, 0xc000388870, 0x1, 0x1, 0x20, 0x128f060, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/drive/drive.go:1641 +0x201
github.com/ncw/rclone/backend/crypt.(*Fs).put(0xc0002fa140, 0x15dd1c0, 0xc00050a780, 0x147644a0d698, 0xc00064c9c0, 0xc000388870, 0x1, 0x1, 0xc0013e9c30, 0x0, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:357 +0x195
github.com/ncw/rclone/backend/crypt.(*Fs).Put(0xc0002fa140, 0x15dd1c0, 0xc00050a780, 0x147644a0d698, 0xc00064c9c0, 0xc000388870, 0x1, 0x1, 0x1611f20, 0xc000346e20, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:389 +0xd5
github.com/ncw/rclone/fs/operations.Copy(0x1611ea0, 0xc0002fa140, 0x0, 0x0, 0xc00003bbd0, 0x43, 0x16128a0, 0xc00064c9c0, 0x1611f20, 0xc000346e20, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/operations/operations.go:317 +0x14c4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc0000aa000, 0xc0001209c0, 0x1611ea0, 0xc0002fa140, 0xc0000aa0d8)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:295 +0x251
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startTransfers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:321 +0x9f

Just did this:
rclone copy --verbose --transfers 7 --checkers 4 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --bwlimit 8700K --stats 1s --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

And got that

EDIT:
Ok, I can see what you are talking about. It says new besides the copy process.

Like so:
2019/04/09 08:13:48 INFO : Band of Brothers/Season 01/Band of Brothers - S01E04 - Replacements.nfo: Copied (new)

The thing is I’m running like this now:
rclone copy -vv --stats 2s --no-update-modtime --size-only --fast-list --no-traverse --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries6 --low-level-retries 15 --tpslimit 5 --bwlimit 8700K --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

I’m using every option to avoid it to compare dates and time, to ignore existing files but it doesn’t matter.
Rclone is doing whatever it thinks it should.

0 Likes

#9

Instead of trying to conquer the world, just pick 1 file that you want to test with and copy that and share the output:

[felix@gemini ~]$ rclone copy -vv /etc/hosts GD:
2019/04/09 07:18:29 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:"]
2019/04/09 07:18:29 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:18:30 DEBUG : hosts: Size and modification time the same (differ by -56.293µs, within tolerance 1ms)
2019/04/09 07:18:30 DEBUG : hosts: Unchanged skipping
2019/04/09 07:18:30 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 0, -
Elapsed time:       500ms

2019/04/09 07:18:30 DEBUG : 4 go routines active
2019/04/09 07:18:30 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:"]

and my dry-run output:

[felix@gemini ~]$ rclone copy -vv /etc/hosts GD: --dry-run
2019/04/09 07:19:28 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:" "--dry-run"]
2019/04/09 07:19:28 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:19:28 DEBUG : hosts: Size and modification time the same (differ by -56.293µs, within tolerance 1ms)
2019/04/09 07:19:28 DEBUG : hosts: Unchanged skipping
2019/04/09 07:19:28 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 0, -
Elapsed time:       600ms

2019/04/09 07:19:28 DEBUG : 4 go routines active
2019/04/09 07:19:28 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/etc/hosts" "GD:" "--dry-run"]

Unless there is a specific fix you are testing, just use the stable version as it makes thing harder to debug by using a beta.

0 Likes

#10

I use the beta because it’s what people at unraid forum’s recommend, it has been working fine till now.

I think I found the reason for my problem.

If you use --no-traverse, rclone will not list the directory, so it can’t know what files are present or not.
removed it and it skiped tons of files :slight_smile:

0 Likes

#11

“Why” did they recommend though? What’s the fix in the beta that is there? That’s the key.

0 Likes

#12

Can you also share an example of your command and no-traverse not working? I don’t think that should be the case as it can’t list a directory but still shouldn’t copy the file. I’m not sure what your command and output looks like though as I can’t seem to replicate it.

I put a directory in another one and ran with no-traverse and it properly skips the files already present.

[felix@gemini dir1]$ rclone copy -vv /home/felix/dir1 GD:dir1 --no-traverse
2019/04/09 07:35:30 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "-vv" "/home/felix/dir1" "GD:dir1" "--no-traverse"]
2019/04/09 07:35:30 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/09 07:35:31 DEBUG : dir2/1: Size and modification time the same (differ by -485.807µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/2: Size and modification time the same (differ by -828.82µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/2: Unchanged skipping
2019/04/09 07:35:31 INFO  : Google drive root 'dir1': Waiting for checks to finish
2019/04/09 07:35:31 DEBUG : dir2/hosts: Size and modification time the same (differ by -804.361µs, within tolerance 1ms)
2019/04/09 07:35:31 DEBUG : dir2/hosts: Unchanged skipping
2019/04/09 07:35:31 DEBUG : dir2/1: Unchanged skipping
2019/04/09 07:35:31 INFO  : Google drive root 'dir1': Waiting for transfers to finish
2019/04/09 07:35:31 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 3 / 3, 100%
Transferred:            0 / 0, -
Elapsed time:        1.4s

2019/04/09 07:35:31 DEBUG : 4 go routines active
2019/04/09 07:35:31 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "-vv" "/home/felix/dir1" "GD:dir1" "--no-traverse"]
0 Likes

#13

I’m not sure why it’s recommended.
One of the places I see it is here:

As far as I understand this Spaceinvader guy is a veteran in the unraid forum.

And looking at your output, it’s WAY better to understand compared to the beta. Maybe I should go to the stable release indeed.

Right now everything is working proplerly with this command:
rclone copy -P --no-update-modtime --size-only --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 9–bwlimit 9500K /mnt/user/PLEX/TV\ Shows/ secure:STREAM

-P is way more elegant display in the shell compared to --stats 1s.

By the way, one thing I noticed is that files don’t get overwritten in Google Drive except the upload is finished.
When rclone was starting the transfer to those Band of Brothers episodes and I would cancel with ctrl+c. The files would still be present and playable.

So, if I understand rclone settings if you set it as default

  • it will check for size AND mod time. If size is exactly the same but mod time is different it will replace the file.
  • using --size-only will check only the size and if that matches will skip files.
  • using --checksum is not possible with crypt (that’s a shame it would be very handy to have it) and will use size only.

Since I use emby for streaming instead of Plex it creates .nfo files for each episode and tv show.
For some reason those are still copied no matter what. I guess emby updates it constantly?
I have no idea.

Eitherway, right now, from a folder with tons of TV Shows it’s copying some that starts with R and T.
So I guess it worked as I was hopping.

0 Likes

#14

Emby doesn’t create nfo files by deafult. That’s an option you turn on in Emby yourself.

Dashboard-Library-select each library-enable Show Advanced settings-scroll to Metadata Savers

I personally always kept that off when I was tesitng Emby.

You can’t resume files in Google Drive. If a transfer aborts, you have to re-upload it so that’s why you don’t see anything until it is finished.

So you aren’t able to share a use case of --no-traverse not working with some output as I still can’t understand your issue or why it wouldn’t work as I can’t reproduce it.

0 Likes

#15

I’m not sure either. I started using rclone copy with rclone browser and copying the commands from it and adapting it to my unraid shell.

This setting would cause files to still be copied:
rclone copy --verbose --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --stats 1s /mnt/user/PLEX/TV\ Shows/ secure:STREAM

This also:
rclone copy -vv --transfers 10 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --dry-run /mnt/user/PLEX/TV\ Shows/ secure:STREAM

rclone copy --verbose --transfers 7 --checkers 4 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 10 --no-traverse --bwlimit 8700K --stats 1s --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

rclone copy -vv --stats 2s --no-update-modtime --size-only --fast-list --no-traverse --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries6 --low-level-retries 15 --tpslimit 5 --bwlimit 8700K --ignore-existing /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Every attempt I did caused files to still be copied.
When I switched to this:
rclone copy -P --no-update-modtime --size-only --transfers 5 --checkers 5 --contimeout 60s --timeout 300s --retries 6 --low-level-retries 15 --tpslimit 9–bwlimit 9500K /mnt/user/PLEX/TV\ Shows/ secure:STREAM

Everything worked.
I don’t know if it’s a bug or if I screwed something up somewhere (which is way more likely).

0 Likes

#16

Can you run that command with -vv and share the output? If you let it copy a few files, that should be fine as it doesn’t need to run to completion.

0 Likes

#17

Ok, will do it, I just want to finish the transfers I’m doing at the moment and will get back to this after it’s finished.

0 Likes