Sync/Copy Fails?

Help and Support
1

Hey All,

So I have a local folder ("A") that I am trying to sync to Google Drive. I run a sync command with the -vv command line argument and it runs to completion without any errors. But when I run the exact same command it says there is data to copy (about 19 GB). So I let it sync. Then .... guess what. When I run it again there is 19 GB to copy... and so on. It always seems like there is something to copy. There are NO errors or or API issues, etc in the debug log. Thoughts?

My command is
rclone sync /mnt/merged/FOLDERA gdmedia:FOLDERA --log-file /root/rclone_upload.log -vv --exclude-from /opt/rclone/rclone_excludes -c --fast-list --bwlimit "08:00,24M 22:00,off"

-Eric

2

The questions in the template are the best way to get started.

What version are you running?
What OS?
Seems to be Google Drive.

Can you run the command with -vv and you can look at the log and it'll tell you why it's being copied or we can help you look at the log if you share it.

3

hello and welcome to the forum,

perhaps run the command with https://rclone.org/docs/#n-dry-run and post the debug log.

you say it is a local folder, but it is in /mnt?
can you tell us more about what that is?
/mnt/merged/FOLDERA

4

Sorry.

Running the command inside of a screen session so I can leave it running.

Version:
rclone v1.52.1-072-g4808958f-beta
- os/arch: linux/amd64
- go version: go1.14.4

OS:
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

The folder I am copying is a mergerfs folder that is a mount containing two local drives which are using snapraid.

I ran it with a dry run, I see a bunch of files that it claimed were identical and skipped, and then a bunch that it needed to upload.

Transferred: 0 / 22.383 GBytes, 0%, 0 Bytes/s, ETA -
Checks: 18 / 192, 9%
Transferred: 0 / 118, 0%
Elapsed time: 1m56.1s
Checking:

etc would pop up randomly as it scanned the files.

For each file that it said was the same I would get

2020/06/16 11:40:17 DEBUG : FILE.avi: MD5 = 190c4de4f0303fec5eff544bd82c18df OK
2020/06/16 11:40:17 DEBUG : FILE.avi Size and MD5 of src and dst objects identical
2020/06/16 11:40:17 DEBUG : FILE.avi: Unchanged skipping

Then I got a bunch of files it needed to upload.... what's odd is that I have uploaded more than 222 files, but that seems to be the number of "checks" it always has.

Those lines looked like

2020/06/16 11:42:10 NOTICE: FILE2.avi: Skipped copy as --dry-run is set
2020/06/16 11:42:10 NOTICE: FILE3.avi: Skipped copy as --dry-run is set
2020/06/16 11:42:10 NOTICE: FILE4.avi: Skipped copy as --dry-run is set

... etc.

Then I got:

Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 192 / 192, 100%
Transferred: 118 / 118, 100%
Elapsed time: 5m46.0s

2020/06/16 11:42:10 DEBUG : 5 go routines active

I'll run it again without the dry run set and post back....

5

Update: Its running now but whats interesting is that I get these status updates
Transferred: 0 / 20.678 GBytes, 0%, 0 Bytes/s, ETA -
Checks: 56 / 202, 28%
Transferred: 0 / 108, 0%
Elapsed time: 2m57.9s
Checking:

So.... this time there were 202 checks but last time 192, but nothing was uploaded in between....

It's now uploading so I'll just let it go... no errors.... Just a bunch of lines like this:
2020/06/16 11:55:32 DEBUG : FILE5.avi: Sending chunk 100663296 length 8388608
2020/06/16 11:55:34 DEBUG : FILE6.avi: Sending chunk 100663296 length 8388608

6

Update #2.... finished.

Transferred: 20.678G / 20.678 GBytes, 100%, 3.457 MBytes/s, ETA 0s
Checks: 202 / 202, 100%
Transferred: 108 / 108, 100%
Elapsed time: 1h42m4.7s

2020/06/16 13:26:38 DEBUG : 11 go routines active

It shows that it copied a bunch of files successfully with now matching md5s. No errors at all.

So in theory when I re-run the same command it shouldn't have to copy anything.

So I re-run the dry-run command just to see....

rclone sync /mnt/merged/FOLDERA gdmedia:FOLDERA -vv --exclude-from /opt/rclone/rclone_excludes -c --fast-list --bwlimit "08:00,24M 22:00,off" --check-first -n | tee SYNC_DEBUG2

And low and behold....

Transferred: 0 / 19.123 GBytes, 0%, 0 Bytes/s, ETA -
Checks: 4 / 214, 2%
Transferred: 0 / 96, 0%
Elapsed time: 53.7s
Checking:

But no files have changed.... Thoughts?

7

those log snippets,
are you wating until the command has finished before posting it?

8

Last thing I found... in the logs....

2020/06/16 13:26:18 DEBUG : FILE.avi: Sending chunk 109051904 length 8388608
2020/06/16 13:26:20 DEBUG : FILE.avi: Sending chunk 117440512 length 8388608
2020/06/16 13:26:22 DEBUG : FILE.avi: Sending chunk 125829120 length 8388608
2020/06/16 13:26:25 DEBUG : FILE.avi: Sending chunk 134217728 length 8388608
2020/06/16 13:26:27 DEBUG : FILE.avi: Sending chunk 142606336 length 8388608
2020/06/16 13:26:29 DEBUG : FILE.avi: Sending chunk 150994944 length 8388608
2020/06/16 13:26:31 DEBUG : FILE.avi: Sending chunk 159383552 length 8388608
2020/06/16 13:26:31 INFO :
Transferred: 20.658G / 20.678 GBytes, 100%, 3.458 MBytes/s, ETA 5s
Checks: 202 / 202, 100%
Transferred: 107 / 108, 99%
Elapsed time: 1h41m57.9s
Transferring:

  • FILE.avi: 88% /174.537M, 2.686M/s, 7s

2020/06/16 13:26:33 DEBUG : FILE.avi: Sending chunk 167772160 length 8388608
2020/06/16 13:26:35 DEBUG : FILE.avi: Sending chunk 176160768 length 6854656
2020/06/16 13:26:38 DEBUG : FILE.avi: MD5 = 984c48b2aaaa9a2a8e1057021747bbdf OK
2020/06/16 13:26:38 INFO : FILE.avi: Copied (new)

Then in the next run its "missing" again....

2020/06/16 14:48:03 NOTICE: FILE.avi: Skipped copy as --dry-run is set

So even though I seem to see it upload with no errors, the next run it seems to be missing and wants to copy it all over again.

But even more interesting is if I browse google drive I see the file sitting there.

9

Yes... I am running to completetion before posting.

10

With --fast-list
Transferred: 0 / 20.504 GBytes, 0%, 0 Bytes/s, ETA -
Checks: 7 / 203, 3%
Transferred: 0 / 107, 0%
Elapsed time: 58.5s
Checking:

Without....

Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 32 / 310, 10%
Elapsed time: 59.0s
Checking:

So it seems like --fast-list is somehow missing files.

11

Seems like this may be the issue:

It's a shared drive and fast list is missing files.

12

You're pasting a lot of snippets of a log. If you want some help, we really need the whole debug log. Would be ideal to have it from two runs.

13

It's all good. Was an issue with fast-list which seems fixed now/

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.