Rclone copy, doesn't transfer the files, but is using the ram memory

[suneel_giree]

What is the problem you are having with rclone?

Incomplete upload, even though there is fast internet connection. the download doesn't stop.

What is your rclone version (output from rclone version)

yes latest version. rclone v1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy D:\AS mylove:AS

The rclone config contents with secrets removed.

D:\rclone-v1.56.0-windows-amd64\rclone-v1.56.0-windows-amd64>rclone config
Current remotes:

Name                 Type
====                 ====
myl                  drive
mylove               drive

A log from the command with the -vv flag

2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_favicon.ico: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_index.html: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_karma.conf.js: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_main.ts: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_polyfills.ts: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_styles.css: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_test.ts: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tsconfig.app.json: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tsconfig.spec.json: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tslint.json: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_styles.css: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_test.ts: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tsconfig.app.json: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tsconfig.spec.json: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/02 The Basics/__MACOSX/basics-assignment-3-solution/src/_tslint.json: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_.DS_Store: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_karma.conf.js: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_karma.conf.js: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_app: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_app: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_polyfills.ts: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_assets: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_browserslist: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_environments: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_index.html: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_favicon.ico: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_.DS_Store: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_main.ts: Size and modification time the same (differ by 0s, within tolerance 1ms)
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_polyfills.ts: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_assets: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_browserslist: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_environments: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_index.html: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_favicon.ico: Unchanged skipping
2021/08/29 12:05:59 DEBUG : angular-complete-guide/06 Course Project - Components & Databinding/__MACOSX/prj-basics-final/src/_styles.css: Size and modification time the same (differ by 0s, within tolerance 1ms)

2021/08/29 12:06:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/29 12:06:09 DEBUG : pacer: Rate limited, increasing sleep to 1.563639697s
2021/08/29 12:06:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/29 12:06:09 DEBUG : pacer: Rate limited, increasing sleep to 2.755194706s
2021/08/29 12:06:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/29 12:06:09 DEBUG : pacer: Rate limited, increasing sleep to 4.106152663s
2021/08/29 12:06:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/29 12:06:09 DEBUG : pacer: Rate limited, increasing sleep to 8.469052447s
2021/08/29 12:06:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/08/29 12:06:09 DEBUG : pacer: Rate limited, increasing sleep to 16.144493701s
2021/08/29 12:06:11 DEBUG : pacer: Reducing sleep to 0s


Lots of like this. It never ended.  I had to stop it. 

[asdffdsa (jojothehumanmonkey)]

hello,

can you post the contents of the config file, make sure to redact id/secret/token?
rclone config show

as per the debug log
error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
rclone is sending too many api requests too quickly and gdrive is forcing rclone to slow down.
this is normal behavior.

be default, all rcloners using gdrive, share the same client id, leading to these kind of problems
so make sure to create your own client id,
https://rclone.org/drive/#making-your-own-client-id
"When you use rclone with Google drive in its default configuration you are using rclone's client_id. This is shared between all the rclone users. There is a global rate limit on the number of queries per second "

[Ole (Ole Frost)]

Hi suneel_giree,

Your token and client secrets are publicly available to anybody pressing the pen icon in the upper right corner of your post above (including malicious web crawlers). This information gives access to your data without username, password and 2FA.

I therefore recommend you immediately log into https://myaccount.google.com/permissions and revoke access for your personal client ID and then recreate your rclone remote to get access again.

[Ole (Ole Frost)]

Deleting the post isn't enough, sorry

[suneel_giree]

Even though i edited the keys then posted(random letters), i will take extra step to be safe. FRom the link you gave me, i saw only the apps, no client ids. THank you for your help.

[Ole (Ole Frost)]

@suneel_giree

posted(random letters)

OK, then it may be a non issue, but better safe than sorry. Next time write ***** or REDACTED, then we know

@suneel_giree

i saw only the apps, no client ids

It is my understanding that deleting the (personal) rclone app will invalidate all your tokens associated with it. It will be recreated again when you recreate your remote.

[suneel_giree]

Thank you so much. You guys are awesome!

@asdffdsa @Ole , i tried with client id before but still it didn't fix the problem. I shouldn't have tried to transfer 392 gb folder all at once? 3 gb of transfer left, i don't know from which part of the folder it is(all courses), and i have having hard time with it.

[Ole (Ole Frost)]

I guess it will complete eventually. It may just take some time if you have many files and folders. So you need to be (very) patient.

Here are a few tricks that may make it faster:

Fastest way to check for changes in 2.5+ Million files?

Here are some additional ideas to speed up the checking: Make sure you have your own Client ID Remove --tpslimit=10 (unless a test has shown that it is needed, if so then ignore the other ideas too) Add --drive-pacer-min-sleep=10ms (to use the latest limits from Google) Change --checkers=16 (to increase concurrency) I don’t know how they will play with all the other parameters. I would perform some tests to find the 2-3 parameters that makes the most difference and then leave the rest at d…

You can also add --log-level=INFO and --progress to monitor progress.

PS: It will start checking everything from the beginning in case you interrupt and then start again. This may also trigger the Google throttling, that you see as pacer entries in the log.

@asdffdsa Sorry for the interruption, just jump in again, if you fancy

[Ole (Ole Frost)]

Bonus tip: --fast-list, please read about pros/cons before using it