Advice wanted for 6xxxxx (lot s of ) small files non-continuously upload to g suit? thx

hi~

I got 1TB data, (cppcryptfs encrypted, only make files name longer)
Total objects: 646219
Total size: 984.493 GBytes (1057091213791 Bytes)

I upload only when I am at home, may be about 40hr per session,
then i am out for 8hr, then repeat.

I am using this command to upload:

/snap/bin/rclone --config /home/xxxx/grnprogs/rclone/rclone-cur.conf copy --ignore-existing --verbose --transfers 40 --checkers 80 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s /media/xxxx/asdvd005-cp2r1-cfs/20180524-004349—asdvd005-cp2r1-iat011 target:base-a002/20180527-01/20180524-004349—asdvd005-cp2r1-iat011

say yesterday,
the -v displayed
checked 240000 files
transferred 120000 files.

just wanna ask some questions which I cant google and get.

  1. i have to upload dis-continuously, so each time it will spend LOT’s of
    time checking if the files are already on the target first?

  2. so after each session, it’s more and more un-efficient?

  3. the above command is default from rclone-browser, i
    enlarged the transfer and check option by 10x,
    any other parameter may help?

thank you very much.

ps:
that 1 TB… I have uploaded for a week but still only 800GB out of 1TB uploaded.
Gsuit’s “750GB” per day limit is indeep not the bottle neck.

ps2:
I have tried making my own google ID,
but somehow it say unauthorized xxx in the upload,
so I turn back to using the default ID.

thanks

it’s slow but seems acceptable.

today it spent 8hr to checked that 4xxxxx files are the same,
and uploading the remaining from the 6xxxxx files.

I got another drive of 4TB which have over 5 xxx xxx files,
that will really be a challenge to my system.

thanks

Yes. This phase is particularly slow with google drive compared to other remotes.

You can investigate using the cache backend here as one of the things it does is cache the directory structure.

I think you’ve probably got the right parameters. There seems to be a rate limit of about 2 transfers per second with google drive.

I recommend you have another go since rclone’s ID is somewhat oversubscribed and google refused to give me any more quota :frowning: