Copy everything (from gdrive to termux)

cregox · January 2, 2021, 6:24pm

not sure this belongs to support, but it sure is a request for help.

got about 50gb to copy from gdrive (this is probably irrelevant, but my gdrive is way over its quota now i stopped paying for it).

the problem is i also got over 3000 errors after trying to sync twice (didn't notice at first that the each time it was actually 3 retries done already by default, and that i actually wanted to copy, not sync), of which:

303 are NOTICES of duplicated files (already tried to now use dedupe to fix this, oddly it took the expected 44min but got 7 errors which i could manually fix and 88 notices which might already be nearly too much to go manual), and;
most of everything else seem to be related with unaccepted file names (but there are other errors as well).

all in all, apparently over 97% of the errors are related to file names.

in case this is really relevant here:

rclone v1.53.3-DEV
os/arch: android/arm64
go version: go1.15.3

main question

is there any way to just and simply copy ALL files no matter what? rename them for whatever name can be accepted, "deduped", etc.

secondary

in case there's no answer to the main question, as i would expect, this could also help me and couldn't find an answer yet: how can i copy over those ill named files?

actually, at this point it's just a guess this is indeed the issue. so, this is the most common error message i could find:

ERROR : (∆16G)to organize/(∆3.7G)mac+documents/zone/BlahTherapy - May "premier plane".rtfd/TXT.rtf: Failed to copy: open /data/data/com.termux/files/home/achoz/.nosync/toreview2020dec/(∆16G)to organize/(∆3.7G)mac+documents/zone/BlahTherapy - May "premier plane".rtfd/TXT.rtf: operation not permitted

above all

thanks so much for this amazing little tool! it works mostly wonderfully, even before i learned so much more about it. ;-*

asdffdsa · January 2, 2021, 6:54pm

hello,

when you posted, you should have been asked for information?
what is the command you are using?
what is the dest, an android phone?

cregox · January 2, 2021, 7:03pm

thanks for the feedback!

yes, but i believe i gave in all relevant information already. and then some more. i was hoping the question is clearer as it is.

do you really feel the need for redundancy here?

i'm using several commands from sync to dedupe and copy. the issue isn't specific to any of them, i got the same issues with them all.

termux is only for android, as far as i know, yes. but i bet this issue can be found among many different destinations and origins.

ncw · January 6, 2021, 2:31pm

The first thing to do is to get the rclone dedupe working properly with no errors. Can you run that and post the log output? Or at minimum the command line you used and the errors you got.

Thanks

cregox · January 6, 2021, 3:46pm

already done back then. copied.txt (14.4 KB)

here's the full log + errors and command from the first time. bakerr.txt (3.9 MB)
rclone sync gdrive:toreview2020dec .

not sure how either can be that helpful or add any insight from what i already posted, though.

still haven't done much, or anything, ever since... might get back to it today, or soon enough.

and thank you (both) for trying to help!

ncw · January 7, 2021, 11:18am

I see in your copied.txt log

2021-01-02 17:44:37 ERROR : XXX: Failed to rename: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions

So that is one reason the de-dupe is failing. Are these files owned by another user maybe?

2020/12/31 17:47:28 ERROR : XXX.png: Failed to copy: open data/data/com.termux/files/home/achoz/.../XXX.png: operation not permitted

These errors look like permissions errors on Android - termux not being able to open those files for some reason. Can you (say) using head on one of those files in termux and see what it says there? What do the permissions look like on that file?

cregox · January 7, 2021, 2:30pm

yeah, i saw those too but don't worry about those 7 errors there. as i mentioned, they're just 7, i can manually handle them later on. if you're still curious i can report back later. they're apparently completely unrelated to my main concerns... unless you think they are related, i prefer to let it be for now and focus on the rest.

ncw · January 7, 2021, 7:42pm

What is your main concern? Can you send a log for that?

cregox · January 8, 2021, 10:03am

sorry, but that sounded a lot like a joke to me!

first post here has a big "main question" with that answer. there should be no log needed for that.

ncw · January 9, 2021, 12:44pm

I'm just trying to understand the problem!

OK

We've gone over duplicated files

I haven't seen any error message about file names

A copy with rename has been suggested in the past but it doesn't exist yet

That looks like a permissions error as discussed above.

I guess it is possible that it is related to unusual characters in the name, but I would have thought that android would be fine with any characters in a file name since it is linux based.

So as far as I can see we've talked about all the individual problems you've brought up.

Hopefully I've answered that.

I don't think we've got to the bottom of those operation not permitted errors which are the root cause of your problem as far as I can see.

Can you create a file or directory like the one you are trying to copy using the tools in termux? mkdir cat etc?

cregox · January 10, 2021, 10:53am

absolutely awesome!!

thanks a lot for this.

if you had all this in your head before, i now feel bad for laughing on your previous question. i'll be reading everything again many times now to see what i may have missed.

i also feel you didn't need to go in that much details, but i'll also re analyse this...

meanwhile...

yes, creating a folder /data/data/com.termux/files/home/achoz/.nosync/toreview2020dec/(∆16G)to organize/(∆3.7G)mac+documents/zone/BlahTherapy - May "premier plane".rtfd/TXT.rtf using just mkdir -p ' and closing quotes later worked flawlessly (i'm on telegram termux group right now discussing a caveat there because i first tried it, "wrongly" so, using ", but that's probably unrelated here).

termux has never had any such weird bug during my nearly 1 year experience with it.

ncw · January 11, 2021, 4:34pm

No worries!

That is useful to know.

How about if you try the same thing with rclone, so rclone mkdir '/data/data....'? (Try it on a path which doesn't exist).

You could then try writing a file with rclone into the directory echo hello | rclone rcat '/data/....../file.txt'

I use termux too and I find it works very well.

cregox · January 17, 2021, 8:17pm

both rclone commands worked, no issues.

i just used the same string without the first slash data/data/com.termux/files/home/achoz/.nosync/toreview2020dec/(∆16G)to organize/(∆3.7G)mac+documents/zona/BlahTherapy - May "premier plane".rtfd/TXT.rtf and inside a temp folder.

trying to copy this time, instead of sync, got me 7 new unidentified errors (3237 instead of 3230).

this time it was for sure after running the dedupe.

copy3.txt (3.8 MB)

real 161m56.875s
user 8m11.480s
sys 3m57.603s

the one file already mentioned, i can download it manually (on gdrive) or even using:

rclone copy 'gdrive:toreview2020dec/(∆16G)to organize/(∆3.7G)mac+documents/zona/BlahTherapy - May "premier plane".rtfd/TXT.rtf' .

i think i have now less clues on what could be going wrong than in the beginning!

ps: one of the subfolder might be wrongly spelled "zone" before, but i didn't use it on my testings anyway.

ncw · January 18, 2021, 10:55am

Looking through your logs, all the failiures have a : or | symbol in them.

Neither of these characters is allowed on a Windows file system. Is the destination formatted with a Windows file system (FAT) maybe?

You could try encoding these characters with

--local-encoding Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot

And see if that helps.

cregox · January 19, 2021, 12:51am

no windows or fat, thanks god!

coincidentally, the one failure i took as instance don't have those 2 characters you mentioned...

not so surprisingly, and still amazingly, the local encoding you suggested seem to have fixed it! just 18 errors now!!

time rclone copy gdrive:toreview2020dec . --progress --retries 1 --local-encoding Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot > copy4.txt 2>&1

real 53m57.174s
user 2m18.577s
sys 1m19.503s

copy4.txt (2.2 MB)

this generated a terrible log though... i'll make a better one tomorrow. and double check everything!

just wanted to share the awesome results for now.

(to be edited soon enough)

ncw · January 19, 2021, 2:53pm

There must be something about that filing system which doesn't like non Windows friendly characters. Mobile phones are complicated beasts!

If you run mount or cat /proc/mounts you may be able to see what type of file system it is on.

All the errors seem to be like this

Failed to copy: failed to open source object: open file failed: Use the --drive-acknowledge-abuse flag to download this file: googleapi: Error 403: This file has been identified as malware or spam and cannot be downloaded., cannotDownloadAbusiveFile

If you do as it says and use the --drive-acknowledge-abuse flag these will download.

cregox · January 19, 2021, 2:53pm

weird. can't edit my previous message, it seems...

time rclone copy gdrive:toreview2020dec . --local-encoding Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot > copy5.txt 2>&1

real 147m1.749s
user 6m24.717s
sys 3m18.933s

copy5.txt (30.7 KB)

just 18 errors indeed. amazing!

and i forgot to count before, but now all i care is that there are only 59 duplicate notices.

thanks so much for all your help.

hope you'll be able to enjoy achoz someday, which will be born from many contributions such as this!

ncw · January 19, 2021, 2:54pm

Sorry our messages crossed! The same comments apply to this log too - see above

cregox · January 19, 2021, 3:00pm

lol, yeah... coincidences...

i was aware of that "abuse" flag as well, but i rather go one by one now. i like the serendipity of it.

both mount commands bring a huge chunk of text. do you want to bother?

all i know is that i'm in pixel 3 and android 11 using shared storage access offered by termux, which is indeed a bit different from termux file system otherwise. perhaps i forgot to mention this before and it could be in the heart of the issue.

my only question, out of curiosity, is why the straight rclone copy of one defective file goes through, while the folder in which it is contained doesn't?

i'll even do one more test now...

edit: indeed, it works individually.

rclone copy 'gdrive:toreview2020dec/(∆16G)to organize/(∆3.7G)mac+documents/zona/BlahTherapy - May "premier plane".rtfd/' ./test/

both folders have now only that TXT.rtf file. test folder was created by this very command.

i bet it would be the same for any of the other 3k files.

i hope you don't mind, but my next step should be to delete (most of) my google drive (after going through the remaining files).

ncw · January 19, 2021, 4:47pm

I tried it on my pixel5... I can't create files with : in the storage folders either

You can create files with : in your home directory on termux. I don't know why!