Is there a way to sync while keeping file description on the destination?

Khar00f · February 27, 2020, 6:10pm

I have remote1 and remote2 on two different GSuite accounts

I'm migrating all the users and drive files from remote1 to remote2
In order for me to restore the permissions via API i need a quick and efficient way to match the source file with the destination file.

Can't rely on modifying dates and name is too common.

I was able to update all the files in remote1 with their respective ID, the idea is that if the description transfers over using RClone then I can have a unique identifier for each one and i can cross reference the description with the old ID to know exactly which file it is and then I can apply the permissions applicable that I've downloaded from the source (I don't know how clear that is).

Using RClone 1.51 on Windows 10 64bits

I found this existing post which stopped at @ncw asking the user what purpose that would serve but it's so old i don't want to revive it.

Any ideas would be greatly appreciated,

Thank you,

ncw · February 28, 2020, 10:01am

I think if you do a server side copy, so enable --drive-server-side-across-configs when you do the copy the description should copy across. Can you try it?

Khar00f · February 28, 2020, 11:45am

I've only been using server side copy since I started syncing drives, but it still overwrites the description with the filename.

ncw · February 28, 2020, 1:11pm

Ah, you could patch the source to stop it doing that I think...

Try this

diff --git a/backend/drive/drive.go b/backend/drive/drive.go
index 211fb40d2..fc5f86825 100644
--- a/backend/drive/drive.go
+++ b/backend/drive/drive.go
@@ -2056,6 +2056,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 	if err != nil {
 		return nil, err
 	}
+	createInfo.Description = ""
 
 	var info *drive.File
 	err = f.pacer.Call(func() (bool, error) {

Khar00f · February 28, 2020, 4:40pm

I will def try that, but i would need a couple of information from you if you can help.

I'm not a programmer/scripter but i'll give it a try

Can you tell me:

What language is rclone based on (i'm guessing C since it's based on Rsync, but i'm not sure)?
in which file do I add that line of code (I checked in backend/drive/drive.go) but i don't see that block of code in there to add that line), so i'm guessing I have the wrong file?

I'll add it and try and compile it afterwards,

Thank you for your help

ncw · February 28, 2020, 4:46pm

I did it for you - give this a go

https://beta.rclone.org/branch/v1.51.0-068-gb1706719-fix-drive-description-beta/ (uploaded in 15-30 mins)

Khar00f · February 28, 2020, 4:54pm

Thank you,

I'll test it right away

ncw · February 28, 2020, 5:02pm

The build failed on that one, try this one

https://beta.rclone.org/branch/v1.51.0-069-g18e012c5-fix-drive-description-beta/ (uploaded in 15-30 mins)

Khar00f · February 28, 2020, 5:22pm

I tried both 068 and 069

They both copy the description as intended (AWESOME) BUT it doesn't copy the description for Google files (sheet, doc, etc...) and folders.

The pdf, xlsx, csv, etc... all have the proper description copied over but not native google files or folders.

ncw · February 29, 2020, 8:28am

I'm glad the files work!

I don't know why the GDocs don't work, I do the same request for the Gdocs as the files

2020/02/29 08:22:21 DEBUG : POST /drive/v3/files/1o-ELHHIir1uNInYEo1xyLQIKmtD7x4r7KaHmrSZUpEk/copy?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink&keepRevisionForever=false&prettyPrint=false&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.51.0-DEV
Content-Length: 112
Authorization: XXXX
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.11.0 gdcl/20200223
Accept-Encoding: gzip

{"modifiedTime":"2020-02-29T08:19:48.592000000Z","name":"Docs","parents":["12tTxU5kZk9Y5EWJ1ftXLX47XDSWbs232"]}

Note no description there...

So I'm not sure why it doesn't copy across.

Rclone won't read the description for folders - it will make and destroy them as needed.

Khar00f · March 9, 2020, 1:40pm

Sorry about the late reply, was ouf of country for the week (March Break).

Folders it's not so bad, because they're easy to identify and even if we mix the names they'll go in the right place regardless because of the path.

Is it possible that the description for native G files is stored elsewhere and would have to be retrieved on it's own?

Thank you,

ncw · March 9, 2020, 9:45pm

What we do is a server side copy, so we say copy this object from here to here. When we do that with files the description copies too. When we do that with gdocs the description doesn't! We could potentially set the description for gdocs after the copy to work around this.

I'd say this is a bug and it is probably described in the google issue tracker if you can be bothered to search for it! https://issuetracker.google.com/savedsearches/5604220?q=componentid:191650%20copy%20description

Khar00f · March 11, 2020, 12:16pm

Is this something that would have to be coded or can it be included with a flag of some sort as I didn't see any flags to update the description?

Is there a way to print the fileID in the log? I know that it writes the MD5 sum, but if there's a way to add the ID instead, that could be beneficial and possibly a workaround.

I looked in the issue tracker yesterday to see if I can find any bugs listed for that issue and couldn't locate any.

I don't know if the componentid above (191650) is the ticket number, but the link gives me an access denied error.

ncw · March 11, 2020, 4:12pm

It would have to be coded.

OK

I think the link gives you access denied because it is one of my saved searches - sorry.

Try this - it has a work-around by reading the description from the google doc before copying it.

https://beta.rclone.org/branch/v1.51.0-094-g16f37282-fix-drive-description-beta/ (uploaded in 15-30 mins)

Khar00f · March 11, 2020, 5:24pm

It worked, it copied over the description for Google native files, thank you,

I have one last question, I noticed that every time i run the sync command it duplicates all google files, files which are not google native get skipped over and it recognizes that the file already there, but files which are google native it's as if it's unable to recognize that it exists and creates doubles, triples, etc...

is there a flag that can be used that will make it check the file before it creates a new copy?

Thank you,

ncw · March 12, 2020, 10:41am

Great!

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52

Hmm that shouldn't happen!

can you paste your rclone config removing secrets and the exact rclone command you are using? I suspect you are using --drive-shared-with-me which I think does that but I'm not sure.

Khar00f · March 13, 2020, 1:07pm

[remote1]
type = drive
client_id = xxx
scope = drive
service_account_file = C:\yyy.json
root_folder_id = zzz

[remote1s]
type = drive
client_id = xxx
scope = drive
service_account_file = C:\yyy.json
root_folder_id = zzz
shared_with_me = true

Commands:

Since remote1s is a folder that was shared with owner of remote1 and given writer access, these are the commands I run.

set RCLONE_CONFIG_remote1_IMPERSONATE=email@mymail.com
set RCLONE_CONFIG_remote1s_IMPERSONATE=email@mymail.com

rclone sync -vv --drive-server-side-across-configs --exclude _FOR_MIGRATION_ONLY/** "remote1:" "remote1s:path/dir" --track-renames --drive-auth-owner-only --log-file=C:...bbb.log

The destination is a shared folder, but not the source, is there a reason why it does that? Maybe by understanding the logic I can work around it.

The reason I do it this way is because if I use for destination a remote that is not the same impersonated user, it skips over some files files with error "file not found" (and it seems to be the GSuite files that it skips)

ERROR : Dir/Userlistfil.xlsx: Failed to copy: googleapi: Error 404: File not found: 1JOUoO_Xm8GaO5qS_fljbwaryDFAXmdo3ZrYNScAy7ew., notFound

ERROR : Goldy.xlsx: Failed to copy: googleapi: Error 404: File not found: 1fi7OG4wpCbHyUPOszfH0OieVtfFysdED-SsVm7_M81k., notFound

Both of these files are native sheets not xlsx files.

ncw · March 13, 2020, 9:21pm

I think this might be the same issue as

It might be worth trying the latest beta as I fixed some stuff with shared-with-me recently.

Khar00f · March 16, 2020, 3:09pm

Nope, still duplicating with beta 103.

to go around the issue,

once I'm done with the syncs, could i run a dedupe with "newest" argument and would that reliably remove older dupes?

Thank you,

ncw · March 16, 2020, 4:19pm

Yes dedupe will clear up for you. Use the --dry-run argument first to see what it would do.