Try to re-parent Drive folders before creating a new one

chscott · December 21, 2022, 4:55pm

Currently, Rclone seems to always attempt to create a new folder in the target. For example, I see this when attempting a server-side move from a user's My Drive to a shared drive in the same organization:

2022/12/21 10:31:36 DEBUG : HTTP REQUEST (req 0xc000943200)
2022/12/21 10:31:36 DEBUG : POST /drive/v3/files?alt=json&fields=id&prettyPrint=false&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: XXXX
Content-Length: 125
Authorization: XXXX
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.19.3 gdcl/0.91.0
Accept-Encoding: gzip

{"description":"Source2","mimeType":"application/vnd.google-apps.folder","name":"Source2","parents":["0AN_D0eSZFObfUk9PVA"]}

But when moving a file inside that folder, a re-parenting operation is performed:

2022/12/21 10:31:37 DEBUG : HTTP REQUEST (req 0xc000943800)
2022/12/21 10:31:37 DEBUG : PATCH /drive/v3/files/1fS-TOgVmI-Hno6A9sXdRo2z2t_4FTwX-?addParents=1U8dtgx9reaP3Xi2LcM0JsbgXqD1TZsEr&alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%2CresourceKey&prettyPrint=false&removeParents=1gV7ZRz7voQWQEVQEZc8sMvh89Q68KR-V&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: XXXX
Content-Length: 99
Authorization: XXXX
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.19.3 gdcl/0.91.0
Accept-Encoding: gzip

{"description":"sample2.txt","modifiedTime":"2022-12-21T15:09:44.000000000Z","name":"sample2.txt"}

This behavior doesn't match what happens in the UI if I drag and drop the Source2 folder from My Drive into the shared drive. In the UI case, the folder is re-parented, and all sharing permissions are thus preserved.

I should note that re-parenting works here because the shared drive is in the same organization as the My Drive. If it were instead in a different organization, that operation would presumably fail because the corresponding UI case also fails.

My proposal is thus to try either of these approaches:

If the organization identity can be determined, compare the source and target to see if they are in the same organization. If they are, use re-parenting.
If the organization identity cannot be determined, first try to re-parent a folder and fall back to creating a new one if it fails.

ncw · December 22, 2022, 9:37am

I think rclone does reparent the directory when using the DirMove primitive

github.com

rclone/rclone/blob/00e853144e0effa68988392e55852a0a9d0c4cac/backend/drive/drive.go#L2758


      
          }
          
          
// DirMove moves src, srcRemote to this remote at dstRemote
          // using server-side move operations.
          //
          // Will only be called if src.Fs().Name() == f.Name()
          //
          // If it isn't possible then return fs.ErrorCantDirMove
          //
          // If destination exists then return fs.ErrorDirExists
          func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error {
          	srcFs, ok := src.(*Fs)
          	if !ok {
          		fs.Debugf(srcFs, "Can't move directory - not same remote type")
          		return fs.ErrorCantDirMove
          	}
          
          
	srcID, srcDirectoryID, srcLeaf, dstDirectoryID, dstLeaf, err := f.dirCache.DirMove(ctx, srcFs.dirCache, srcFs.root, srcRemote, f.root, dstRemote)
          	if err != nil {
          		return err
          	}

Which kind of suggests that somehow rclone avoided it for your example.

Can you post the command line you used, the redacted config and a log with -vv from your example?

Rclone has the --server-side-across-configs flag which enables this sort of server side move as I've never been able to reliably determine whether they are going to work or not.

chscott · December 22, 2022, 2:49pm

Command line:

2022/12/22 08:43:42 DEBUG : rclone: Version "v1.60.1" starting with parameters ["C:\\Users\\Chad\\Documents\\PowerShell\\Modules\\TMCTools\\bin\\rclone.exe" "move" "--check-first" "--config" "C:\\Users\\Chad\\AppData\\Local\\Temp\\tmpBB5D.tmp" "--log-file" "C:\\Users\\Chad\\Domains\\cdsconsulting.co\\rclone\\Move_20221222_084342\\chad@cdsconsulting.co.log" "--order-by" "size,mixed" "--retries" "1" "--stats" "5s" "--stats-one-line-date" "--update" "--user-agent" "ISV|Transend Corporation|TMCTools/1.0" "--log-level" "DEBUG" "--max-depth" "20" "--drive-skip-dangling-shortcuts" "--drive-skip-shortcuts" "--drive-auth-owner-only" "--drive-server-side-across-configs" "--drive-upload-cutoff" "1000T" "--create-empty-src-dirs" "--delete-empty-src-dirs" "Source:" "Temp:"]

Log:

2022/12/22 08:43:42 DEBUG : Creating backend with remote "Source:"
2022/12/22 08:43:42 DEBUG : Using config file from "C:\\Users\\Chad\\AppData\\Local\\Temp\\tmpBB5D.tmp"
2022/12/22 08:43:42 DEBUG : Source: detected overridden config - adding "{EW92g}" suffix to name
2022/12/22 08:43:43 DEBUG : Google drive root '': 'root_folder_id = 0AJMfyeX0r9BWUk9PVA' - save this in the config to speed up startup
2022/12/22 08:43:43 DEBUG : fs cache: renaming cache item "Source:" to be canonical "Source{EW92g}:"
2022/12/22 08:43:43 DEBUG : Creating backend with remote "Temp:"
2022/12/22 08:43:43 DEBUG : Temp: detected overridden config - adding "{EW92g}" suffix to name
2022/12/22 08:43:43 DEBUG : fs cache: renaming cache item "Temp:" to be canonical "Temp{EW92g}:"
2022/12/22 08:43:43 INFO  : Google drive root '': Running all checks before starting transfers
2022/12/22 08:43:43 DEBUG : Google drive root '': read info from Shared Drive "Intermediate2"
2022/12/22 08:43:44 DEBUG : Google drive root '': Waiting for checks to finish
2022/12/22 08:43:44 INFO  : Google drive root '': Checks finished, now starting transfers
2022/12/22 08:43:44 DEBUG : Google drive root '': Waiting for transfers to finish
2022/12/22 08:43:46 INFO  : Directory1/sample2.txt: Moved (server-side)
2022/12/22 08:43:46 DEBUG : Directory1: Making directory
2022/12/22 08:43:46 DEBUG : Google drive root '': copied 1 directories
2022/12/22 08:43:46 INFO  : Directory1: Removing directory
2022/12/22 08:43:47 DEBUG : Google drive root '': deleted 1 directories
2022/12/22 08:43:47 INFO  : There was nothing to transfer
2022/12/22 08:43:47 INFO  : 2022/12/22 08:43:47 -          21 B / 21 B, 100%, 0 B/s, ETA -
2022/12/22 08:43:47 DEBUG : 10 go routines active

Config:

[Source]
type = drive
scope = drive
service_account_file = C:\Users\Chad\Domains\cdsconsulting.co\cdsconsulting.co.json
impersonate = chad@cdsconsulting.co
[Temp]
type = drive
scope = drive
team_drive = 0AN_D0eSZFObfUk9PVA
service_account_file = C:\Users\Chad\Domains\cdsconsulting.co\cdsconsulting.co.json
impersonate = chad@cdsconsulting.co

ncw · December 23, 2022, 3:28pm

You are trying to move the root directory... Source:

Can you try moving a subdirectory? Source:dir

chscott · December 23, 2022, 4:00pm

That doesn't seem to do a move at all. Here's what I see after trying to move Source:Folder1:

The folder is now both in My Drive and in the shared drive, so it couldn't have been moved.

2022/12/23 09:53:42 DEBUG : rclone: Version "v1.60.1" starting with parameters ["C:\\Users\\Chad\\Documents\\PowerShell\\Modules\\TMCTools\\bin\\rclone.exe" "move" "--check-first" "--config" "C:\\Users\\Chad\\AppData\\Local\\Temp\\tmp4C22.tmp" "--log-file" "C:\\Users\\Chad\\Domains\\cdsconsulting.co\\rclone\\Move_20221223_095341\\chad@cdsconsulting.co.log" "--order-by" "size,mixed" "--retries" "1" "--stats" "5s" "--stats-one-line-date" "--update" "--user-agent" "ISV|Transend Corporation|TMCTools/1.0" "--log-level" "DEBUG" "--max-depth" "20" "--drive-skip-dangling-shortcuts" "--drive-skip-shortcuts" "--drive-auth-owner-only" "--drive-server-side-across-configs" "--drive-upload-cutoff" "1000T" "--create-empty-src-dirs" "--delete-empty-src-dirs" "Source:Folder1" "Temp:Folder1"]

2022/12/23 09:53:42 DEBUG : Creating backend with remote "Source:Folder1"
2022/12/23 09:53:42 DEBUG : Using config file from "C:\\Users\\Chad\\AppData\\Local\\Temp\\tmp4C22.tmp"
2022/12/23 09:53:42 DEBUG : Source: detected overridden config - adding "{EW92g}" suffix to name
2022/12/23 09:53:42 DEBUG : Google drive root 'Folder1': 'root_folder_id = 0AJMfyeX0r9BWUk9PVA' - save this in the config to speed up startup
2022/12/23 09:53:42 DEBUG : fs cache: renaming cache item "Source:Folder1" to be canonical "Source{EW92g}:Folder1"
2022/12/23 09:53:42 DEBUG : Creating backend with remote "Temp:Folder1"
2022/12/23 09:53:42 DEBUG : Temp: detected overridden config - adding "{EW92g}" suffix to name
2022/12/23 09:53:43 DEBUG : fs cache: renaming cache item "Temp:Folder1" to be canonical "Temp{EW92g}:Folder1"
2022/12/23 09:53:43 INFO  : Google drive root 'Folder1': Running all checks before starting transfers
2022/12/23 09:53:44 DEBUG : Google drive root 'Folder1': Waiting for checks to finish
2022/12/23 09:53:44 INFO  : Google drive root 'Folder1': Checks finished, now starting transfers
2022/12/23 09:53:44 DEBUG : Google drive root 'Folder1': Waiting for transfers to finish
2022/12/23 09:53:45 INFO  : sample1.txt: Moved (server-side)
2022/12/23 09:53:45 INFO  : There was nothing to transfer
2022/12/23 09:53:45 INFO  : 2022/12/23 09:53:45 -          21 B / 21 B, 100%, 0 B/s, ETA -
2022/12/23 09:53:45 DEBUG : 10 go routines active

ncw · December 23, 2022, 5:15pm

What you should see is a server side directory move like this

$ rclone moveto -vv drive:test/2001test5/  drive:test/2001test5-moved/
2022/12/23 17:13:59 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "moveto" "-vv" "drive:test/2001test5/" "drive:test/2001test5-moved/"]
2022/12/23 17:13:59 DEBUG : Creating backend with remote "drive:test/2001test5/"
2022/12/23 17:13:59 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2022/12/23 17:14:00 DEBUG : fs cache: renaming cache item "drive:test/2001test5/" to be canonical "drive:test/2001test5"
2022/12/23 17:14:00 DEBUG : Creating backend with remote "drive:test/2001test5-moved/"
2022/12/23 17:14:01 DEBUG : fs cache: renaming cache item "drive:test/2001test5-moved/" to be canonical "drive:test/2001test5-moved"
2022/12/23 17:14:01 DEBUG : Google drive root 'test/2001test5-moved': Using server-side directory move
2022/12/23 17:14:02 INFO  : Google drive root 'test/2001test5-moved': Server side directory move succeeded
2022/12/23 17:14:02 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         3.3s

chscott · December 23, 2022, 5:19pm

Any ideas why I don't see that in my case? Even if I can get the single-directory approach to work, it wouldn't solve my use case, in which I don't know the directories to move ahead of time. My goal is to server-side move all directories that happen to be there, whatever they are.

ncw · December 24, 2022, 9:38am

Can you try your example with moveto like I did? I wonder if that is the difference?

DirMove can't move the root directory - I just checked. Can you arrange to move from the directory above?

If rclone can't DirMove it will Move each individual file which is what you are seeing. I'm not sure why DirMove isn't kicking in for you - if you are able, tracing the flow of rclone would be useful here to figure out why!

It could be Move each file and DirMove each directory in the root of the transfer - I think this is more what you want to achieve.

chscott · December 24, 2022, 3:49pm

DirMove isn't being used because the source and target configs are different, yet both are Google Drive. Here's what I see with some added logging:

2022/12/24 09:35:16 operations.go:726: DEBUG : Google drive root '': SameConfig(Target{rO-Pz}, Source{rO-Pz})
2022/12/24 09:35:16 operations.go:726: DEBUG : Google drive root '': SameConfig(Source{rO-Pz}, Target{rO-Pz})
2022/12/24 09:35:16 sync.go:1151: DEBUG : Google drive root '': Can't server-side directory move. sameConfig is false

If I remove that check, it gets past the block on the server-side directory move, but it fails when trying to move the drive root, just as you said it would.

2022/12/24 09:37:39 operations.go:726: DEBUG : Google drive root '': SameConfig(Target{rO-Pz}, Source{rO-Pz})
2022/12/24 09:37:39 sync.go:1146: INFO  : Google drive root '': Test 2
2022/12/24 09:37:39 sync.go:1150: DEBUG : Google drive root '': Using server-side directory move
2022/12/24 09:37:39 sync.go:1160: ERROR : Google drive root '': Server side directory move failed: can't move root directory
2022/12/24 09:37:39 cmd.go:283: ERROR : Attempt 1/1 failed with 1 errors and: can't move root directory
2022/12/24 09:37:39 stats.go:484: INFO  : 2022/12/24 09:37:39 -           0 B / 0 B, -, 0 B/s, ETA -
2022/12/24 09:37:39 cmd.go:298: DEBUG : 10 go routines active
2022/12/24 09:37:39 cmd.go:334: Failed to move: can't move root directory

Two questions:

What is the rationale behind restricting server-side moves to an identical configuration, even if the differing source and target are the same backend?
How can I server-side move anything inside the drive root without knowing what is there and without using filters, which I believe also causes server-side moves to be disabled?

ncw · December 24, 2022, 9:33pm

This is what --server-side-across-configs is supposed to override. Maybe that isn't working here.

You can't move the root of your drive, but if the rclone path points to a subdirectory it should work.

You could use rclone lsf to list the directory and script the individual moves.

No reason why rclone couldn't do this instead of blowing up with can't move root error, it would just need more code!

chscott · December 24, 2022, 9:42pm

The problem is that no folder move at all is allowed at the API level when going from My Drive to a shared drive. That issue is being tracked here: Google Issue Tracker. What's really disappointing is that the behavior is allowed in the UI: Google Workspace Updates: Improved Admin experience for moving folders from My Drive to shared drives.

This seems like a dead end until Google allows the same UI behavior via the API or somebody reverse-engineers the internal APIs used in the UI approach.

system · February 22, 2023, 9:43pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.