Rclone copys always copys files of a symlink'd folder to root of Google Drive instead of entire folder

What is the problem you are having with rclone?

As you know Google Drive Desktop app ignores junction symlinks (atleast on Windows) created with mklink /J

I have a folder which is full of folders that are actually directory junction symlinks. Each folder is full of more folders which have non empty files in them.

rclone copy C:\NonSymlinkFolder remotename: -P

The above command, without any symlinks involved, will copy NonSymlinkFolder and its contents to the root of my Google Drive remotename by default. This is what I want to do with SymlinkFolder too.

However, if there is a symlink involved (below command) - RClone consistently only copies the contents of SymlinkFolder to the root of my remotename - instead of putting those files instead of a new folder called SymlinkFolder in the root of my remotename.

rclone copy C:\SymlinkFolder remotename: -L -P

What is your rclone version (output from rclone version)

rclone v1.56.0

  • os/version: Microsoft Windows 10 Enterprise 2009 (64 bit)
  • os/kernel: 10.0.19042.867 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: dynamic
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

See above

Which cloud storage system are you using? (eg Google Drive)

Google Drive (personal account)

The rclone config contents with secrets removed.

[remotename]
type = drive
client_id = 
client_secret = 
scope = drive
token = {"access_token":"","token_type":"","refresh_token":"","expiry":"2021-08-30T17:25:16.9846814+01:00"}

A log from the command with the -vv flag

2021/08/30 16:55:17 DEBUG : rclone: Version "v1.56.0" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.56.0-windows-amd64\\rclone.exe" "copy" "C:\\Users\\[REDACTED]\\Documents\\Other\\Configurations\\SymlinkFolder" "remotename:" "-P" "-L" "-vv"]
2021/08/30 16:55:17 DEBUG : Creating backend with remote "C:\\Users\\[REDACTED]\\Documents\\Other\\Configurations\\SymlinkFolder"
2021/08/30 16:55:17 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2021/08/30 16:55:17 DEBUG : Using config file from "C:\\Users\\[REDACTED]\\.config\\rclone\\rclone.conf"
2021/08/30 16:55:17 DEBUG : fs cache: renaming cache item "C:\\Users\\[REDACTED]\\Documents\\Other\\Configurations\\SymlinkFolder" to be canonical "local{12rtk}://?/C:/Users/[REDACTED]/Documents/Other/Configurations/SymlinkFolder"
2021/08/30 16:55:17 DEBUG : Creating backend with remote "remotename:"
2021/08/30 16:55:18 DEBUG : Google drive root '': 'root_folder_id = [REDACTED]' - save this in the config to speed up startup
2021-08-30 16:55:19 DEBUG : Google drive root '': Waiting for checks to finish
2021-08-30 16:55:19 DEBUG : Test.txt: Size and modification time the same (differ by -600.5µs, within tolerance 1ms)
2021-08-30 16:55:19 DEBUG : sdf.txt: Size and modification time the same (differ by -16µs, within tolerance 1ms)
2021-08-30 16:55:19 DEBUG : Test.txt: Unchanged skipping
2021-08-30 16:55:19 DEBUG : sdf.txt: Unchanged skipping
2021-08-30 16:55:19 DEBUG : Google drive root '': Waiting for transfers to finish
2021-08-30 16:55:20 DEBUG : NEW.txt: md5 = e0686c18a9bd898ca3fa7075ea310647 OK
2021-08-30 16:55:20 INFO  : NEW.txt: Copied (new)
Transferred:              8 / 8 Byte, 100%, 7 Byte/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:            1 / 1, 100%
Elapsed time:         2.4s
2021/08/30 16:55:20 INFO  :
Transferred:              8 / 8 Byte, 100%, 7 Byte/s, ETA 0s
Checks:                 2 / 2, 100%
Transferred:            1 / 1, 100%
Elapsed time:         2.4s

2021/08/30 16:55:20 DEBUG : 5 go routines active

hello and welcome to the forum,

not sure i understand your example, but if you want rclone to put files into the folder, perhaps specify the folder

rclone copy C:\SymlinkFolder remotename:SymlinkFolder -L -P

if that does not help, can you give more info about the source folders and files.

thanks for your response. I'm aware I can do this, however it is a workaround and not a fix for this change in behaviour
as I said in my post I have many symlink folders and it would be very tedious doing this for every individual file

rclone does what you are describing by default for normal folders without any path needing to be specified. It doesn't work for symlinks though for some reason

The behaviour change also affects rclone sync with symlinks. even though I do source then dest in the command, it shows the same symptoms of this issue - tries to sync the entire Google drive folder against my.local, resulting in mass deletion of my Google drive.

What more info would you like about the folders? I think I gave everything needed

as we cannot see into your screen, hard to work together on this.

to test with, i would create the simplest set of test folders/file(s)
post the mklink commands
post rclone ls on the source

we can try to replicate the problem, perhaps it could be a bug.

rclone ls:

C:\>rclone ls C:\Test
        9 sdfhkfjdsh.txt

C:\>rclone ls C:\Test2
        9 sdfhkfjdsh.txt

C:\>rclone ls C:\

2021/08/30 17:47:05 NOTICE: Test2: Can't follow symlink without -L/--copy-links
     9 Test/sdfhkfjdsh.txt
     9 Test2/sdfhkfjdsh.txt

it correctly detects the folder and puts a notice for it but there is no comment about files

perhaps i do not understand your example.
that command will not create a folder named NonSymlinkFolder in remotename

  1. this is documented at
    https://rclone.org/commands/rclone_copy/
    "it's the contents of source:path that are copied, not the directory name and contents."

  2. i did a quick test

rclone lsd gdrive-a1b2:testfolder 

rclone ls gdrive-a1b2:testfolder 

rclone copy C:\NonSymlinkFolder gdrive-a1b2:testfolder -v 
INFO  : file.txt: Copied (new)

rclone lsd gdrive-a1b2:testfolder 

rclone ls gdrive-a1b2:testfolder 
        1 file.txt
  1. this can be seen in the debug log
    rclone copy C:\NonSymlinkFolder gdrive-a1b2:testfolder -vv will output
    Creating backend with remote "gdrive-a1b2:testfolder"
    --
    rclone copy C:\NonSymlinkFolder gdrive-a1b2:testfolder/NonSymlinkFolder -vv will output
    Creating backend with remote "gdrive-a1b2:testfolder/NonSymlinkFolder"

Ok, my apologies. Genuinely though, rclone copy has always copied the folder for me on local stuff (non symlinks), whether it be local to local, gdrive to local, local to gdrive. That's just how its been for me.

Ok so now I know that isnt intended behaviour - is there a flag or combo of flags I can use to automatically copy the directory structure? or foldername? So it is done automatically rather than manually specifying path for each new file added to the folder.

I had a very good look in the wiki but didn't find anything

now that we agree on rclone behaviour, not sure what you are asking?

can you post an example of the problem.
try to follow my example, using rclone lsd, rclone ls before and after rclone copy

Lets say I have text.txt in a folder called "Test".

I want to rclone copy the "Test" folder including its contents to google drive, but not doing this:

rclone copy C:\Test gdrive-a1b2:Test

Because that way, I'd have to manually write the source folder name as the dest folder name in order to get what I want.

I want a more automated way to do this. For example via a flag:

rclone copy C:\Test gdrive-a1b2: --copy-dir-and-contents

Or via a variable definition in CMD:

set /p sourcename=Test
rclone copy C:\%sourcename% gdrive-a1b2:%sourcename%
rclone lsd gdrive-a1b2:Test
2021/08/30 18:47:41 ERROR : : error listing: directory not found
2021/08/30 18:47:41 Failed to lsd with 2 errors: last error was: directory not found

rclone ls gdrive-a1b2:Test
2021/08/30 18:48:01 Failed to ls: directory not found

<rclone copy command to do what i want>
INFO  : Text.txt: Copied (new)

rclone lsd gdrive-a1b2:Test

rclone ls gdrive-a1b2:Test
        1 Text.txt

So it creates a directory name/directory structure with the same name/directory structure as the source automatically by itself then puts the files i am copying in there

sorry for the confusion, hope that helps

start a new topic, using feature template and a request a new flag.
if enough rcloners post there, perhaps it will get implemented.

If you give rclone a directory, it copies that directory and all contents already so your example is exactly how you would do it.

felix@gemini:~$ cd test
felix@gemini:~/test$ rclone copy /home/felix/test GD:test -vv
2021/08/30 14:19:20 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/08/30 14:19:20 DEBUG : Setting --user-agent "animosityapp" from environment variable RCLONE_USER_AGENT="animosityapp"
2021/08/30 14:19:20 DEBUG : Setting --rc-user "felix" from environment variable RCLONE_RC_USER="felix"
2021/08/30 14:19:20 DEBUG : Setting --rc-pass "felix" from environment variable RCLONE_RC_PASS="felix"
2021/08/30 14:19:20 DEBUG : Setting default for drive-pacer-min-sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
2021/08/30 14:19:20 DEBUG : Setting default for drive-pacer-burst="1000" from environment variable RCLONE_DRIVE_PACER_BURST
2021/08/30 14:19:20 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "/home/felix/test" "GD:test" "-vv"]
2021/08/30 14:19:20 DEBUG : Creating backend with remote "/home/felix/test"
2021/08/30 14:19:20 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/08/30 14:19:20 DEBUG : Creating backend with remote "GD:test"
2021/08/30 14:19:20 DEBUG : Setting drive_pacer_min_sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
2021/08/30 14:19:20 DEBUG : Setting drive_pacer_burst="1000" from environment variable RCLONE_DRIVE_PACER_BURST
2021/08/30 14:19:20 DEBUG : GD: detected overridden config - adding "{TKSWb}" suffix to name
2021/08/30 14:19:20 DEBUG : Setting drive_pacer_min_sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
2021/08/30 14:19:20 DEBUG : Setting drive_pacer_burst="1000" from environment variable RCLONE_DRIVE_PACER_BURST
2021/08/30 14:19:21 DEBUG : fs cache: renaming cache item "GD:test" to be canonical "GD{TKSWb}:test"
2021/08/30 14:19:21 DEBUG : Google drive root 'test': Waiting for checks to finish
2021/08/30 14:19:21 DEBUG : Google drive root 'test': Waiting for transfers to finish
2021/08/30 14:19:23 DEBUG : hosts: md5 = e9b49c993fe22326c398ecea2fd9b219 OK
2021/08/30 14:19:23 INFO  : hosts: Copied (new)
2021/08/30 14:19:25 DEBUG : one/hosts: md5 = e9b49c993fe22326c398ecea2fd9b219 OK
2021/08/30 14:19:25 INFO  : one/hosts: Copied (new)
2021/08/30 14:19:26 DEBUG : two/hosts: md5 = e9b49c993fe22326c398ecea2fd9b219 OK
2021/08/30 14:19:26 INFO  : two/hosts: Copied (new)
2021/08/30 14:19:26 DEBUG : three/hosts: md5 = e9b49c993fe22326c398ecea2fd9b219 OK
2021/08/30 14:19:26 INFO  : three/hosts: Copied (new)
2021/08/30 14:19:26 INFO  :
Transferred:   	        884 / 884 Byte, 100%, 176 Byte/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:         5.6s

2021/08/30 14:19:26 DEBUG : 9 go routines active
felix@gemini:~/test$ rclone ls GD:test
      221 hosts
      221 one/hosts
      221 three/hosts
      221 two/hosts

felix@gemini:~/test$ rclone ls /home/felix/test
      221 hosts
      221 two/hosts
      221 three/hosts
      221 one/hosts

thanks for your help

Best to make a new post and use the help and support template.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.