Copy & synchronize remote directory structure to local destination

Is there a way to copy and/or synchronize a remote directory structure (including nested sub-directories) to a local destination without copying or synchronizing files?

A similar question was asked about replicating directory structures for a secondary remote

Is there a similar solution for local destination since the command C:\rclone-v1.53.2-windows-amd64\rclone.exe lsjson OneDriveEncrypted:/ | ConvertFrom-Json | ForEach-Object { $_ } | Where-Object { $_.IsDi r } | Select-Object -ExpandProperty Path | ForEach-Object { rclone mkdir c:/test/$_ } for example fails to execute?

When run, PowerShell returns ```Enter configuration password:`` however there is no ability to enter it.

What is your rclone version (output from rclone version)

rclone v1.53.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Microsoft Windows 10 Professional version 1909 Build 18636.1110

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

set a variable, for a .cmd
set RCLONE_CONFIG_PASS=password

Thanks @asdffdsa. Is there an option not to set a variable and enter the credentials at the point it is required?

You could prompt for the password from the PowerShell command line and set the entered value in the mentioned environment variable:

$Env:RCLONE_ASK_PASSWORD='FALSE';$Env:RCLONE_CONFIG_PASS=$(Read-Host -Prompt 'Enter password');amd64\rclone.exe ...

(this is also setting the ask-password option to false, as recommended here).

The environment variable will be set temporarily for the powershell session only, but if you plan to continue using the same PowerShell session for a while you may consider removing it with Remove-Item Env:RCLONE_CONFIG_PASS at the end. In any case the password will be kept as plain text in the environment of the powershell and rclone processes while running. Not a problem in most cases, but in some cases this is considered an issue.

An alternative is to use --password-command like below. This avoids keeping the plain text password in environment variable. Since you are already in powershell, you would start rclone from powershell, and rclone then start another powershell process to ask for password. So feels a bit more "heavy", and could add a second or so of delay before asking the password, but will probably work fine as well.

amd64\rclone.exe --password-command "powershell -NoProfile -Command Read-Host -Prompt 'Enter password'" lsjson ...

By the way, you use rclone to create the local directory at the end of your command: rclone mkdir c:/test/$_. You could also just use powershell command for this: New-Item -ItemType Directory -Path "c:\test\$_"

1 Like

Thanks for the detailed answer @albertony. The second option of using --password-command the perfect solution.

Can you please elaborate on By the way, you use rclone to create the local directory at the end of your command: rclone mkdir c:/test/$_. You could also just use powershell command for this: New-Item -ItemType Directory -Path "c:\test\$_"? I didn't quite follow it.

I just meant that you don't need to execute rclone to create a local directory when already in powershell, powershell is quite capable of doing that "natively" (and probably faster).

Thanks @albertony. How would Powershell do that if it needs to replicate a remote? Would it not have to first connect to a remote, traverse all directories (excluding files) and create them locally? Sorry, maybe I'm missing something.

I mean just the last part.

From your original example, instead of:

C:\rclone-v1.53.2-windows-amd64\rclone.exe lsjson OneDriveEncrypted:/ | ConvertFrom-Json | ForEach-Object { $_ } | Where-Object { $.IsDir } | Select-Object -ExpandProperty Path | ForEach-Object { rclone mkdir c:/test/$ }

Then:

C:\rclone-v1.53.2-windows-amd64\rclone.exe lsjson OneDriveEncrypted:/ | ConvertFrom-Json | ForEach-Object { $_ } | Where-Object { $.IsDir } | Select-Object -ExpandProperty Path | ForEach-Object { New-Item -ItemType Directory -Path "c:\test$" }

In other words, instead of:

rclone mkdir c:/test/$_ }

Then:

New-Item -ItemType Directory -Path "c:\test$_"

Righto. Thanks for clarifying @albertony.

@albertony, the command doesn't appear to return a prompt to enter in a password. When executed, Powershell displays that the script is running.

Strange. Works fine here. You are in a PowerShell prompt when executing the command? And it was not started with -NonInteractive?

What does it display - exactly?

It's run using Powershell ISE. The command was executed with amd64\rclone.exe --password-command "powershell -NoProfile -Command Read-Host -Prompt 'Enter password'" lsjson ...

Nothing. It simply runs the command. There is no ability to enter in or paste any data.

I see the same when running from PowerShell ISE. Don't know why, but its something about the way ISE runs the executables, I guess. Running from regular PowerShell command line it works. (I've switched from using ISE to VSCode for PowerShell development myself).

It's odd since I use the Powershell ISE to run other executables. In trying the Powershell command line, it returns the error $.IsDir : The term '$.IsDir' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

Yeah, but here it is one executable starting another executable. And also that "final" executable is powershell, same as ISE also is running itself for the command prompt. Perhaps ISE intercepts the powershell process somehow. Just guessing..

Regarding the error: Are you missing the underscore? It should be $_.IsDir.

Thanks @albertony. May have to research it further.

The change in the command only worked partially. The command is ./rclone.exe lsjson remotename | ConvertFrom-Json | ForEach-Object { $_ } | Where-Object { $_.IsDir } | Select-Object -ExpandProperty Path | ForEach-Object { New-Item -ItemType Directory -Path "c:\{name_of_directory}" }

It creates the directory test but not the remote hierarchy.

Replace:

New-Item -ItemType Directory -Path "c:\test"

With:

New-Item -ItemType Directory -Path "c:\test\${_}"

And if there are more than one level of directories you must add -R option to the lsjson to make it recursive:

./rclone.exe lsjson -R ...

Does that work? If it does, maybe post back the final command - for future readers..

1 Like

Thanks @albertony. That worked. The complete command is

./rclone.exe lsjson -R remotename | ConvertFrom-Json | ForEach-Object { $_ } | Where-Object { $_.IsDir } | Select-Object -ExpandProperty Path | ForEach-Object { New-Item -ItemType Directory -Path "localpath\${_}" }

remotename is the name of the remote and localpath is the client location e.g. c:\{directory}

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.