I can't figure out how to build an rclone command that copies multiple folders across multiple drives/partitions on Windows. I want those folders to get copied to GDrive with the same structure dir-tree structure
Run the command 'rclone version' and share the full output of the command.
rclone v1.65.0
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.3693 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: cmount
PS C:\Users\dd\Documents\rclone-v1.65.0-windows-amd64>
Which cloud storage system are you using? (eg Google Drive)
Local
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Aside:
I also spent a couple hours banging my head on errors like:
PS C:\Users\dd\Documents\rclone-v1.65.0-windows-amd64> .\rclone.exe copy D: dd:ddDriveD/ --include-from C:\Users\dd\Documents\rclone-backups-dirs.txt --dry-run
2023/12/10 13:32:15 ERROR : Local file system at //?/D:/: error reading source root directory: directory not found
2023/12/10 13:32:15 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2023/12/10 13:32:15 ERROR : Local file system at //?/D:/: error reading source root directory: directory not found
2023/12/10 13:32:15 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2023/12/10 13:32:15 ERROR : Local file system at //?/D:/: error reading source root directory: directory not found
2023/12/10 13:32:15 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2023/12/10 13:32:15 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 1.5s
2023/12/10 13:32:15 Failed to copy: directory not found
PS C:\Users\dd\Documents\rclone-v1.65.0-windows-amd64>
and no permutation of \, \\, /, //, D: was getting me there till I used --local-nounc and finally rclone understood the path.
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[dd]
type = crypt
remote = ddSharedDrive:2023 ddFolder
password = XXX
[UnionD]
type = union
upstreams = D:\Movies D:\Recitals D:\Photos
[ddSharedDrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = XXX
root_folder_id =
[mylocal]
type = alias
remote = D: # I don't know. doesn't work...
A log from the command that you were trying to run with the -vv flag
I think this is issue #7468 which is fixed in the latest beta. Maybe give that a try? (Alternatively, downgrading to v1.64.2 should also work, if you don't want to use beta.)
Yes that issue is solved, but how do I point to multiple folders across different partitions? I tried this as my filter-from file but it doesn't pick up the other folders:
This sounds wise. Would you please share your motivation behind this advice? I'm not familiar as to why this is risky.
Also, my main issue for this post isn't the local-unc path bug/workaround...
It is the titular bit about how to copy/access multiple folders across different drives in a single filter-from file. Is that not possible? I just want to queue up the folders in a single rclone job. (that I can then cron/schedule)
To my knowledge there is no default root folder where all partitions (drive letters) live so you can not use one --filter-from to include multiple partitions on Windows. But I am not using Windows so my expertise here is limited.
However based on @asdffdsa post maybe you can use mklink and create "virtual" backup folder:
That's an interesting idea! Is there a way via rclone union backend? You'll notice I was playing around with it in my rclone config
One (small) reason why I'm slightly averse to file symlinks is that it is OS level, and I'd like to keep the OS as-is as much as possible while storing all backup config in the rclone side (easier to migrate/export between machines as well, then!)
very risky,
on dos/windows, a full path requires two parts, drive letter + path
if no path is given, dos/windows appends the current work directory(cwd)
confusing but critical:
each drive letter has it own cwd
default cwd= \
same command dir d: /b, different outputs
:: output the files in drive letter 'd:', effective path= 'd:'+'\'
c:\>dir d: /b
Program Files
Program Files (x86)
:: change cwd of drive letter 'd:' from `\` to '\zork'
c:\>cd /d d:\zork
:: now switch back to c:\
d:\zork>cd /d c:\
:: output the files in drive leter 'd:' , effective path= 'd:'+'\zork'
c:\>dir d: /b
file.ext
and we can see this in practice.
here, rclone passes d: to the os, and the os replies with d:/zork
rclone ls d: -vv
2023/12/11 13:25:16 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "ls" "d:" "-vv"]
2023/12/11 13:25:16 DEBUG : Creating backend with remote "d:"
2023/12/11 13:25:16 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2023/12/11 13:25:16 DEBUG : fs cache: renaming cache item "d:" to be canonical "//?/d:/zork"
14 file.ext
fwiw, with my backup script, every time it is run, creates an temp link, run rclone, then deletes the link.
i do this as part of creating a VSS snapshot, which is the source for rclone.
this is using variant #3 from my handy, dandy wiki https://github.com/rclone/rclone/wiki/How-to-enable-VSS-for-rclone
another workaround, also from my wiki, but using variant #2
rclone lsd c:\
2023/12/11 13:49:59 ERROR : : error listing: directory not found
2023/12/11 13:49:59 Failed to lsd with 2 errors: last error was: directory not found
net use x: \\localhost\c$
The command completed successfully.
rclone lsd x:\
-1 2023-06-16 09:45:13 -1 $Recycle.Bin
-1 2023-11-30 20:05:18 -1 Program Files
-1 2023-11-13 18:17:01 -1 Program Files (x86)
-1 2023-10-02 09:43:11 -1 ProgramData
-1 2023-12-10 12:46:21 -1 System Volume Information
...