like command rclone rcd --rc-serve to list an object in any remote with http://127.0.0.1:5572/[remote:path]/path/to/object, is it possible to make it not ro (read-only), but +w writable? In another words, can we mount all remotes together?
For example, I have remote_1, remote_2, .... remote_10.
using rclone serve webdav command without give specific remote, rclone then shows [remote_1:], [remote_2:], ... [remote_10:] as subfolders, which have fully read, write, list, etc functions.
Do we have a similar way to achieve it?
It would be a great function, without mounting every remote as a separate process.
Thank you.
From the debug output, I only copy one file into mega_1, but it looks rclone creates all backend remotes first and then make the transfer to mega_1? Creating all backend remotes together will cost memory resource? Or am I wrong? Thanks again.
$ rclone copy file1 combine:mega_1/test -vv
2022/12/14 18:50:56 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "file1" "combine:mega_1/test" "-vv"]
2022/12/14 18:50:56 DEBUG : Creating backend with remote "file1"
2022/12/14 18:50:56 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2022/12/14 18:50:56 DEBUG : fs cache: adding new entry for parent of "file1", "/home/username/.config/rclone"
2022/12/14 18:50:56 DEBUG : Creating backend with remote "combine:mega_1/test"
2022/12/14 18:50:56 DEBUG : Creating backend with remote "mega_2:data"
2022/12/14 18:50:56 DEBUG : Creating backend with remote "mega_3:data"
2022/12/14 18:50:56 DEBUG : Creating backend with remote "mega_1:data"
2022/12/14 18:51:00 DEBUG : file1: Need to transfer - File not found at Destination
2022/12/14 18:51:04 INFO : file1: Copied (new)
2022/12/14 18:51:04 INFO :
Transferred: 10 MiB / 10 MiB, 100%, 3.334 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 8.0s
rclone config file:
[mega_1]
type = mega
user = xxxxx
pass = xxxxx
[mega_2]
type = mega
user = xxxxx
pass = xxxxx
[mega_3]
type = mega
user = xxxxx
pass = xxxxx
[combine]
type = combine
upstreams = mega_1=mega_1:data mega_2=mega_2:data mega_3=mega_3:data
Is it possible only create one remote backend when it is needed, not all together every time?
rclone rcd --rc-serve only create one needed backend when request, but read only.
Here is the test result:
$ /usr/bin/time -v rclone copy file1 mega_1:data/test --ignore-times -vv
DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "file1" "mega_1:data/test" "--ignore-times" "-vv"]
DEBUG : Creating backend with remote "file1"
DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
DEBUG : fs cache: adding new entry for parent of "file1", "/home/username/.config/rclone"
DEBUG : Creating backend with remote "mega_1:data/test"
DEBUG : file1: Need to transfer - File not found at Destination
INFO : file1: Copied (new)
2022/12/14 20:14:53 INFO :
Transferred: 10 MiB / 10 MiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 2.1s
DEBUG : 12 go routines active
Command being timed: "rclone copy file1 mega_1:data/test --ignore-times -vv"
User time (seconds): 0.26
System time (seconds): 0.05
Percent of CPU this job got: 14%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 65680
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 9089
Voluntary context switches: 1895
Involuntary context switches: 62
Swaps: 0
File system inputs: 20480
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
$ /usr/bin/time -v rclone copy file1 combine:mega_1/test2 --ignore-times -vv
DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "copy" "file1" "combine:mega_1/test2" "--ignore-times" "-vv"]
DEBUG : Creating backend with remote "file1"
DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
DEBUG : fs cache: adding new entry for parent of "file1", "/home/username/.config/rclone"
DEBUG : Creating backend with remote "combine:mega_1/test2"
DEBUG : Creating backend with remote "mega_3:data"
DEBUG : Creating backend with remote "mega_1:data"
DEBUG : Creating backend with remote "mega_2:data"
DEBUG : file1: Need to transfer - File not found at Destination
INFO : file1: Copied (new)
INFO :
Transferred: 10 MiB / 10 MiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 4.5s
DEBUG : 22 go routines active
Command being timed: "rclone copy file1 combine:mega_1/test2 --ignore-times -vv"
User time (seconds): 0.46
System time (seconds): 0.05
Percent of CPU this job got: 11%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.55
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 67660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 9567
Voluntary context switches: 2383
Involuntary context switches: 44
Swaps: 0
File system inputs: 20480
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
User time (seconds): 0.26 -> 0.46
Maximum resident set size (kbytes): 65680 -> 67660 increase a little bit. I'm using 3 backend remotes. If I use 2000 remotes, will this memory usage becomes influence?
Perhaps, I don't know. Probably a lot easier said than done, it would require some kind of on-demand loading of nested remotes - that could easily get very complex.
Let's make a guestimate: 65680KB + (67660 KB - 65680 KB) * 2000 / 3 = 1.3 GB
How much memory do you expect to be used by 2000 mounts started individually or using rclone rcd?