Best/easiest way to copy a dir and its contents from local to rclone

What is the problem you are having with rclone?

I am trying to copy a local dir with multiple files to a mount but it only ever seems to copy the contents of the dir not the dir and its contents. I can create the dir in the remote and then copy but that is awkward as i work from a phone a lot so copy and paste is not ideal.

Surely there has to be a better way than what i am doing.

I dont see any other options on the rclone copy docs. rclone copy

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 4.15.0-192-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -vP --transfers 10 /mnt/storage-drive/requests/Some.dir.with.a.really.long.name gdrive-crypt:f/

this command does seem to work but its a nightmare as i have to manually enter the path on the remote since tab wont auto complete it. Please note i do a lot of work from a phone so its not that easy for me to copy and paste stuff.

rclone copy -vP --transfers 10 /mnt/storage-drive/requests/Some.dir.with.a.really.long.name gdrive-crypt:f/Some.dir.with.a.really.long.name

Ok this does seem to work but is there a better way

rclone copy -vP --transfers 10 /mnt/storage-drive/requests/ --include "Some.dir.with.a.really.long.*" gdrive-crypt:f/

If you share a log file with the info in it, we can see what's going on as copy will copy all the contents.

If the file length is a problem, you can use a script I'd imagine to put it all in there.

1 Like

Not sure what you mean.

From the copy docs it states that the dir is not copied over so not sure how logs are going to help when this is expected behavior

For example in the docs its states

Not to

destpath/sourcepath/one.txt

That's why we ask for a log file as trying to figure out what you mean isn't always clear, but a log file helps to clear that up as it shows exactly what command you run and the actual output.

It's about context so without a logfile to see what you are talking about, it's really tough.

Here is a copy command copying the directories and all the files in the directories.

felix@gemini:~$ rclone copy test1 test2 -vv
2022/10/07 07:57:44 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/10/07 07:57:44 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "test1" "test2" "-vv"]
2022/10/07 07:57:44 DEBUG : Creating backend with remote "test1"
2022/10/07 07:57:44 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/10/07 07:57:44 DEBUG : fs cache: renaming cache item "test1" to be canonical "/home/felix/test1"
2022/10/07 07:57:44 DEBUG : Creating backend with remote "test2"
2022/10/07 07:57:44 DEBUG : fs cache: renaming cache item "test2" to be canonical "/home/felix/test2"
2022/10/07 07:57:44 DEBUG : Local file system at /home/felix/test2: Waiting for checks to finish
2022/10/07 07:57:44 DEBUG : one: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 07:57:44 INFO  : one: Copied (new)
2022/10/07 07:57:44 DEBUG : Local file system at /home/felix/test2: Waiting for transfers to finish
2022/10/07 07:57:44 DEBUG : two: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 07:57:44 INFO  : two: Copied (new)
2022/10/07 07:57:44 DEBUG : adirectory/afileinadirectory: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 07:57:44 INFO  : adirectory/afileinadirectory: Copied (new)
2022/10/07 07:57:44 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            3 / 3, 100%
Elapsed time:         0.0s

2022/10/07 07:57:44 DEBUG : 5 go routines active
felix@gemini:~$ ls -al test1
total 0
drwxrwxr-x 1 felix felix  32 Oct  7 07:57 .
drwxr-x--- 1 felix felix 258 Oct  7 07:57 ..
drwxrwxr-x 1 felix felix  34 Oct  7 07:57 adirectory
-rw-rw-r-- 1 felix felix   0 Oct  7 07:56 one
-rw-rw-r-- 1 felix felix   0 Oct  7 07:57 two
felix@gemini:~$ ls -al test2
total 0
drwxrwxr-x 1 felix felix  32 Oct  7 07:57 .
drwxr-x--- 1 felix felix 258 Oct  7 07:57 ..
drwxrwxr-x 1 felix felix  34 Oct  7 07:57 adirectory
-rw-rw-r-- 1 felix felix   0 Oct  7 07:56 one
-rw-rw-r-- 1 felix felix   0 Oct  7 07:57 two

If you want to share a log and show what you are talking about, happy to help.

1 Like

here is the log file

rclone copy -vv --transfers 10 /mnt/storage-drive/requests/Some.dir.with.a.really.long.name gdrive-crypt:test/
2022/10/07 12:14:39 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "copy" "-vv" "--transfers" "10" "/mnt/storage-drive/requests/Some.dir.with.a.really.long.name" "gdrive-crypt:test/"]
2022/10/07 12:14:39 DEBUG : Creating backend with remote "/mnt/storage-drive/requests/Some.dir.with.a.really.long.name"
2022/10/07 12:14:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/10/07 12:14:39 DEBUG : Creating backend with remote "gdrive-crypt:test/"
2022/10/07 12:14:39 DEBUG : Creating backend with remote "gdrive:crypt/j59ksr71kbeeofe1l4a46sbr9s"
2022/10/07 12:14:40 DEBUG : Google drive root 'crypt/j59ksr71kbeeofe1l4a46sbr9s': 'root_folder_id = 0AGgoUxjK2ka5Uk9PVA' - save this in the config to speed up startup
2022/10/07 12:14:41 DEBUG : Encrypted drive 'gdrive-crypt:test/': Waiting for checks to finish
2022/10/07 12:14:41 DEBUG : Encrypted drive 'gdrive-crypt:test/': Waiting for transfers to finish
2022/10/07 12:14:42 DEBUG : file3.txt: md5 = 62c36600b94ff9bdd50d8909479861df OK
2022/10/07 12:14:42 INFO  : file3.txt: Copied (new)
2022/10/07 12:14:42 DEBUG : file2.txt: md5 = cfddb5fb7b9586e2cc897e9d2cc9a014 OK
2022/10/07 12:14:42 INFO  : file2.txt: Copied (new)
2022/10/07 12:14:43 DEBUG : file4.txt: md5 = d53ee100b86f7015ad7dc5b62712dc9d OK
2022/10/07 12:14:43 INFO  : file4.txt: Copied (new)
2022/10/07 12:14:43 DEBUG : file1.txt: md5 = d07a6a875d46d498c50c04ad98533d68 OK
2022/10/07 12:14:43 INFO  : file1.txt: Copied (new)
2022/10/07 12:14:43 INFO  :
Transferred:            200 B / 200 B, 100%, 99 B/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:         3.5s

2022/10/07 12:14:43 DEBUG : 13 go routines active

contents of rclone drive after copy

rclone ls gdrive-crypt:test/
        2 file4.txt
        2 file3.txt
        2 file2.txt
        2 file1.txt

and what did you expect to happen that did not?

what i am looking for is an easy way to copy a directory and it contents to an rclone drive.

Similar to if i wanted to copy a directory on lunix to another path i would run this command.

cp -r /path/to/some/dir /other/path/

That's what it did based on the output you shared.

What in that output is not what you are expecting?

no it copied the contents not the dir name after copying it should look like this.

rclone ls gdrive-crypt:test/
       dir 
               2 file4.txt
               2 file3.txt
               2 file2.txt
               2 file1.txt

so that the dir in is the test directory and then the files1, file2 etc... are in test/dir/

The way rclone works is that it does not work like a regular copy command as if you give it a path, it is relative and copies down.

If you want the same directory name on the destination and it not to be relative.

felix@gemini:~/test2$ ls
test
felix@gemini:~/test2$ rm -rf test
felix@gemini:~/test2$ ls
felix@gemini:~/test2$ cd
felix@gemini:~$ ls -alr test1
total 0
-rw-rw-r-- 1 felix felix   0 Oct  7 07:57 two
-rw-rw-r-- 1 felix felix   0 Oct  7 07:56 one
drwxrwxr-x 1 felix felix  34 Oct  7 07:57 adirectory
drwxr-x--- 1 felix felix 258 Oct  7 07:57 ..
drwxrwxr-x 1 felix felix  32 Oct  7 07:57 .

felix@gemini:~$ rclone copy test1 test2/test1 -vv
2022/10/07 08:43:27 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/10/07 08:43:27 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "test1" "test2/test1" "-vv"]
2022/10/07 08:43:27 DEBUG : Creating backend with remote "test1"
2022/10/07 08:43:27 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/10/07 08:43:27 DEBUG : fs cache: renaming cache item "test1" to be canonical "/home/felix/test1"
2022/10/07 08:43:27 DEBUG : Creating backend with remote "test2/test1"
2022/10/07 08:43:27 DEBUG : fs cache: renaming cache item "test2/test1" to be canonical "/home/felix/test2/test1"
2022/10/07 08:43:27 DEBUG : Local file system at /home/felix/test2/test1: Waiting for checks to finish
2022/10/07 08:43:27 DEBUG : Local file system at /home/felix/test2/test1: Waiting for transfers to finish
2022/10/07 08:43:27 DEBUG : one: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 08:43:27 DEBUG : two: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 08:43:27 INFO  : two: Copied (new)
2022/10/07 08:43:27 INFO  : one: Copied (new)
2022/10/07 08:43:27 DEBUG : adirectory/afileinadirectory: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/07 08:43:27 INFO  : adirectory/afileinadirectory: Copied (new)
2022/10/07 08:43:27 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            3 / 3, 100%
Elapsed time:         0.0s

2022/10/07 08:43:27 DEBUG : 3 go routines active

That's not how rclone generally works as it copies to a folder. The slash for this case, doesn't matter.

felix@gemini:~$ rclone copy /home/felix/test1 GD:test2/ -vv
2022/10/13 10:24:20 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/10/13 10:24:20 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "/home/felix/test1" "GD:test2/" "-vv"]
2022/10/13 10:24:20 DEBUG : Creating backend with remote "/home/felix/test1"
2022/10/13 10:24:20 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/10/13 10:24:20 DEBUG : Creating backend with remote "GD:test2/"
2022/10/13 10:24:21 DEBUG : fs cache: renaming cache item "GD:test2/" to be canonical "GD:test2"
2022/10/13 10:24:21 DEBUG : Google drive root 'test2': Waiting for checks to finish
2022/10/13 10:24:21 DEBUG : Google drive root 'test2': Waiting for transfers to finish
2022/10/13 10:24:23 DEBUG : filetest: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/13 10:24:23 INFO  : filetest: Copied (new)
2022/10/13 10:24:23 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.7s

2022/10/13 10:24:23 DEBUG : 5 go routines active
felix@gemini:~$ rclone copy /home/felix/test1 GD:test3 -vv
2022/10/13 10:24:25 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/10/13 10:24:25 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "/home/felix/test1" "GD:test3" "-vv"]
2022/10/13 10:24:25 DEBUG : Creating backend with remote "/home/felix/test1"
2022/10/13 10:24:25 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/10/13 10:24:25 DEBUG : Creating backend with remote "GD:test3"
2022/10/13 10:24:26 DEBUG : Google drive root 'test3': Waiting for checks to finish
2022/10/13 10:24:26 DEBUG : Google drive root 'test3': Waiting for transfers to finish
2022/10/13 10:24:27 DEBUG : filetest: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/10/13 10:24:27 INFO  : filetest: Copied (new)
2022/10/13 10:24:27 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.1s

2022/10/13 10:24:27 DEBUG : 7 go routines active

and

felix@gemini:~$ rclone ls GD:test2
        0 filetest
felix@gemini:~$ rclone ls GD:test3
        0 filetest
1 Like

even better :slight_smile:
I; m trying currently to promte the use of rclone on our supercomputer (Snellius, NL). I have been using rsync quite a lot, hence the misunderstanding.

THX for pointing it out, I removed my answer to avoid confusing the users :slight_smile:

Regards,

Enric

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.