What is the problem you are having with rclone?
Using rclone to copy a directory tree from a SFTP remote (actually a Linux container on the same host) copies some files repeatedly. This only affects a small subset of files. Narrowing this down suggests that it is not the specific files that are the issue, but the containing directory.
Issue detected on latest stable and reproduced on latest beta.
Run the command 'rclone version' and share the full output of the command.
rclone v1.59.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-48-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: none
also tested on:
rclone v1.60.0-beta.6466.2aa264b33
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-48-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
SFTP to local filesystem
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
./rclone copy -vv orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/ /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/
The rclone config contents with secrets removed.
[orchestra-jammy]
type = sftp
host = 10.51.175.82
user = ubuntu
key_file = ~/.ssh/id_rsa
key_file_pass = <removed>
shell_type = unix
md5sum_command = /usr/bin/md5sum
sha1sum_command = /usr/bin/sha1sum
A log from the command with the -vv
flag
[snip]
2022/10/01 12:21:40 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/: Shell path "/home/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md"
2022/10/01 12:21:40 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/: Running remote command: /usr/bin/md5sum /home/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md
2022/10/01 12:21:40 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/: Remote command result: 34a3da2c76628872ceafaa9a6307df42 /home/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md
2022/10/01 12:21:40 DEBUG : template/disclaimer.md: Parsed hash: 34a3da2c76628872ceafaa9a6307df42
2022/10/01 12:21:40 DEBUG : template/disclaimer.md: md5 = 34a3da2c76628872ceafaa9a6307df42 OK
2022/10/01 12:21:40 INFO : template/disclaimer.md: Copied (new)
2022/10/01 12:21:40 INFO :
Transferred: 788.268 KiB / 788.268 KiB, 100%, 0 B/s, ETA -
Checks: 6 / 6, 100%
Transferred: 8 / 8, 100%
Elapsed time: 0.8s
2022/10/01 12:21:40 DEBUG : 57 go routines active
2022/10/01 12:21:40 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/: Closing 6 unused connections
Repeating the command results in the same output, with all 8 files in the template
remote sub-directory reported as being copied each time, including the final file disclaimer.md
shown in log.
The copy should not happen as it is present on both the remote and local file system:
david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone lsl orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md
994 2022-08-15 10:07:21.000000000 disclaimer.md
david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone lsl /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md
994 2022-08-15 10:07:21.000000000 disclaimer.md
However if just the template
sub-directory itself is copied, or just the disclaimer.md
file within it, then the existing version is detected as expected and no copy takes place. It only happens when copying the higher level directory.
sub-directory:
david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone copy -vv orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/ /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/
2022/10/01 12:29:32 DEBUG : rclone: Version "v1.59.2" starting with parameters ["./rclone" "copy" "-vv" "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/" "/home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/"]
2022/10/01 12:29:32 DEBUG : Creating backend with remote "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/"
2022/10/01 12:29:32 DEBUG : Using config file from "/home/david/.config/rclone/rclone.conf"
2022/10/01 12:29:32 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/: New connection 10.51.175.1:48456->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/01 12:29:33 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/: Shell type "unix" from config
2022/10/01 12:29:33 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/: Relative path resolved to "/home/ubuntu/github/ltng-documentation/template"
2022/10/01 12:29:33 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/: Using root directory "/home/ubuntu/github/ltng-documentation/template"
2022/10/01 12:29:33 DEBUG : Creating backend with remote "/home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/"
2022/10/01 12:29:33 DEBUG : cover.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : finalpage_A4.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : cover.pdf: Unchanged skipping
2022/10/01 12:29:33 DEBUG : finalpage_A4.pdf: Unchanged skipping
2022/10/01 12:29:33 DEBUG : .gitignore: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : eisvogel.tex: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : .gitignore: Unchanged skipping
2022/10/01 12:29:33 DEBUG : eisvogel.tex: Unchanged skipping
2022/10/01 12:29:33 DEBUG : disclaimer.md: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : disclaimer.md: Unchanged skipping
2022/10/01 12:29:33 DEBUG : background.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : background.pdf: Unchanged skipping
2022/10/01 12:29:33 DEBUG : finalpage_A3.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : finalpage_A3.pdf: Unchanged skipping
2022/10/01 12:29:33 DEBUG : Local file system at /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/: Waiting for checks to finish
2022/10/01 12:29:33 DEBUG : templateParams.md: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:29:33 DEBUG : templateParams.md: Unchanged skipping
2022/10/01 12:29:33 DEBUG : Local file system at /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/: Waiting for transfers to finish
2022/10/01 12:29:33 INFO : There was nothing to transfer
2022/10/01 12:29:33 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 8 / 8, 100%
Elapsed time: 0.5s
2022/10/01 12:29:33 DEBUG : 12 go routines active
2022/10/01 12:29:33 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/: Closing 1 unused connections
file:
david@alpe:~/Bin/rclone-v1.59.2-linux-amd64$ ./rclone copy -vv orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md /home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/
2022/10/01 12:30:57 DEBUG : rclone: Version "v1.59.2" starting with parameters ["./rclone" "copy" "-vv" "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md" "/home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/"]
2022/10/01 12:30:57 DEBUG : Creating backend with remote "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md"
2022/10/01 12:30:57 DEBUG : Using config file from "/home/david/.config/rclone/rclone.conf"
2022/10/01 12:30:57 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md: New connection 10.51.175.1:42648->10.51.175.82:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3"
2022/10/01 12:30:58 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md: Shell type "unix" from config
2022/10/01 12:30:58 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md: Relative path resolved to "/home/ubuntu/github/ltng-documentation/template/disclaimer.md"
2022/10/01 12:30:58 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md: Using root directory "/home/ubuntu/github/ltng-documentation/template"
2022/10/01 12:30:58 DEBUG : fs cache: adding new entry for parent of "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md", "orchestra-jammy:github/ltng-documentation/FunctionalDescription/MFS/template"
2022/10/01 12:30:58 DEBUG : Creating backend with remote "/home/david/Temp/lxc-backups/orchestra-jammy/ubuntu/github/ltng-documentation/FunctionalDescription/MFS/template/"
2022/10/01 12:30:58 DEBUG : disclaimer.md: Size and modification time the same (differ by 0s, within tolerance 1s)
2022/10/01 12:30:58 DEBUG : disclaimer.md: Unchanged skipping
2022/10/01 12:30:58 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 0.5s
2022/10/01 12:30:58 DEBUG : 11 go routines active
2022/10/01 12:30:58 DEBUG : sftp://ubuntu@10.51.175.82:22/github/ltng-documentation/FunctionalDescription/MFS/template/disclaimer.md: Closing 1 unused connections