m7eesn
October 3, 2020, 2:33pm
1
What is the problem you are having with rclone?
trying to upload files that has the same name, using --suffix is not working as intended.
What is your rclone version (output from rclone version)
rclone v1.53.1
os/arch: linux/amd64
go version: go1.15
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Distributor ID: Debian
Which cloud storage system are you using? (eg Google Drive)
sftp
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy "folder/" "sshd:/storage/" --suffix -2020_10_03 --suffix-keep-extension -vv --ignore-checksum --ignore-times
The rclone config contents with secrets removed.
[sshd]
type = sftp
host = hostname.tld
user = rclone
port = 2222
key_file = /home/user/.ssh/id_pub
key_use_agent = false
md5sum_command = md5sum
sha1sum_command = sha1sum
A log from the command with the -vv flag
2020/10/03 17:21:08 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "folder/" "sshd:/storage/" "--suffix" "-2020_10_03" "--suffix-keep-extension" "-vv" "--ignore-checksum" "--ignore-times"]
2020/10/03 17:21:08 DEBUG : Creating backend with remote "folder/"
2020/10/03 17:21:08 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2020/10/03 17:21:08 DEBUG : fs cache: renaming cache item "folder/" to be canonical "/home/user/folder"
2020/10/03 17:21:08 DEBUG : Creating backend with remote "sshd:/storage/"
2020/10/03 17:21:09 DEBUG : sftp://rclone@hostname.tld:2222//storage/: New connection ip:34854->ip:28467 to "SSH-2.0-OpenSSH_8.3"
2020/10/03 17:21:09 ERROR : Fatal error received - not attempting retries
2020/10/03 17:21:09 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (fatal error encountered)
Elapsed time: 0.3s
2020/10/03 17:21:09 DEBUG : 11 go routines active
2020/10/03 17:21:09 Failed to copy: --suffix must be used with a file or with --backup-dir
using the same command on a direct file works see log
2020/10/03 17:20:45 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "folder/file.mkv" "sshd:/storage/" "--suffix" "-2020_10_03" "--suffix-keep-extension" "-vv" "--ignore-checksum" "--ignore-times"]
2020/10/03 17:20:45 DEBUG : Creating backend with remote "folder/file.mkv"
2020/10/03 17:20:45 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2020/10/03 17:20:45 DEBUG : fs cache: adding new entry for parent of "folder/file.mkv", "/home/user/folder"
2020/10/03 17:20:45 DEBUG : Creating backend with remote "sshd:/storage/"
2020/10/03 17:20:45 DEBUG : sftp://rclone@hostname.tld:2222//storage/: New connection ip:60742->ip:28467 to "SSH-2.0-OpenSSH_8.3"
2020/10/03 17:20:46 DEBUG : file.mkv: Transferring unconditionally as --ignore-times is in use
2020/10/03 17:20:46 INFO : file.mkv: Moved (server side)
2020/10/03 17:20:46 INFO : file.mkv: Copied (new)
2020/10/03 17:20:46 INFO :
Transferred: 4 / 4 Bytes, 100%, 23 Bytes/s, ETA 0s
Checks: 1 / 1, 100%
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
the remote is docker instance of linuxserver/openssh-server
ncw
October 3, 2020, 7:12pm
2
I did a bit of investigation and I think most likely this is a bug...
I will investigate further once I am in front of me computer.
m7eesn
October 3, 2020, 7:28pm
3
indeed. it seems —suffix is checking the initial source if that happen to be a file it checks out and if it was a directory then it errors out.
ncw
October 4, 2020, 3:42pm
4
It looks like this bug has been present for a while...
I've attempted to make a fix for this here - can you give it a go?
v1.54.0-beta.4794.db4bbf952.fix-suffix on branch fix-suffix (uploaded in 15-30 mins)
asdffdsa
October 4, 2020, 4:25pm
5
the fix worked!
C:\data\rclone\versions\rclone-v1.54.0-beta.4794.db4bbf952.fix-suffix-windows-amd64\rclone-v1.54.0-beta.4794.db4bbf952.fix-suffix-windows-amd64.exe copy C:\ex\test2 seedboxsftp:test2 -vv --suffix -2020_10_03 --suffix-keep-extension --ignore-checksum --ignore-times
2020/10/04 13:25:15 DEBUG : rclone: Version "v1.54.0-beta.4794.db4bbf952.fix-suffix" starting with parameters ["C:\\data\\rclone\\versions\\rclone-v1.54.0-beta.4794.db4bbf952.fix-suffix-windows-amd64\\rclone-v1.54.0-beta.4794.db4bbf952.fix-suffix-windows-amd64.exe" "copy" "C:\\ex\\test2" "seedboxsftp:test2" "-vv" "--suffix" "-2020_10_03" "--suffix-keep-extension" "--ignore-checksum" "--ignore-times"]
2020/10/04 13:25:15 DEBUG : Creating backend with remote "C:\\ex\\test2"
2020/10/04 13:25:15 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/10/04 13:25:15 DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
2020/10/04 13:25:15 DEBUG : fs cache: renaming cache item "C:\\ex\\test2" to be canonical "//?/C:/ex/test2"
2020/10/04 13:25:15 DEBUG : Creating backend with remote "seedboxsftp:test2"
2020/10/04 13:25:16 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: New connection 192.168.255.10:51570->172.20.31.3:9999 to "SSH-2.0-rclone/v1.53.1"
2020/10/04 13:25:16 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: Using absolute root directory "/test2"
2020/10/04 13:25:17 DEBUG : archive.txt: Transferring unconditionally as --ignore-times is in use
2020/10/04 13:25:17 DEBUG : hidden.txt: Transferring unconditionally as --ignore-times is in use
2020/10/04 13:25:17 DEBUG : system.txt: Transferring unconditionally as --ignore-times is in use
2020/10/04 13:25:17 DEBUG : readonly.txt: Transferring unconditionally as --ignore-times is in use
2020/10/04 13:25:17 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: Waiting for checks to finish
2020/10/04 13:25:17 INFO : archive.txt: Moved (server side)
2020/10/04 13:25:17 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: New connection 192.168.255.10:51571->172.20.31.3:9999 to "SSH-2.0-rclone/v1.53.1"
2020/10/04 13:25:17 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: New connection 192.168.255.10:51572->172.20.31.3:9999 to "SSH-2.0-rclone/v1.53.1"
2020/10/04 13:25:18 INFO : archive.txt: Copied (new)
2020/10/04 13:25:18 INFO : hidden.txt: Moved (server side)
2020/10/04 13:25:18 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: New connection 192.168.255.10:51573->172.20.31.3:9999 to "SSH-2.0-rclone/v1.53.1"
2020/10/04 13:25:18 INFO : system.txt: Moved (server side)
2020/10/04 13:25:18 INFO : hidden.txt: Copied (new)
2020/10/04 13:25:18 INFO : readonly.txt: Moved (server side)
2020/10/04 13:25:18 DEBUG : sftp://ecCsXa9Y3zHQGheKOcwS@appbox.local:9999/test2: Waiting for transfers to finish
2020/10/04 13:25:19 INFO : system.txt: Copied (new)
2020/10/04 13:25:19 INFO : readonly.txt: Copied (new)
2020/10/04 13:25:19 INFO :
Transferred: 10 / 10 Bytes, 100%, 4 Bytes/s, ETA 0s
Checks: 8 / 8, 100%
Renamed: 4
Transferred: 4 / 4, 100%
Elapsed time: 3.9s
2020/10/04 13:25:19 DEBUG : 38 go routines active
m7eesn
October 4, 2020, 6:25pm
6
thank you for testing im still at work i'll not be able to test until tomorrow
1 Like
asdffdsa
October 4, 2020, 6:30pm
7
sure,
i think that the log file is not as informative as it can be.Renamed: 4
perhapshidden.txt: Moved (server side)hidden.txt: (server side move + renamed to hidden-2020_10_03.txt)
m7eesn
October 5, 2020, 11:30am
8
i can confirm it's working on this build.
when do we expect this branch to be in the release build?
Thank you for your hard work.
ncw
October 5, 2020, 2:14pm
9
Thanks for testing.
I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.53.2 which should be reasonably soon (1 week maybe?)
Nice idea...
I'd probably do
hidden.txt: Moved (server side) to "path/whatever"
where the to ... but was only added if the "remote" was different which it won't be in most cases.
I could do the same thing to Copy too.
What do you think - worth doing? If so then please please make a new issue on github
m7eesn
October 5, 2020, 2:32pm
10
@ncw thank you very much, and since we are talking about fixing log messages, could you please add 1 extra message on transfer complete that easily parsable by scripts? regardless of completion or failure, right now i have a script that parse rclone logs to know how much upload quota i've used. and it's mutli line which is harder to parse. i'd love something simpler like
2020/10/03 17:20:46 INFO : (Transfer|Operation) Complete. Transferred 00000000 Bytes.
the time and amount is very important. i use them to track how much upload quota i've used on daily basis. what rclone uses right now is like this
2020/10/05 14:17:37 INFO :
Transferred: 1.295G / 1.295 GBytes, 100%, 38.733 MBytes/s, ETA 0s
ncw
October 5, 2020, 2:41pm
11
If you are using -v then you'll get the stats message INFO : Transferred: 1.295G / 1.295 GBytes, 100%, 38.733 MBytes/s, ETA 0s right at the end of the log which should have all the info in, just not easy to parse. If you want easy to parse then use --use-json-log. Is that OK?
m7eesn
October 5, 2020, 2:45pm
12
i version my log files daily eg rclone.2020_10.05.log, and the log file will have multiple transfers operations not single, so we have to parse the entire log file. regardless and having clear indicator that the operation completed eg "DATE INFO: Operation Complete. transferred 0 bytes." would simplify the parsing. either way it's something to think about if you feel it's worth it. my script is working right now with what we have.
1 Like
asdffdsa
October 5, 2020, 3:09pm
13
i tried to add the issue but the submit new issue is disabled...
ncw
October 6, 2020, 4:35pm
14
Odd! Did you log in to GitHub?
asdffdsa
October 6, 2020, 5:52pm
15
i tried again just now and i was able to submit the new issue.
1 Like
system
December 6, 2020, 1:52pm
16
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
