FTP backend is not detecting changes on some files on copy or sync
I just started using the FTP backend for the first time. I have had great success for years with other backends. great product!
I am finding that a sync or copy will not properly detect changed files. for example, i updated the file production and it will not choose to upload - saying its the Unchanged and Sizes identical. When i do a long listing locally and via rclone lsl i see the times on the FTP are indeed in the past. DEBUG doesnt seem to help give more info.
-rw-r--r--@ 1 sphen staff 6153785 Aug 6 11:16 /Volumes/tv-repo/catalogs/production
6153785 2024-08-06 11:06:00.000000000 production
I have attempted other options related to ignoring size --ignore-size as well as --refresh-times and --local-time-type with no change. I can only run with -I to get it to push but this is less than ideal as it pushes ALL files again.
ideas? TIA!
rclone v1.67.0
- os/version: darwin 15.0 (64 bit)
- os/kernel: 24.0.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.22.4
- go/linking: dynamic
- go/tags: cmount
FTP (BunnyCDN Storage)
rclone copy /tmp remote:tmp)rclone copy --exclude '.DS_Store' --delete-excluded --inplace --transfers=16 /Volumes/tv-repo/catalogs/ bunny:catalogs/ -P -v
[bunny]
type = ftp
host = XXX
user = XXX
pass = XXX
-vv flag2024/08/06 11:22:51 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "copy" "--exclude" ".DS_Store" "--delete-excluded" "--inplace" "--transfers=16" "/Volumes/tv-repo/catalogs/" "bunny:catalogs/" "-P" "-vv"]
2024/08/06 11:22:51 DEBUG : Creating backend with remote "/Volumes/tv-repo/catalogs/"
2024/08/06 11:22:51 DEBUG : Using config file from "/Users/sphen/.config/rclone/rclone.conf"
2024/08/06 11:22:51 DEBUG : fs cache: renaming cache item "/Volumes/tv-repo/catalogs/" to be canonical "/Volumes/tv-repo/catalogs"
2024/08/06 11:22:51 DEBUG : Creating backend with remote "bunny:catalogs/"
2024/08/06 11:22:51 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Connecting to FTP server
2024/08/06 11:22:51 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","ny.storage.bunnycdn.com:21")
2024/08/06 11:22:51 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","84.17.35.218:18492")
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","84.17.35.218:23420")
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Waiting for checks to finish
2024/08/06 11:22:52 DEBUG : cchinckley: Sizes identical
2024/08/06 11:22:52 DEBUG : cchinckley: Unchanged skipping
2024/08/06 11:22:52 DEBUG : donercx: Sizes identical
2024/08/06 11:22:52 DEBUG : donercx: Unchanged skipping
2024/08/06 11:22:52 DEBUG : donerjamf: Sizes identical
2024/08/06 11:22:52 DEBUG : donerjamf: Unchanged skipping
2024/08/06 11:22:52 DEBUG : curemark: Sizes identical
2024/08/06 11:22:52 DEBUG : disabled: Sizes identical
2024/08/06 11:22:52 DEBUG : all: Sizes identical
2024/08/06 11:22:52 DEBUG : development-bohemiancoding-Sketch: Sizes identical
2024/08/06 11:22:52 DEBUG : cdx: Sizes identical
2024/08/06 11:22:52 DEBUG : colorblends: Sizes identical
2024/08/06 11:22:52 DEBUG : bakes: Sizes identical
2024/08/06 11:22:52 DEBUG : curemark: Unchanged skipping
2024/08/06 11:22:52 DEBUG : kirschbaum: Sizes identical
2024/08/06 11:22:52 DEBUG : foote: Sizes identical
2024/08/06 11:22:52 DEBUG : disabled: Unchanged skipping
2024/08/06 11:22:52 DEBUG : all: Unchanged skipping
2024/08/06 11:22:52 DEBUG : development-bohemiancoding-Sketch: Unchanged skipping
2024/08/06 11:22:52 DEBUG : cdx: Unchanged skipping
2024/08/06 11:22:52 DEBUG : pca: Sizes identical
2024/08/06 11:22:52 DEBUG : colorblends: Unchanged skipping
2024/08/06 11:22:52 DEBUG : bakes: Unchanged skipping
2024/08/06 11:22:52 DEBUG : kirschbaum: Unchanged skipping
2024/08/06 11:22:52 DEBUG : socialalchemy: Sizes identical
2024/08/06 11:22:52 DEBUG : socialalchemy: Unchanged skipping
2024/08/06 11:22:52 DEBUG : tcs: Sizes identical
2024/08/06 11:22:52 DEBUG : foote: Unchanged skipping
2024/08/06 11:22:52 DEBUG : lawandcrime: Sizes identical
2024/08/06 11:22:52 DEBUG : lawandcrime: Unchanged skipping
2024/08/06 11:22:52 DEBUG : oagroup: Sizes identical
2024/08/06 11:22:52 DEBUG : overabove: Sizes identical
2024/08/06 11:22:52 DEBUG : pca: Unchanged skipping
2024/08/06 11:22:52 DEBUG : production: Sizes identical
2024/08/06 11:22:52 DEBUG : sacredbee: Sizes identical
2024/08/06 11:22:52 DEBUG : tcs: Unchanged skipping
2024/08/06 11:22:52 DEBUG : zabbix: Sizes identical
2024/08/06 11:22:52 DEBUG : zabbix: Unchanged skipping
2024/08/06 11:22:52 DEBUG : testing: Sizes identical
2024/08/06 11:22:52 DEBUG : testing: Unchanged skipping
2024/08/06 11:22:52 DEBUG : thirdvantage: Sizes identical
2024/08/06 11:22:52 DEBUG : thirdvantage: Unchanged skipping
2024/08/06 11:22:52 DEBUG : oagroup: Unchanged skipping
2024/08/06 11:22:52 DEBUG : overabove: Unchanged skipping
2024/08/06 11:22:52 DEBUG : timex: Sizes identical
2024/08/06 11:22:52 DEBUG : timex: Unchanged skipping
2024/08/06 11:22:52 DEBUG : production: Unchanged skipping
2024/08/06 11:22:52 DEBUG : sacredbee: Unchanged skipping
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Waiting for transfers to finish
2024/08/06 11:22:52 INFO : There was nothing to transfer
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 24 / 24, 100%
Elapsed time: 1.0s
2024/08/06 11:22:52 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 24 / 24, 100%
Elapsed time: 1.0s
2024/08/06 11:22:52 DEBUG : 5 go routines active
2024/08/06 11:22:52 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: closing 1 unused connections
asdffdsa
August 6, 2024, 3:48pm
2
hi,
Stephen_Boyle:
some files on copy
to keep the logs simple, for testing, pick a single file, use copy, not sync
can you run rclone checkrclone lsl on both the source and dest
thx
rclone check /Volumes/tv-repo/catalogs/production bunny:catalogs/ -vv
2024/08/06 12:34:17 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "check" "/Volumes/tv-repo/catalogs/production" "bunny:catalogs/" "-vv"]
2024/08/06 12:34:17 DEBUG : Creating backend with remote "/Volumes/tv-repo/catalogs/production"
2024/08/06 12:34:17 DEBUG : Using config file from "/Users/sphen/.config/rclone/rclone.conf"
2024/08/06 12:34:17 DEBUG : fs cache: adding new entry for parent of "/Volumes/tv-repo/catalogs/production", "/Volumes/tv-repo/catalogs"
2024/08/06 12:34:17 DEBUG : Creating backend with remote "bunny:catalogs/"
2024/08/06 12:34:17 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Connecting to FTP server
2024/08/06 12:34:17 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","ny.storage.bunnycdn.com:21")
2024/08/06 12:34:17 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","156.146.59.119:18208")
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:34:18 ERROR : No common hash found - not using a hash for checks
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Waiting for checks to finish
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","156.146.59.119:31158")
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:34:18 DEBUG : production: OK - could not check hash
2024/08/06 12:34:18 NOTICE: ftp://ny.storage.bunnycdn.com:21/catalogs: 0 differences found
2024/08/06 12:34:18 NOTICE: ftp://ny.storage.bunnycdn.com:21/catalogs: 1 hashes could not be checked
2024/08/06 12:34:18 NOTICE: ftp://ny.storage.bunnycdn.com:21/catalogs: 1 matching files
2024/08/06 12:34:18 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 1.0s
2024/08/06 12:34:18 DEBUG : 6 go routines active
2024/08/06 12:34:18 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: closing 1 unused connections
rclone lsl /Volumes/tv-repo/catalogs/production
6153785 2024-08-06 12:33:53.236817086 production
rclone lsl bunny:catalogs/production
6153785 2024-08-06 11:30:00.000000000 production
rclone copy --inplace /Volumes/tv-repo/catalogs/production bunny:catalogs/ -P -vv
2024/08/06 12:36:38 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "copy" "--inplace" "/Volumes/tv-repo/catalogs/production" "bunny:catalogs/" "-P" "-vv"]
2024/08/06 12:36:38 DEBUG : Creating backend with remote "/Volumes/tv-repo/catalogs/production"
2024/08/06 12:36:38 DEBUG : Using config file from "/Users/sphen/.config/rclone/rclone.conf"
2024/08/06 12:36:38 DEBUG : fs cache: adding new entry for parent of "/Volumes/tv-repo/catalogs/production", "/Volumes/tv-repo/catalogs"
2024/08/06 12:36:38 DEBUG : Creating backend with remote "bunny:catalogs/"
2024/08/06 12:36:38 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: Connecting to FTP server
2024/08/06 12:36:38 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","ny.storage.bunnycdn.com:21")
2024/08/06 12:36:38 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:36:39 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","156.146.59.119:34700")
2024/08/06 12:36:39 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:36:39 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: dial("tcp","156.146.59.119:25660")
2024/08/06 12:36:39 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: > dial: conn=*fshttp.timeoutConn, err=<nil>
2024/08/06 12:36:39 DEBUG : production: Sizes identical
2024/08/06 12:36:39 DEBUG : production: Unchanged skipping
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.0s
2024/08/06 12:36:39 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 1.0s
2024/08/06 12:36:39 DEBUG : 5 go routines active
2024/08/06 12:36:39 DEBUG : ftp://ny.storage.bunnycdn.com:21/catalogs: closing 1 unused connections
asdffdsa
August 6, 2024, 4:55pm
4
so the source has a modtime that is newer than the dest, correct?
yes correct -
source: 12:33:53.236817086
remote: 11:30:00.000000000
weird right? its like just ignoring that? any way to get deeper DEBUG?
asdffdsa
August 6, 2024, 5:14pm
6
well not sure we can do anything - if i dump, i clearly see the mod date/time (in UTC):
2024/08/06 13:28:40 DEBUG : FTP Rx: "-rwxrwxrwx 2 2003 2003 6153785 Aug 06 15:30 production"
i wonder if rclone is ignoring times because this remote does not allow you to set the mod date:rclone backend features bunny:
"Name": "bunny",
"Root": "",
"String": "ftp://ny.storage.bunnycdn.com:21",
"Precision": 3153600000000000000,
"Hashes": [],
wonder what happens if i throw a hasher in front of it
asdffdsa
August 6, 2024, 5:36pm
8
can test that using rclone touch
sftp?
well i see this as a bug - but genius - sftp.... it was not well documented but bunnycdn supports sftp as well. when i switch to that it is working well. the mod times on the remote are set to when they were uploaded so i need to add the -u flag, but it does upload changed files properly now. thanks for the help
jwink3101
August 6, 2024, 11:08pm
10
It is probably worth reading and testing as per the ModTime section of the docs . Your lsl results show a ModTime but that doesn't mean the backend is reporting that it has the precision to use it (even if it does).
What do you get when you run:
rclone backend features bunny:
???
Thanks. Yeah the backend shows it has no precision on time. Even though it shows it properly. Oh well. Either way the SFTP backend has resolved it for this particular case. So that’s a win. Also looks like someone was working recently on an api backed backend for bunnycdn
asdffdsa
August 7, 2024, 12:02am
12
good.rclone config redacted bunny: ?
super simple:
[bunny]
type = ftp
host = XXX
user = XXX
pass = XXX
asdffdsa
August 7, 2024, 12:31pm
14
Stephen_Boyle:
super simple:
sorry, my bad, i meant to ask about the sftp remote?
[bunny]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none
sftp shows 1 second precision vs ftp - none
ncw
August 8, 2024, 2:55pm
16
The FTP backend does support setting modification times, but only if the server supports it.
The server needs to support MFMT or MDTM
In general syncing to a backend which can't set modification times means effectively using --size-only or you can add the --update flag.
The sftp protocol is much better designed so it is a good choice, though ftp may be faster since it is simpler and just blasts stuff at TCP sockets whereas sftp packetizes everything.
system
September 7, 2024, 2:56pm
17
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
