--sftp-path-override breaks on wrapped remotes

Suspected Bug
1

Assume the following config:

[crypttest]
type = crypt
remote = sftptest:a/b/c
password = [redacted]

[sftptest]
type = sftp
host = [redacted]
user = [redacted]
port = [redacted]
pass = [redacted]
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

[aliastest]
type = alias
remote = sftptest:a/b/c

And assume the typical Synology NAS format mentioned in the docs where the SSH path has an extra /volume1 that the SFTP path does not.

The following commands have the following results:

rclone copy a sftptest:a --sftp-path-override '/volume1'

:x: Fails.

rclone copy a sftptest:a --sftp-path-override '/volume1/a'

:white_check_mark: Succeeds.

rclone copy f aliastest:d/e/f --sftp-path-override '/volume1'

:x: Fails.

rclone copy f aliastest:d/e/f --sftp-path-override '/volume1/a/b/c'

:x: Fails.

rclone copy f aliastest:d/e/f --sftp-path-override '/volume1/a/b/c/d/e/f'

:white_check_mark: Succeeds.

rclone copy f crypttest:d/e/f --sftp-path-override '/volume1'

:x: Fails.

rclone copy f crypttest:d/e/f --sftp-path-override '/volume1/a/b/c'

:x: Fails.

rclone copy f crypttest:d/e/f --sftp-path-override '/volume1/a/b/c/d/e/f'

:x: Fails.

rclone copy f crypttest:d/e/f --sftp-path-override '/volume1/a/b/c/e2dvum7tosiskgm13qioobd724/1fbvn12o0fm0ojkihl69f0rsrc/jounpnkjf4pnmlo6732gjeclls'

:white_check_mark: Succeeds.

(didn't use my real password -- don't worry.)

I also tried setting it from the path_override config variable -- didn't seem to change anything.

Is this expected behavior? If so, is there any workaround? (Other than --ignore-checksum)

In my opinion, it would be more intuitive if --sftp-path-override were the path to the SFTP remote's root, not the path to the root + subpath + wrapping-subpath. In other words, in the above examples, this should suffice: --sftp-path-override '/volume1' and rclone should be able to add the relative subpaths automatically (including unwrapping/unencrypting as necessary). That way, it could just be set once in the config, without needing to re-specify for each command. Perhaps this could be implemented as a new flag (for backward compatibility)?

Currently, the error is:

2023/07/16 23:04:20 ERROR : test.txt.yefanib2.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5sum /volume1/test.txt.yefanib2.partial": md5sum: /volume1/test.txt.yefanib2.partial: No such file or directory: Process exited with status 1
2023/07/16 23:04:20 ERROR : test.txt.yefanib2.partial: corrupted on transfer: md5 hash differ "098f6bcd4621d373cade4e832627b4f6" vs ""

Or, for crypt:

2023/07/17 00:13:27 ERROR : test.txt: Failed to copy: failed to read destination hash: failed to calculate md5 hash: failed to run "md5sum /volume1/1642p05ttgmjk9aem9iva6cbb3t51p80atv1a5ikhga1ro303g90": md5sum: /volume1/1642p05ttgmjk9aem9iva6cbb3t51p80atv1a5ikhga1ro303g90: No such file or directory: Process exited with status 1

rclone version:

rclone v1.64.0-DEV
- os/version: darwin 13.4.1 (64 bit)
- os/kernel: 22.5.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.1
- go/linking: dynamic
- go/tags: none
2

Yes possibly you are right. This would be a backwards incompatible change though so need a new config parameter if we changed it. Or we could add some glyph onto the path to denote that it indicated an absolute path, eg @/path.

Yes I think that would work.

The new parameter could be /path to start from the absolute root or path which would start from the users home directory. It might need to contain ../sums or something.

Do you want to have a go at this @nielash ?

1 Like
3

Sure, I'm happy to take a look.

That's very clever about the glyph. I like it. :slightly_smiling_face:

1 Like
4

For anyone following along here: I opened the following PR.

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.