SFTP to SharePoint - problem with illegal characters

Help and Support
1

What is the problem you are having with rclone?

I need to copy an SFTP file system to a site on SharePoint, I have managed to set everything up so that it is transferring most files. The problem that I am having is that some files are not being transferred. The common factor with these files seems to be that they all have a special character within the filename, a dot or a dash that I assume is messing up the transfer.

Is there a way to automatically rename these files to legal names that won't cause problems.

It's also possible that this isn't the reason that they aren't transferring , if that is the case then any advice on how to fix my issues would be greatly appreciated.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.2486 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

SharePoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy SFTP: Sharepoint:

The rclone config contents with secrets removed.

- type: onedrive
- client_id: REDACTED
- client_secret: REDACTED
- auth_url: https://login.microsoftonline.com/2dc2b710-9530-42d2-a55e-b74bebe5124e/oauth2/v2.0/authorize
- token_url: https://login.microsoftonline.com/2dc2b710-9530-42d2-a55e-b74bebe5124e/oauth2/v2.0/token
- token:   REDACTED
- drive_id: b!ybIit53EjUumAObRfEX66lSl9Te5wUBDscWuhGqR7KC2LshYYNwNQIKs8jZcf8TQ
- drive_type: documentLibrary
- encoding: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot
  • type: sftp
  • host: REDACTED
  • user: optimummedical
  • port: 5010
  • key_file: C:\Users\MichaelTaylor\Downloads\optimummedical
  • use_insecure_cipher: true
  • shell_type: cmd
  • md5sum_command: md5sum
  • sha1sum_command: sha1sum

A log from the command with the -vv flag

https://pastebin.com/Kpepu9Kg

Let me know if there is any other information that would be useful to you

2

The log file doesn't work.

I'd assume you want to fix all these items:

3

Ah, yeah it looks like it got taken down. I have gone through all the failed folders and files and they all contain invalid characters. Now I just need a magical solution to go through and change the full stops to dashes. Thank you for the link, it really helped.

4

Sharepoint special characters is one of the bane's of my existence at work..

5

Is this something rclone's encoding should be fixing? It should be translating these characters already I would have hoped.

image
image986×328 34.1 KB

Here is the encoding

--onedrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)

The braces {} seem to be missing from that list. And that list appears to have . in? Are you not allowed . in file names? Or directory names?

6

Hello,

Sorry it took a while to get back to you, I have been a bit ill. I have tried the default encoding, no encoding and re inserting the encoding you have listed. Like you've said there doesn't seem to be an encoding option for {}, however the files that are causing trouble are ones with filepaths or names such as 2023/03. March/

I would have thought the Dot in the encoding would be able to fix it, but for some reason it still seems to be erroring out. here is a paste from the log.

2023/02/15 08:38:24 ERROR : Sunon/Manufacturing Procedures: error reading source directory: error listing "Sunon/Manufacturing Procedures": connection lost
2023/02/15 08:41:15 ERROR : Primecare: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:41:23 ERROR : Fukang: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:41:53 ERROR : Turkuaz: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:42:01 ERROR : Viecura: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:42:07 ERROR : Vitalcare/Manufacturing Procedures/Environmental Monitoring: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:42:15 ERROR : Sunsmed/Batch Records/RETROSPECTIVE - BSI MAJOR - 2022/Nov 2022 [manufactured]: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:42:21 ERROR : Vitalcare/Batch Records/2023/02. February: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:42:23 ERROR : Jiangsu JEK/QMS Procedures: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:45:16 ERROR : Sunsmed/Batch Records/RETROSPECTIVE - BSI MAJOR - 2022/Oct 2022 [manufactured]: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:45:24 ERROR : Sunsmed/Batch Records/2023/08. August: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:45:54 ERROR : Vitalcare/Batch Records/RETROSPECTIVE - BSI MAJOR - 2022/Nov 2022 [manufactured]: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:46:02 ERROR : Vitalcare/Batch Records/Archive/Batch Records: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:46:08 ERROR : Sunon/Manufacturing Validations: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:46:16 ERROR : Vitalcare/Batch Records/2023/03. March: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:46:22 ERROR : Jiangsu JEK/Batch Records: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:46:24 ERROR : Sunon/Batch Records/2023: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:49:17 ERROR : Sunsmed/Batch Records/2023/09. September: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2023/02/15 08:49:25 ERROR : Vitalcare/Batch Records/RETROSPECTIVE - BSI MAJOR - 2022/Oct 2022 [manufactured]: error reading source directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
7

Sorry for the delay responding.

I tried to create this folder on onedrive for business and it worked fine. All your examples seemed to work fine.

I don't have access to a sharepoint though to test...

Can you try, using the sharepoint web interface, creating directories and see if you can get it to give you an error on one of those directory names.