What is the problem you are having with rclone?
I am attempting to copy files from a remote machine to S3 but during the copy process, it is failing to calculate the md5 hash on the sources-side which causes the transfer to fail. The error returned during the MD5 hash states the file in source destination cannot be found despite the file still existing in the source.
Run the command 'rclone version' and share the full output of the command.
$ rclone version
rclone v1.68.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-79-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy --contimeout=10s --progress --ignore-case "remote-machine:/C:/path/" "my-s3-bucket:/my-s3-path/" --filter="+ **/*.{tdms,tdms_index}" --filter="- *"
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[remote-machine]
type = sftp
shell_type = unix
host = XXX
port = 22
user = XXX
pass = XXX
md5sum_command = md5sum
sha1sum_command = sha1sum
[my-s3-bucket]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-west-2
storage_class = STANDARD
A log from the command that you were trying to run with the -vv flag
Not all files transferred have the following issue. Some files transfer without problem, but others have the issues below. The files are between 100mb - 2gb, not large files.
$ sudo rclone copy --contimeout=10s --progress --ignore-case "remote-machine:/C:/path/" "my-s3-bucket:/my-s3-path" --filter="+ **/*.{tdms,tdms_index}" --filter="- *" -vvv
...
2024/12/11 14:11:56 DEBUG : sftp://username@ip_address//C:/path_to_directory/: Shell path_to_directory "/C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index"
2024/12/11 14:11:56 DEBUG : sftp://username@ip_address//C:/path_to_directory/: Running remote command: md5sum /C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index
...
024/12/11 14:11:58 DEBUG : sftp://username@ip_address//C:/path_to_directory/: Shell path_to_directory "/C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index"
2024/12/11 14:11:58 DEBUG : sftp://username@ip_address//C:/path_to_directory/: Running remote command: md5sum /C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index
...
2024/12/11 14:11:59 ERROR : TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index: Failed to calculate src hash: failed to calculate md5 hash: failed to run "md5sum /C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index": /usr/bin/md5sum: '/C:/path_to_directory/TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index': No such file or directory: Process exited with status 1
2024/12/11 14:11:59 ERROR : TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index: corrupted on transfer: md5 hashes differ src(sftp://username@ip_address//C:/path_to_directory/) "" vs dst(S3 bucket my-s3-bucket path path_to_directory/) "f7cfc7a8ad13f0562ba2a20cc3d7ef6d"
2024/12/11 14:11:59 INFO : TCW_SN1001001_ATP/TCW_SN1001001_Mission16_Run4.tdms_index: Removing failed copy
...
I believe the order of operations is:
- The file transfer begins and the file is copied over from source remote machine to destination AWS s3 directory
- Transfer finishes
- MD5 hash is calculated for both source and destination file
- File in source location cannot be located and MD5 hash returns an empty string which does not match the MD5 hash in the destination location. However, I can confirm that the file still exists in the source destination
- MD5 hashes between source & destination location differ. The transfer is canceled, no files appear in AWS S3